Grounded Force Feedback with Haptic Glove

Description:

In collaboration with HaptX and as part of my Senior Design Capstone Team Project, this project was designed to be the upper half of ForceBot. HaptX provided virtual reality haptic gloves which the user wears to get feedback on their fingertips from objects they interact with in the virtual environment. This system was extended by mounting the HaptX gloves to the end-effector of a 7-DOF Panda robotic arm. This Panda arm applied forces to the user’s hand based on the objects they were interacting with within the virtual environment. For example, if the user picked up a baseball, they would feel the weight of the baseball in their hands from the force being simulated by the robotic arm. 

My task on this team project was to work out the communication pipeline, the driver software for each computer system, and the data processing for force conversion. The communication pipeline was extremely complex. Unity was used to render the virtual environment and sent virtual interaction data to the high-level code which used the IHMC software for force conversion and mathematical processing. Our high-level code then communicated commands to the Panda arm via a C++ program that used the libfranka library.

The data from Unity is the displacement and velocity vectors of the user’s hand within the virtual environment. This data is received by Unity using a Vive Tracker and placing it on the HaptX glove. The high-level code receives this data and converts the displacement and velocity values to force values. These force values are then sent to the C++ program to command the robotic arm. The user would then feel the desired force on their respective hand completing the communication loop. 

Our work was demonstrated at the end of the year Senior Design Exposition in which other students were able to try out our system. Many virtual environments were developed for the expo; the most common environment used was the Towers of Hanoi. In this environment, the user moves the rings manually to each of the pegs and can experience feedback from the rings, pegs, and the static table.