Projects

Wheelchair Mounted Robotic Arm

WMRA is a project that combines a wheelchair's mobility control and a 7-joint robotic arm's manipulation control in a single control mechanism that allows people with disabilities to do many activities of daily living (ADL) with minimum or no assistance, some of these activities and tasks are otherwise hard or impossible for people with disabilities to accomplish.

Hardware Design and Control
  • Seven motorized revolute joints on the arm; can reach any position in space within its range of motion.
  • Modular arm; to change the workspace of the arm to accommodate individual needs.
  • Custom designed gripper to pick up a range of objects from a sheet of paper to a soccer ball. Adjustable claw tilt to accommodate different shapes.
  • Servo control boards and motors.
  • Control system combines mobility and manipulation, in a smart 9-degree-of-freedom control algorithm.
  • Redundancy resolution is used for increased maneuver-ability, to eliminate singularities, and avoid joint limits and obstacles.
  • Weight and Payload: –24 lbs / 8.5 lbs
Task-Oriented Control of the WMRA System
  • Optimization methods are used to fulfill two separate trajectories simultaneously, one for the gripper and the other for the wheelchair.
  • Having two separate trajectories can be utilized to achieve various ADL tasks, such as opening a spring-loaded door inwards and going through the door while maintaining the pose of the gripper holding the door knob.
  • Redundancy was resolved to maximize the manipulability measure during the task performance. The wheelchair motion was used to compensate for the decrease in manipulability measure of the arm while going through the door.
Object shape and pose recovery of unknown objects
  • Reconstruct the shape and pose of a novel object to a sufficient degree of accuracy such that it permits grasp and manipulation planning.
  • This is implemented two different ways: capture three images of the object and generate a silhouette and a point cloud that approximates the surface of the object. Then improve the approximation by fitting a superquadric shape to the points.
  • Another method to do this is using Saliency Detection. Effective shapes can be extracted from a complex object image. These shapes can then be reconstructed and used for grasping.
  • Microsoft Kinect is used to get the image and 3D information of objects of interest.
Motion scaling and grasping
  • Grasps which involve a human and a robot working together are more robust than those which are either autonomous or teleoperated.
  • Intention based assistance to the human teleoperating the arm in the form of motion scaling can enhance the arm's usability and the user's capabilities.
  • Using Kinect to extract basic shape and pose information of objects, a pre-shape configuration for grasping is determined through intention estimation algorithms (Hidden-Markov Model).
  • Motion assistance is provided to the user based on the pre-shape configuration by scaling up the user's motion towards that configuration and scaling down the user's motion against that configuration.
  • Fine adjustment of the gripper's pose is done using simple geometrical methods and finally the grasp is realized.
  • Omni Phantom Haptic device from Sensable Technologies is used as the master device and WMRA as the remote robot.
Mobility & Manipulation using Visual Servoing Control
  • Image-based visual servoing (IBVS) approach with scale-invariant feature transform (SIFT) was used for combined autonomously control of mobility and manipulation for the 9-DoF WMRA system.
  • Physical implementation with a "Go to and Pick Up" task and a "Go to and Open the Door" task was developed and presented.
  • A Logitech C910 USB webcam mounted in eye in hand configuration on the end effector was used to capture 30 fps video stream of the environment with the object in sight.
  • For estimating the depth distance from the camera to the goal object, a Sharp GP2Y0A21YK infrared proximity sensor mounted just beneath the camera was utilized.
  • Users operate the system with a laptop using a GUI developed for the application.
Autonomous Navigation, Simultaneous Localization and Mapping (SLAM)
  • Simultaneous localization and mapping (SLAM) algorithm is used to localize the wheelchair in an unstructured environment, and to build a dynamically changing map of the environment.
  • Motion planning and navigation can be optimized based on SLAM data to reach a moving target and achieve a task while avoiding moving obstacles.
  • Human perception and observations can be integrated to the algorithm to achieve better results and more accurate task implementation.
  • Microsoft Kinect is utilized to capture the 3D information of the environment.
  • Extended Kalman Filter is used for localization.
Vision-based system for selecting objects and tasks using a Brain-Computer Interface (BCI)
  • Using the Brain-Computer Interface can be very tiring and lengthy in order to communicate choices to the system. Vision-based algorithm is being developed to make it easier to use any user interface system to control WMRA, including the BCI.
  • This interface is useful for patients who are completely locked-in and are unable to use any body parts for doing their daily activities.
  • In order for this selection system to be operable by BCI, the scene is divided into a grid, like that used in the P300 Speller.
  • The user needs to select the cell whose blue dot falls on the object of interest.
  • The cells are flashed in a random order. A cap with electrodes worn by the user detects P300 Electroencephalography (EEG) signals in the users' brain when an Even Related Potential (ERP) is generated.
  • An ERP is generated based on the Oddball Paradigm in which a signal occurs when an odd event occurs from a sequence of events. The odd event in this case is the flashing of the cell of interest out of the random flashing.
  • The BCI2000 program detects the cell that the user is focusing his/her attention on and then by using a Flood-fill algorithm and edge detection, segments the object.
  • Once the object is selected, it runs through an object recognition algorithm to recognize the object and create a list of possible tasks to do with the object. This list is displayed to the user and interfaced with the BCI.
  • The program is being developed to include functions for zooming and moving the robot to allow for more precise selection.
  • The gridlines are generated intelligently using edge detection output, to increase the likelihood of selection in a single iteration.
  • This system will also be operable by a myriad of other user input devices, including eye tracking, voice control, and touch-screen.
  • Previous implementations included directional control of the WMRA system using BCI for teleoperation. This implementation includes the vision based algorithm that is integrated to the BCI for autonomous motion.
Experimental evaluation of WMRA devices
  • JACO arm from Kinova and iARM from Exact Dynamics are two commercially available wheelchair-mounted robotic arms (WMRAs) internationally.
  • Experimental evaluation of commercially available WMRAs in a controlled test environment has been conducted to study the efficacy of using such devices.
  • The goal was to quantitatively compare each device through a standardized testing protocol. The study produced theoretical manipulability measurements as well as efficacy ratings of each device based on Denavit-Hartenberg kinematic parameters and physical testing, respectively.
  • Both the manipulator and control devices of WMRA systems were evaluated. The iARM was found to be more effective than the JACO arm based on kinematic analysis. Despite this, the JACO arm was shown to be more effective than the iARM system in three of four experimental tasks.
  • Effective design features were brought to light with these results. The study and its procedures may serve as a source of quantitative and qualitative data for the commercially available WMRAs.