We propose to research, develop, and demonstrate Handler, an autonomous semantic detection, planning, and grasping affordance module capable of fast online inference and adaptation, as well as continuous learning and improvement. Handler will work by combining semantic and primitive pose-recognition algorithms with a rich affordance template library and online grasp-finding algorithms. Uniquely, it will also feature a pipeline for expanding its semantic recognition and affordance library through continuous learning using state-of-the-art mesh generation tools. It will then estimate grasping locations from learned grasp generation models using these created meshes. Handler will consist of four primary tools: Handler Environment Constructor, which will leverage state-of-the-art object classifier and pose extraction algorithms to automatically create a digital twin of the real environment and the objects within. Handler Dynamic Affordance Template Library, which is a database of modifiable objects that encode a mesh and defined object interactions, including candidate grasp locations and suggested trajectories between grasp points. These templates can be applied to objects found by the environment constructor, and govern how they can be manipulated by robots. Handler User Interface, which allows users to view the digital twin of the environment and manage the affordance library, including tweaking existing affordance templates, adjusting upcoming interactions or grasp methods, or capturing perception and semantic data for construction of new templates. Handler Affordance Builder, which can automatically create object meshes from captured video and use them to both train pose estimation and semantic classifier networks, as well as create new affordance templates using automatic grasp calculators. Anticipated
Benefits: At the end of Phase II, we expect to demonstrate significant progress in assembly, maintenance, and logistics tasks well representative of typical NASA IVR and lunar surface activities. Development and testing will be done on the NASA robot R5 Valkyrie in order to coordinate with ongoing NASA development efforts, and also on the Nadia humanoid in order to promote cross platform capabilities, with basic demonstrations being shown on R5 in Phase I. The Handler library will overcome many barriers associated with the adoption of robotic manipulators in commercial settings. It will also lead to more reliable task performance and more situational awareness in applications involving remote robotics operations like EOD or CONMIT. Additionally, it will support online creation of affordance templates to address complex and unexpected situations.
