CARPOS Deliverable 3 - Fruit Picking Action Identification and Skill Extraction from Visual Observation

This deliverable (D3) documents the methodologies and outcomes of the research conducted in the context of “Fruit picking action identification and skill extraction from visual observation” within the CARPOS project. The work is part of Work Package 2 (Active Perception) and focuses on enabling the robot to learn harvesting skills by observing human demonstrations. As the action is performed by a human hand, an in-hand RGB-D camera (i.e., a camera attached to the robot’s end-effector) is utilized to capture the configuration of the human hand, the contact points with the fruit, and the fruit’s geometry and pose. State-of-the art computer vision methods were applied to detect and track these features in real-time, while active perception strategies were employed to mitigate occlusion issues. The recorded trajectories and interaction points were mapped into a robotic representation suitable for subsequent skill encoding (WP4). The results demonstrate the system’s reliability to identify the fruit-picking actions and extract the essential motion and contact patterns required for autonomous reproduction by the robotic gripper. This deliverable, therefore, provides a critical link between raw visual observation and the multimodal learning framework of the CARPOS project.