D2.3 - Simulation infrastructure for handling component training

HARTU has developed a set of simulation tools (SimEnv) to help system integrators configure some features of a handling application. This document describes these tools. SimEnv includes the SyntheticImageDatasetGenerator component, which assists in the creation of a synthetic image dataset using Unity. The dataset will be used to train an object detection YOLOv5 model whose output will be part of the input to a generic object segmentation model to perform the segmentation of the object in a real scene. The SimEnv component of the HARTU reference architecture includes also the LocalGraspPointTester component, which assists in the validation of grasping points proposed by LocalGraspModeller, and GlobalGraspPolicyTester component, which assists the GlobalGraspModeller in defining the GlobalGraspModel to decide which object in a cluttered
scene is the best candidate to be picked up.

These components included in SimEnv use two well-known simulation engines: Unity as a general framework and for the generation of realistic images, and MuJoCo as a physics engine. In addition, the Learning from Demonstration component also uses MuJoCO to record, refine, and pre-evaluate the assembly skills demonstrated by the user. While the recording of skills will mainly be done in real-world scenarios, the refinement of skills via Inverse Reinforcement Learning, can only be done in simulation, as it usually requires many evaluations. This simulation environment is not intended to simulate the complete sequence of actions in a handling or assembly robotic application, but to assist the system builder in configuring it.