Application of Reinforcement Learning to UR10 Positioning for Prioritized Multi-Step Inspection in NVIDIA Omniverse

Sequential multi-step operations have long played an important role in manufacturing systems. In high level multi-step manufacturing processes, multiple operations are carried out in sequence on different machines/robots, while in low level applications, multi-step operations are performed by a single robot. This paper in particular studies the multi-step inspection problem with focus on robot-positioning. Traditionally, multi-step robot positioning was implemented in engineering cycles under a set of predetermined conditions. Nonetheless, due to recent trends towards mass-customization, the dynamic nature of working environments coupled with uncertainty has imposed multi-step robot positioning tasks with challenges that transcend static engineering cycles and require an automated adaptation process when dealing new products. Based on the requirements associated with multi-step inspection processes (in the context of mass-customization), and the associated dynamicity and uncertainty, this paper investigates the use of reinforcement learning in bringing automation into the development of inspection systems by focusing on the multi-step robot positioning problem. The simulations have been implemented for a set of state of the art reinforcement learning algorithm including DDPG, TD3, TRPO and PPO via NVIDIA Omniverse, Isaac Sim environment. Our experiments indicated that TRPO demonstrated the overall best performance for the accuracy in visiting the inspection points and job rejection rate, whereas PPO, as a theoretically more capable method, exhibited a lower performance.