One Sim to Rule Them All: Advancing Simulations with Neural Radiance Fields

Robotic simulators have long been an essential tool for designing and testing robotic systems as they enable researchers, engineers, and designers to experiment with different robotic tasks without causing damage to hardware or the surrounding environment. This is important across all robotic tasks but even more so for people-centered ones such as the use of robotic assistive technologies. A key factor in the success of these simulators is the ability to create realistic environments that support accurate sensor outputs to support the decision making of the agents. These qualities are important because they increase confidence in refining robotic tasks in a simulation environment that accurately represents the real world, making it safer to transfer testing to the physical environment. 

Historically, 3D assets for simulator environments have been handcrafted, however, this takes a significantly long time which is proportionate to the scale of the environment you are trying to represent. The alternative process, traditional 3D reconstruction, often struggles to capture all the details of complex scenes or objects, resulting in greater inaccuracies. In recent years, neural radiance fields (NeRFs) have emerged as a powerful tool for generating photo-realistic 3D scene representations. NeRF models implicitly represent a 3D scene as a radiance field approximating a 5D function using neural networks. The outputs of these models, color, and volume density, can be used to produce multiple common-place sensors in robotics such as an RGB-D camera and LiDAR. 

In this presentation, we will outline our approach and share the latest advancements in our development of a cutting-edge robotic simulator that utilizes NeRFs to build highly realistic environments and model corresponding sensor readings quickly and accurately. Our methodology and progress to date will be presented, showcasing the potential of this technology in robotic simulation.