Using generative AI to diversify virtual training grounds for robots - Robohub

Researchers at MIT’s CSAIL and the Toyota Research Institute have developed a novel “steerable scene generation” system to create diverse, realistic 3D digital environments—such as kitchens, living rooms, and restaurants—for training robots. Unlike traditional methods that rely on time-consuming real-world demonstrations or handcrafted simulations that often lack physical realism, this approach uses a diffusion model guided by Monte Carlo tree search (MCTS) to generate and refine scenes. Trained on over 44 million 3D rooms, the system places objects in new arrangements while ensuring physical accuracy, such as preventing object clipping, thereby producing lifelike environments that better mimic real-world physics. The key innovation lies in framing scene generation as a sequential decision-making process, where MCTS evaluates multiple scene variations to optimize for specific goals, such as physical realism or object diversity. This method allows the creation of complex scenes that surpass the complexity of the training data, exemplified by a restaurant scene containing up to 34 items—double the number