AI copies human CAD moves to turn sketches into fast 3D models

MIT engineers have developed an AI model that mimics human interactions within CAD software to convert 2D sketches into 3D models by performing precise user interface actions such as clicking buttons, dragging, and selecting tools. This approach aims to reduce the steep learning curve of CAD design, making it faster and more accessible, especially for beginners. Central to this effort is the creation of VideoCAD, a dataset containing over 41,000 videos that capture every detailed UI action taken by designers during the modeling process. By training on this data, the AI learns to replicate the exact sequences of interactions needed to build 3D objects from sketches. The innovation builds on advances in AI-driven user interface agents but addresses the greater complexity of CAD tasks, which require fine control over tool selection, region specification, and shape manipulation. The system translates high-level design commands into pixel-level UI operations, enabling it to operate CAD software autonomously. Initial tests show the AI can successfully create both simple and complex 3D models