Introducing ControlRoom3D by Meta. It’s an innovative tool that generates diverse and plausible 3D room meshes that aligns perfectly with user-defined room layouts and textual descriptions of the room style. Visit the project page or read the paper for more in-depth information.
Our major breakthrough is using the geometric and semantic information that the 3D representation gives us when converted to 2D. We use this to control 2D generative models, allowing for the creation of 3D-consistent textures and geometry aligning with the proxy room.
We put a new spin on scene generation. Instead of creating 2D views individually, we’ve developed a panorama-generation technique that provides a 360° view. This initial view serves as a base and additional viewpoints help in inpainting missing areas.
An innovative addition is our geometry alignment technique. By using the spatial dimensions of 3D bounding boxes, we’ve developed an iterative optimization strategy that aids a depth estimation network to neatly produce 3D structures.
Our new process offers versatility, enabling the creation of a wide range of scene types and layouts, even complicated ones like kitchens.
This project is the result of collaboration with an incredible team of Metamates: Sam Tsai, @LukasHollein, @wu_bichen, Jialiang Wang, Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, @felixwimbauer, Zijian He, Peizhao Zhang, Bastian Leibe, Peter Vajda, @sekunde_.
What's new
How does it work-1
How does it work-2
How does it work-3
kitchens
bathrooms
children's room
overall