Nvidia's latest AI research will make it easier to build 3D objects out of 2D images
Nvidia on Tuesday is debuting new research into 3D rendering that could one day make it easier for graphic artists, architects, and other creators to quickly build 3D models out of 2D images. It's the sort of research that will help Nvidia and other major companies lay the foundations for an easily-accessible metaverse.
Rendering 3D worlds is a complicated technical challenge. One technique known as inverse rendering involves reconstructing 3D scenes from a handful of 2D images. It uses AI to approximate how light behaves in the real world.
Innovation
Nvidia recently showcased its work with neural radiance fields (NeRF), which uses AI to speed up the process of inverse rendering. The downside of NeRF models is that they leave you with images you can't easily edit -- that's not great for a designer creating something new.
Nvidia's new research addresses that problem. The new Nvidia 3D MoMa pipeline reconstructs images with three features: a 3D mesh model, materials, and lighting. The 3D mesh model is built out of triangles. The materials are 2D textures that are overlaid on the 3D mesh like a skin. Lastly, 3D MoMa estimates how a scene is lit.
The finished product is directly compatible with the graphics engines and modeling tools that creators already use, meaning creators can modify and experiment with their models -- changing materials or lighting effects.
Nvidia is presenting the inverse rendering pipeline this week at the Conference on Computer Vision and Pattern Recognition in New Orleans. To demo it, Nvidia's research and creative teams have created 3D models of jazz band instruments. They started with 100 images each of five different instruments. After 3D MoMa created 3D mesh representations of each, the team took the models out of their original scenes and imported them into the Nvidia Omniverse 3D simulation platform to edit.
They changed the materials on the instruments, replacing the trumpet's original plastic with other materials like gold and wood. Then they demonstrated how the different materials reflect light differently.