Nvidia has made another attempt to add depth to shallow images.
The GPU giant has a new one today AI method that converts still photos into 3D objects that creators can easily modify.
Dubbed 3D MoMa, the technique could give game studios an easy way to change images and scenes. This usually relies on time-consuming photogrammetry, which involves taking measurements from photos.
3D MoMa speeds up the task through inverse rendering. This process uses AI to estimate the physical characteristics of a scene — from geometry to lighting — by analyzing still images. The photos are then reconstructed into a realistic 3D form.
David Luebke, Nvidia’s VP of graphics research, describes the technique as “a holy grail that unites computer vision and computer graphics.”
“By formulating each part of the inverse rendering problem as a GPU-accelerated differentiable component, the NVIDIA 3D MoMa rendering pipeline uses the machinery of modern AI and the raw computational power of NVIDIA GPUs to quickly produce 3D objects that creators can import , edit, and unlimited extend into existing tools,” says Lubeke.
3D MoMa generates objects as triangular meshes — a format that is easy to edit with commonly used tools. The models are made in under an hour on a single NVIDIA Tensor Core GPU.
Materials can then be placed on the mesh as skins. The lighting of the scene is also predicted, allowing creators to change the effects on the objects.
3D MoMa was exhibited this week Conference on Computer Vision and Pattern Recognition (CVPR) in New Orleans. As a tribute to the birthplace of jazz, Nvidia researchers used the technique to visually represent the musical genre.
The team first collected hundreds of photos of trumpets, trombones, saxophones, drums and clarinets. Then, 3D MoMa reconstructed the images into 3D representations.
The instruments were then processed and provided with new materials. Thus the trumpet was transformed from cheap plastic to lavish gold.
The newly edited instruments were then ready to be placed in each virtual scene. Nvidia dropped them in a Cornell box, which is used to test the display quality.
The company says all instruments responded to light as they would in the real world, from the brass instruments that reflect brightly to the drum heads that absorb light.
Finally, the 3D objects were rendered in an animated scene.
3D MoMa continues to evolve, but Nvidia believes it could allow game developers and other designers to quickly modify 3D objects — and then add them to any virtual scene.
That could also facilitate our metamorphosis into metaverse forms.
You can read the study paper behind 3D MoMa . read here†