A process known as "inverse rendering" uses artificial intelligence (AI) to mimic the behavior of light in the real world, allowing researchers to reconstruct a 3D scene from a handful of 2D images taken from different angles. NVIDIA has applied this approach to new AI technology called Neural Radiance Fields (NeRFs), which are designed to produce results particularly quickly - and appropriately christened the process "Instant NeRF."
NeRFs use neural networks to display and render realistic 3D scenes based on a collection of 2D images. To do this, the neural network needs a few dozen images taken from different positions around the scene - as well as the camera position for each of those shots. For a scene with people or other moving elements, the faster these shots are taken, the better. If too much movement occurs during 2D image capture, the 3D scene generated by the AI will be blurred.
A NeRF then fills in the gaps by training a small neural network to reconstruct the scene by predicting the color of light radiating from each point in 3D space in each direction. If objects seen in some images are obscured by obstacles such as columns in others, the technique can even factor out those obscurations.
Creating a 3D scene using traditional methods takes hours or longer, depending on the complexity and resolution of the visualization. Early NeRF models rendered crisp scenes without artifacts in minutes, but still required hours to train.
Instant NeRF, on the other hand, cuts rendering time by several orders of magnitude. The result is said to be the fastest NeRF technique to date, achieving more than 1,000x speedup in some cases. The model takes only a few seconds to train on a few dozen photos - though it also needs data on the camera angles from which they were taken. It can then render the resulting 3D scene within ten milliseconds. Since it is a lightweight neural network, it can be trained and run on a single NVIDIA GPU.
Instant NeRF was unveiled this week at the NVIDIA GPU Technolgy Conference and could be used to create avatars or scenes for virtual worlds, capture video conference participants and their surroundings in 3D, or reconstruct scenes for 3D digital maps. So, of course, this technology fits very well with the current hype around the upcoming metaverses.
;)