Analyzing the Internals of Neural Radiance Fields

CVPRW 2024

Workshop on Neural Rendering Intelligence

Lukas Radl   Andreas Kurz   Michael Steiner   Markus Steinberger
Graz University of Technology
arXiv Code Data

Abstract

Modern Neural Radiance Fields (NeRFs) learn a mapping from position to volumetric density leveraging proposal network samplers. In contrast to the coarse-to-fine sampling approach with two NeRFs, this offers significant potential for acceleration using lower network capacity. Given that NeRFs utilize most of their network capacity to estimate radiance, they could store valuable density information in their parameters or their deep features.

To investigate this proposition, we take one step back and analyze large, trained ReLU-MLPs used in coarse-to-fine sampling. Building on our novel activation visualization method, we find that trained NeRFs, Mip-NeRFs and proposal network samplers map samples with high density to local minima along a ray in activation feature space. We show how these large MLPs can be accelerated by transforming intermediate activations to a weight estimate, without any modifications to the training protocol or the network architecture.

With our approach, we can reduce the computational requirements of trained NeRFs by up to 50% with only a slight hit in rendering quality. Extensive experimental evaluation on a variety of datasets and architectures demonstrates the effectiveness of our approach. Consequently, our methodology provides valuable insight into the inner workings of NeRFs.

Image Comparisons

Here, we provide comparisons of renderings from our method against the baseline.

Ours Nerfacto
Ours Nerfacto
Ours NeRF
Ours NeRF
Ours Mip-NeRF
Ours Mip-NeRF

Activation Visualization

Here, we show how different normalizations look for exemplary scenes.

0 1 2 3 4 5 6 7

Coarse Activations

Ours Ground-Truth

Fine Activations

Ours Ground-Truth

Deriving Densities from Activations

Our approach for density estimation is driven by analyzing real-world-activations. We observe that minima in activation feature space correspond to samples with high density. Consequently, our approach does not require fine-tuning and can be applied to various NeRF-based methods.

BibTeX

If you find our work useful, consider using a citation. 

@inproceedings{radl2024nerfinternals,
  author    = {Radl, Lukas and Kurz, Andreas and Steiner, Michael and Steinberger, Markus},
  title     = {{Analyzing the Internals of Neural Radiance Fields}},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  year      = {2024},
}