Neural Radiance Fields (NeRF)

Neural radiance fields (NeRF) is a method for reconstructing a three-dimensional representation of a scene from a set of two-dimensional images. Introduced in 2020, it trains a small neural network to store the geometry and appearance of a scene as a continuous field, and then uses that network to render photorealistic images of the scene from viewpoints that were never photographed, a task known as novel view synthesis. NeRF prompted a wave of research in neural rendering and 3D computer vision and established the family of implicit neural scene representations, sometimes called neural fields.

How it works

A NeRF represents a scene as a continuous function of five variables: the three spatial coordinates of a point and the two angles describing a viewing direction. This function is implemented as a multilayer perceptron, a simple fully connected neural network. Given a point in space and a direction from which it is viewed, the network outputs a colour and a density (opacity) value. Density captures how much light the point blocks, and colour is made view-dependent so that the network can represent effects such as reflections and specular highlights that change with viewing angle.

To turn this function into an image, NeRF uses differentiable volume rendering. For each pixel of a target image, a ray is cast from the camera through the scene, points are sampled along the ray, and the network is queried at each point. The colours and densities along the ray are accumulated, closer and more opaque points contributing more, to produce the final pixel colour. Because this rendering process is differentiable, the whole system can be trained by gradient descent: the network is optimised so that the images it renders match the original photographs. Once trained, the network can be queried from any new camera position to synthesise a fresh view.

A notable property of NeRF is that it is self-supervised in the sense that it requires only images and their known camera poses. It does not need explicit 3D models, depth sensors, or manual labelling, which lowers the barrier to capturing detailed 3D scenes.

Strengths, limitations, and successors

NeRF produces strikingly realistic novel views and captures fine detail and view-dependent effects that mesh-based methods struggle with. Its original limitations were practical: training a NeRF for a single scene could take hours, rendering was slow, and each model represented only one scene. A large research effort addressed these issues, producing variants that train and render far faster, handle unbounded outdoor scenes, work from few input images, and represent dynamic scenes that change over time.

In 2023, 3D Gaussian splatting offered an alternative that reached comparable quality with real-time rendering by using an explicit set of primitives rather than a neural network, and it has since become a popular counterpart. NeRF and Gaussian splatting are now often discussed together as the two dominant approaches to radiance-field reconstruction, each with different trade-offs in speed, editability, and representation.

Applications of NeRF and its descendants include virtual and augmented reality, robotics and autonomous navigation, urban mapping, drone surveying, cultural heritage capture, and visual effects.

Neural radiance fields and related radiance-field methods support several priorities relevant to Malaysia. Heritage documentation is a natural application: the UNESCO World Heritage cities of George Town and Melaka contain intricate colonial and Peranakan architecture that photograph-based 3D reconstruction can preserve digitally, complementing conservation work by bodies such as Think City and university heritage programmes.

In the property, construction, and surveying sectors, NeRF-based capture can generate realistic 3D models of buildings and sites for marketing, planning, and facilities management. Malaysian developers including Sunway, SP Setia, and IOI Properties, along with initiatives under the Construction Industry Development Board (CIDB) to adopt Building Information Modelling, can incorporate radiance-field reconstruction alongside conventional surveying. Drone-based NeRF is relevant to plantation and land management, including monitoring of oil-palm estates operated by companies such as Sime Darby Plantation and FGV, where aerial imagery can be turned into detailed 3D terrain models.

Research capacity is growing at universities such as Multimedia University in Cyberjaya, Universiti Sains Malaysia, and Universiti Teknologi Malaysia, which teach computer vision and graphics. The compute required to train and serve neural rendering models connects to Malaysia's expanding data-centre sector in Johor and Cyberjaya, aligning with the digital-infrastructure aims of the MyDigital blueprint and MDEC.

Tourism promotion by Tourism Malaysia and private operators can also draw on immersive 3D captures of destinations to support virtual previews, contributing to a sector that is a significant part of the national economy.

1. Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV.
2. Gao, K., et al. (2022). NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review. arXiv.
3. Wikipedia contributors. (2025). Neural radiance field. Wikipedia.