Neural Radiance Field (NeRF) Based Sensor Fusion in Continuous Time

In their seminal work, Mildenhall et al. (3), proposed to represent the scene by the means of a neural radiance field. While the idea of representing the scene as a neural network was first used to render novel views from a sparse set of images of objects, recent works building on top of this idea have shown remarkable results on large scale scenes (5, 6). Rematas et al. (4) proposed to extend the NeRF with a method to include not only image data but make use of LiDAR data in the model estimation. While all the approaches so far assume known and exact poses, (2) proposed to include also the camera poses as variables during the training of the NeRF, allowing to deal with imperfect initial poses. In this project we want to investigate the use of a continuous trajectory representation to parametrize the trajectory of our sensor suite. Doing so allows to use information of various sensors which are not necessarily synchronized. By utilizing the concept of (2), we want to utilize NeRF-​based map representation to perform an offline sensor fusion of potentially unsynchronized sensors.

(1) D. Hug, P. Bänninger, I. Alzugaray, and M. Chli. Continuous-​Time Stereo-​Inertial Odometry. IEEE Robotics and Automation Letters, 2022. (2) C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey. BARF: Bundle-​Adjusting Neural Radiance Fields. Proceedings of the International Conference on Computer Vision (ICCV), 2021. (3) B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, and R. Ramamoorthi. NeRF: Representing scenes as neural radiance fieds for view synthesis. Proceedings of the European Conference on Computer Vision (ECCV), 2020. (4) K. Rematas, A. Liu, P. Srinivasan, J. Barron, A. Tagliasacchi, T. Funkhouser, and V. Ferrari. Urban Radiance Fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. (5) M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar. Block-​NeRF: Scalable Large Scene Neural View Synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. (6) H. Turki, D. Ramanan, and M. Satynarayanan. Mega-​NeRF: Scalable Construction of Large-​Scale NeRFs for Virtual Fly-​Throughs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

• Strong analytical skills and independent work-​style
• Solid programming skills (C++/Python) are mandatory
• Familiarity with common geometrical concepts (e.g. 3D rotations etc.)
• Experience with machine learning and data processing

Marco Karrer (marco.karrer@mavt.ethz.ch) David Hug (david.hug@mavt.ethz.ch) Ruben Mascaro (rmascaro@ethz.ch)