Reconstruction Using the Invisible: Intuition from NIR and Metadata for Enhanced 3D Gaussian Splatting

Abstract

While 3D Gaussian Splatting (3DGS) has rapidly advanced, its application in agriculture remains underexplored. Agricultural scenes pose unique challenges for 3D reconstruction methods, notably uneven illumination, occlusions, and limited perspectives. To address these limitations, we introduce NTRPlant, a novel multimodal dataset encompassing Near-Infrared (NIR), RGB imagery, textual metadata, Depth, and LiDAR collected under varied indoor and outdoor lighting conditions. By integrating NIR data, our approach enhances robustness and extracts crucial botanical insights beyond visible spectra. Additionally, we leverage text-based metadata derived from vegetation indices, such as NDVI, NDWI, and chlorophyll index, significantly enriching the contextual understanding of complex agricultural environments. To fully exploit these modalities, we propose NIRSplat, an effective multimodal Gaussian splatting architecture employing a cross-attention mechanism combined with 3D point-based positional encoding, providing robust geometric priors. Comprehensive experiments demonstrate that NIRSplat outperforms existing state-of-the-art methods, including 3DGS and InstantSplat, highlighting its effectiveness in challenging agricultural scenarios.