Sklab Youcef, Ariouat H., Chenin Eric, Prifti Edi, Zucker Jean-Daniel. (2025). SIM-Net : a multimodal fusion network using inferred 3D object shape point clouds from RGB images for 2D classification. IET Computer Vision, 19 (1), p. e70036 [18 p.]. ISSN 1751-9632.
Titre du document
SIM-Net : a multimodal fusion network using inferred 3D object shape point clouds from RGB images for 2D classification
Sklab Youcef, Ariouat H., Chenin Eric, Prifti Edi, Zucker Jean-Daniel
Source
IET Computer Vision, 2025,
19 (1), p. e70036 [18 p.] ISSN 1751-9632
We introduce the shape-image multimodal network (SIM-Net), a novel 2D image classification architecture that integrates 3D point cloud representations inferred directly from RGB images. Our key contribution lies in a pixel-to-point transformation that converts 2D object masks into 3D point clouds, enabling the fusion of texture-based and geometric features for enhanced classification performance. SIM-Net is particularly well-suited for the classification of digitised herbarium specimens-a task made challenging by heterogeneous backgrounds, nonplant elements, and occlusions that compromise conventional image-based models. To address these issues, SIM-Net employs a segmentation-based preprocessing step to extract object masks prior to 3D point cloud generation. The architecture comprises a CNN encoder for 2D image features and a PointNet-based encoder for geometric features, which are fused into a unified latent space. Experimental evaluations on herbarium datasets demonstrate that SIM-Net consistently outperforms ResNet101, achieving gains of up to 9.9% in accuracy and 12.3% in F-score. It also surpasses several transformer-based state-of-the-art architectures, highlighting the benefits of incorporating 3D structural reasoning into 2D image classification tasks.
Plan de classement
Sciences du monde végétal [076]
;
Informatique [122]
