New Paper Alert on Multi-sensor data fusion ❗️

Multi-sensor data fusion using deep learning for bulky waste image classification By Ahmed J. Afifi

Images from all sensors of wood partly covered with metal. From left to right: RGB, hyperspectral NIR, thermography, and terahertz image. To visualize the data, 3 out of 224 NIR channels are shown, whereas for the terahertz data cube, a maximum aggregation over the depth channel is given for visualization (from the paper).

In the realm of computer vision, deep learning techniques have become a go-to solution for tackling a wide array of challenges, such as object recognition, classification, and segmentation based on RGB images. As technology has advanced, we now have access to an array of sensors designed for specific industries. These sensors collect specialized datasets that often feature various modalities. These modalities manifest as differences in channel numbers and pixel values within the images, each requiring unique interpretation.

Leveraging deep learning to achieve optimal results with such multimodal data is no small feat. To boost the performance of classification tasks in this complex environment, one promising approach involves the use of data fusion techniques. Data fusion is a strategy aimed at harnessing all available information from these diverse sensors and melding them together to yield the best possible results.

In this article, we delve into the exploration of early fusion, intermediate fusion, and late fusion using deep learning models, with a specific focus on classifying bulky waste images. Our study employs a multimodal dataset encompassing RGB, hyperspectral near-infrared (NIR), Thermography, and Terahertz images of bulky waste. The findings from our research underscore the significant advantages of multimodal sensor fusion, showcasing its ability to improve classification accuracy when compared to a single-sensor approach, specifically for the dataset we utilized. Among the fusion strategies, late fusion emerged as the top performer, boasting an impressive accuracy rate of 0.921 on our test data, surpassing both intermediate and early fusion techniques.

Enjoy reading the publication here: Manuel Bihler, Lukas Roming, Yifan Jiang, Ahmed J. Afifi, Jochen Aderhold, Dovilė Čibiraitė-Lukenskienė, Sandra Lorenz, Richard Gloaguen, Robin Gruna, and Michael Heizmann "Multi-sensor data fusion using deep learning for bulky waste image classification", Proc. SPIE 12623, Automated Visual Inspection and Machine Vision V, 126230B (11 August 2023); https://doi.org/10.1117/12.2673838