Hierarchical Multimodal Adaptive Fusion (HMAF)‎ Network for Prediction of RGB-D Saliency

Abstract EN

Visual saliency prediction for RGB-D images is more challenging than that for their RGB counterparts.

Additionally, very few investigations have been undertaken concerning RGB-D-saliency prediction.

The proposed study presents a method based on a hierarchical multimodal adaptive fusion (HMAF) network to facilitate end-to-end prediction of RGB-D saliency.

In the proposed method, hierarchical (multilevel) multimodal features are first extracted from an RGB image and depth map using a VGG-16-based two-stream network.

Subsequently, the most significant hierarchical features of the said RGB image and depth map are predicted using three two-input attention modules.

Furthermore, adaptive fusion of saliencies concerning the above-mentioned fused saliency features of different levels (hierarchical fusion saliency features) can be accomplished using a three-input attention module to facilitate high-accuracy RGB-D visual saliency prediction.

Comparisons based on the application of the proposed HMAF-based approach against those of other state-of-the-art techniques on two challenging RGB-D datasets demonstrate that the proposed method outperforms other competing approaches consistently by a considerable margin.