Improving Counting Accuracy of Postdisaster Visual Question Answering for Remote Sensing

In post-disaster damage assessment, visual question answering (VQA) systems are essential in identifying the severity and scope of damage. However, counting-related tasks, such as determining the number of vehicles and flooded buildings, remain a significant challenge for current deep learning models. To address this issue, we propose DeVANet (DeBERTa Vision Attention Network), a novel architecture aimed at enhancing counting accuracy in VQA for post-disaster scenarios. We leverage DeBERTa for language modeling and introduce an innovative image embedding module, where local-global attention guides Vision Mamba features to achieve precise extraction of both small and large objects. Our fusion mechanism employs self-attention for both text and image data, followed by bidirectional cross-attention and co-attention to enhance multimodal integration. We tackle VQA as both a classification and regression problem by employing separate MLPs for each task: one handling discrete class predictions and the other generating continuous values for counting tasks. A joint loss function, combining weighted cross-entropy and negative binomial loss, ensures optimized performance across both tasks. Extensive experiments on the FloodNet and RescueNet datasets demonstrate that DeVANet achieves significant improvements in counting accuracy and overall VQA performance compared to state-of-the-art works, supported by detailed ablation studies that validate the effectiveness of each component in the architecture.