RHATNet: residual and hybrid attention-based transformer network for hyperspectral image classification

Abstract Hyperspectral images comprise many continuous bands in narrow wavelength. They have rich spectral and spatial features widely used in various applications, including vegetation monitoring, land-use change detection, medical diagnosis, and species identification. The existing CNN-based models extract high-dimensional spatial–spectral features. However, they often fail to accurately capture boundary and edge information, which is crucial for segregating neighboring land cover classes. Recently, vision transformers based methods have been used to provide global attention to the feature map. However, computation costs and the requirement for large training datasets make them less feasible for real-time applications. To overcome these challenges, we have proposed the residual hybrid attention transformer network (RHATNet). It consists of a 3D convolutional neural network for extracting the spatial–spectral information. Furthermore, a HResNeXt (modified version of the original ResNeXt) is designed to enhance the spatial features and to reduce computation costs using grouped convolution-based layers. Spectral and spatial features are combined via a cross-attention module to enable global attention and enhance the feature representation. Besides, a hybrid loss function is designed to address class imbalance and overfitting problems. The RHATNet performance is evaluated on different datasets, such as Botswana, IP, PU, and SV datasets and compared against recent existing techniques. The proposed model achieved average accuracies of 88.86%, 97.57%, 98.05%, and 97.16% on the Botswana, SV, IP, and PU datasets, respectively. Moreover, the qualitative results of the proposed method provide finer details of land cover compared to the state-of-the-art methods.