STGAN: Swin Transformer-based GAN to achieve remote sensing image super-resolution reconstruction
Arxiv 2024
Wei Huo1
Xiaodan Zhang1
Shaojie You1
Yongkun Zhang1
Qiyuan Zhang1
Naihao Hu1
1Qinghai University
3Affiliated Hospital of Qinghai University
[arXiv📝]
[code⚙️]

Abstract

Super-resolution (SR) of remote sensing images is essential to compensate for missing information in the original high-resolution (HR) images. Single-image superresolution (SISR) technique aims to recover high-resolution images from low-resolution (LR) images. However, traditional SISR methods often result in blurred and unclear images due to the loss of high-frequency details in LR images at high magnifications. In this paper, a super-segmental reconstruction model STGAN for remote sensing images is proposed, which fuses the Generative Adversarial Networks (GANs) and self-attention mechanism based on the Reference Super Resolution method (RefSR). The core module of the model consists of multiple CNN-Swin Transformer blocks (MCST), each of which consists of a CNN layer and a specific modified Swin Transformer, constituting the feature extraction channel. In image hypersegmentation reconstruction, the optimized and improved correlation attention block (RAM-V) uses feature maps and gradient maps to improve the robustness of the model under different scenarios (such as land cover change). The experimental results show that the STGAN model proposed in this paper exhibits the best image data perception quality results with the best performance of LPIPS and PI metrics in the test set under RRSSRD public datasets. In the experimental test set, the PSNR reaches 31.4151, the SSIM is 0.8408, and the performance on the RMSE and SAM metrics is excellent, which demonstrate the model’s superior image reconstruction details in super-resolution reconstruction and highlighting the great potential of RefSR’s application to the task of super-scalar processing of remotely sensed images..

Overall pipeline of the model architecture of HES-UNet and its modules.

Results on HE Lesion Segmentation

Performance comparisons of our HES-UNet and other existing segmentation models. Bold indicates optimal performance.

Ablation study of the different modules in the HES-UNet. Bold indicates optimal performance.

HE Lesion Segmentation Demos

The following demos illustrate a comparison between HES-UNet and other existing segmentation models on our dataset. In the visualization, true positives (TP) are highlighted in green, false positives (FP) in blue, and false negatives (FN) in hotpink.

Cystic Echinococcosis Lesion Segmentation Sample

Alveolar Echinococcosis Lesion Segmentation Sample


Acknowledgements

Website template was borrowed from Colorful Image Colorization and Nerfies; the code can be found here and here. Thank you (.❛ ᴗ ❛.).