Hybrid CNN-transformer demosaicing for bioinspired single-chip color-near-infrared fluorescence imaging in oncologic surgery

Abstract

Significance: Single-chip multispectral imaging sensors with vertically stacked photodiodes and pixelated spectral filters enable advanced, real-time visualization for image-guided cancer surgery. However, their design inherently reduces spatial resolution. We present a convolutional neural network (CNN)-transformer demosaicing algorithm, validated on both clinical and preclinical datasets that effectively doubles spatial resolution and improves image quality-substantially enhancing intraoperative cancer visualization.

Aim: We present a CNN-transformer-based demosaicing approach specifically optimized for reconstructing high-resolution color and NIR images acquired by a hexachromatic imaging sensor. Approach: A hybrid CNN-transformer demosaicing model was developed and trained on color-image datasets, then rigorously evaluated on color and NIR images to demonstrate superior reconstruction quality compared with conventional bilinear interpolation and residual CNN methods. Results: Our CNN-transformer demosaicing method achieves an average mean squared error (MSE) reduction of ∼85% for color images and 76% for NIR images and improves structural dissimilarity by roughly 72% and 79%, respectively, compared with state-of-the-art CNN-based demosaicing algorithms in preclinical datasets. In clinical datasets, our approach similarly demonstrates significant reductions in MSE and structural dissimilarity, substantially outperforming existing CNN-based methods, particularly in reconstructing high-frequency image details. Conclusions: We demonstrate improvements in spatial resolution and image fidelity for color and NIR images obtained from hexachromatic imaging sensors, achieved by integrating convolutional neural networks with transformer architectures. Given recent advances in GPU computing, our CNN-transformer approach offers a practical, real-time solution for enhanced multispectral imaging during cancer surgery.