Transformer Models for Processing Biological Signal

Abstract

The transformer neural network architecture is a deep learning model, that has been developed recently and as such it’s potential is still being investigated. It is a powerful model due to the their self-attention mechanism that finds use in several domains, but our focus is on transformers used for biological signals processing. Various hybrid model architectures suitable for this type of task are considered in this study: the basic transformer, temporal fusion transformer, time series transformer, convolutional vision transformer and informer. A brief description of the architecture is given. The reasons why they are appropriate for processing biological signals, what makes them unique, along with their strengths and weaknesses, are discussed. Finally, a literature review is made involving actual studies that use these model types for biosignal processing.