Sparse Attention with Linear Units - Rectified Linear Attention (ReLA)

This research paper proposes a new method for achieving sparsity in attention models, called Rectified Linear Attention (ReLA). ReLA replaces the softmax function with a ReLU activation, leading to sparsity by dropping negative attention scores. To stabilise gradient training, layer normalisation with a specialized initialization or gating mechanism is used. Experiments on five machine translation tasks show that ReLA achieves comparable translation performance to softmax-based models, while being more efficient than other sparse attention mechanisms. The authors also conduct in-depth…