Sparse and Continuous Attention Mechanisms

This research paper proposes a novel approach to attention mechanisms in neural networks, extending them from discrete to continuous domains. This extension is based on the concept of deformed exponential families and Tsallis statistics, which allow for the creation of "sparse" families of distributions that can have zero tails. The paper introduces the use of continuous attention mechanisms, particularly with Gaussian and truncated paraboloid distributions, and demonstrates their effectiveness in various applications such as text classification, machine translation, and visual question…