This is an automated archive made by the Lemmit Bot.

The original was posted on /r/machinelearning by /u/bnqj on 2024-03-29 13:04:29.

Yes, with VAPE - Vector Addition Positional Encoding.

I’ve been exploring a new approach to positional encoding that I’m calling VAPE - Vector Addition Positional Encoding.

The Method:

  • borrow some number of channels from queries and keys,
  • run a cumulative (prefix) sum across sequence length on these borrowed channels (add vectors together),
  • normalize - divide by the square root of the vector’s magnitude,
  • we now have position-aware channels,
  • so concatenate them back to queries and keys.

What’s intriguing is that this method can work effectively with just a single channel per head. Using a single channel means that we’re running prefix sum on scalars not vectors and the method still works.

VAPE features:

  • No Extra Parameters: VAPE introduces positional information without adding any new parameters to the model, preserving its simplicity and efficiency.
  • Performance: Early tests indicate that VAPE outperforms methods like RoPE in final perplexity.
  • Extrapolation: Early tests suggest that VAPE can extrapolate beyond training context length quite nicely, there is no explicit positional information added like in RoPE.
  • Compatibility with Flash Attention: It’s fully compatible with Flash Attention.
  • Efficiency: By leveraging just a small number of channels for positional encoding, VAPE maintains model efficiency.
  • Inference Speed: VAPE caches the last positional states for queries and keys - it’s a bit like SSM/RNN, you only need the last state to compute the next.

Seeking Your Insight:

  • What benchmarks or specific comparisons would best demonstrate VAPE’s value to you?
  • Do you know of any methods similar to VAPE?

Benchmarks:

I’ve run some early tests that look very promising for causal language modeling tasks, but I have quite limited resources for doing benchmarks, thus before I put any effort into them I think it’s better to ask the community how to go about it.