This is an automated archive made by the Lemmit Bot.
The original was posted on /r/machinelearning by /u/AccomplishedCode4689 on 2025-06-12 13:43:22+00:00.
We introduce ABBA, a new architecture for Parameter-Efficient Fine-Tuning (PEFT) that significantly outperforms LoRA and all its major variants across a broad range of benchmarks, all under the same parameter budget.
Most PEFT methods, including LoRA, represent weight updates using a low-rank decomposition added to the frozen model weights. While effective, this structure can limit the expressivity of the update, especially at low rank.
ABBA takes a fundamentally different approach:
- Reparameterizes the update as a Hadamard product of two independently learned low-rank matrices
- Decouples the two components of the update from the base model, allowing them to be optimized freely
- Enables significantly higher expressivity and improved performance under the same parameter budget
📈 Empirical Results
ABBA consistently beats state-of-the-art LoRA-based methods like HiRA, DoRA, and LoRA-Pro across four open-source LLMs: Mistral-7B, Gemma-2 9B, LLaMA-3.2 1B, and LLaMA-3.2 3B, on a suite of commonsense and arithmetic reasoning benchmarks. In several cases, ABBA even outperforms full fine-tuning.
📄 Paper: https://arxiv.org/abs/2505.14238
💻 Code: https://github.com/CERT-Lab/abba
We’d love to hear your thoughts, whether you’re working on PEFT methods, fine-tuning, or anything related to making LLMs more adaptable and efficient. We’re happy to answer questions, discuss implementation details, or just hear how this fits into your work.