Skip to content

Collection of Tools and Papers related to Adapters / Parameter-Efficient Transfer Learning/ Fine-Tuning

License

Notifications You must be signed in to change notification settings

calpt/awesome-adapter-resources

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Awesome Adapter Resources

This repository collects important tools and papers related to adapter methods for recent large pre-trained neural networks.

Adapters (aka Parameter-Efficient Transfer Learning (PETL) or Parameter-Efficient Fine-Tuning (PEFT) methods) include various parameter-efficient approaches of adapting large pre-trained models to new tasks.

Content

Why Adapters?

Large pre-trained (Transformer-based) models have become the foundation of various ML domains in recent years. While the most prevalent method of adapting these models to new tasks involves costly full fine-tuning of all model parameters, a series of parameter-efficient and lightweight alternatives, adapters, have been established in recent time.

Using adapters provides multiple benefits. They are ...

  • ... parameter-efficient, i.e. they only update a very small subset (e.g. under 1%) of a model's parameters.
  • ... modular, i.e. the updated parameters can be extracted and shared independently of the base model parameters
  • ... easy to share and easy to deploy at scale due to their small file sizes. E.g. requiring only ~3MB per task instead of ~500MB for sharing a full model.
  • ... often composable, i.e. can be stacked, fused or mixed to leverage their combined knowledge.
  • ... often on-par in terms of performance with full fine-tuning.

Frameworks and Tools

  • AdapterHub: A Framework for Adapting Transformers  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Jonas Pfeiffer, Andreas Rücklé, Clifton A. Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych (2020)

    TLDR AdaptersHub is proposed, a framework that allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages that enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios.

    [Paper PDF]  [Code]  [Website]  [Semantic Scholar]

  • Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Clifton A. Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Arne Engländer, Timo Imhof, Ivan Vuli'c, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer (2023)

    TLDR Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models and allows researchers and practitioners to leverage adapter modularity through composition blocks, enabling the design of complex adapter setups, is introduced.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • OpenDelta  GitHub Repo stars

    [Code]  [Website]

  • PEFT: State-of-the-art Parameter-Efficient Fine-Tuning  GitHub Repo stars

    [Code]

  • LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, R. Lee, Lidong Bing, Soujanya Poria (2023)

    TLDR LLM-Adapters is presented, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks, demonstrating that using adapter- based PEFT in smaller-scale LLMs with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs in zero-shot inference on both reasoning tasks.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Alpaca-LoRA  GitHub Repo stars

    [Code]

Surveys

  • Modular Deep Learning 

    arXiv.org

    Jonas Pfeiffer, Sebastian Ruder, Ivan Vulic, E. Ponti (2023)

    TLDR A survey of modular architectures is offered, providing a unified view over several threads of research that evolved independently in the scientific literature, and various additional purposes of modularity are explored, including scaling language models, causal inference, programme induction, and planning in reinforcement learning.

    [Paper PDF]  [Semantic Scholar]

  • Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning 

    arXiv.org

    Vladislav Lialin, Vijeta Deshpande, Anna Rumshisky (2023)

    TLDR A taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency and fine-tuning multibillion-scale language models is provided.

    [Paper PDF]  [Semantic Scholar]

  • PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques 

    arXiv.org

    Mohammed Sabry, Anya Belz (2023)

    TLDR A reference architecture is presented which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules.

    [Paper PDF]  [Semantic Scholar]

  • Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey 

    arXiv.org

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, Sai Qian Zhang (2024)

    TLDR This survey presents comprehensive studies of various PEFT algorithms, examining their performance and computational overhead, and overview of applications developed using different PEFT algorithms and discusses common techniques employed to mitigate computation costs for PEFT.

    [Paper PDF]  [Semantic Scholar]

Natural Language Processing

Methods

  • Parameter-Efficient Transfer Learning for NLP  GitHub Repo stars

    International Conference on Machine Learning

    N. Houlsby, A. Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, S. Gelly (2019)

    TLDR To demonstrate adapter's effectiveness, the recently proposed BERT Transformer model is transferred to 26 diverse text classification tasks, including the GLUE benchmark, and adapter attain near state-of-the-art performance, whilst adding only a few parameters per task.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters  GitHub Repo stars

    Findings

    Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, Ming Zhou (2020)

    TLDR K-Adapter is proposed, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion and captures richer factual and commonsense knowledge than RoBERTa.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Parameter-Efficient Transfer Learning with Diff Pruning  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Demi Guo, Alexander M. Rush, Yoon Kim (2020)

    TLDR Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s parameters per task and scales favorably in comparison to popular pruning approaches.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Prefix-Tuning: Optimizing Continuous Prompts for Generation  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Xiang Lisa Li, Percy Liang (2021)

    TLDR Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • The Power of Scale for Parameter-Efficient Prompt Tuning  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Brian Lester, Rami Al-Rfou, Noah Constant (2021)

    TLDR This work explores “prompt tuning,” a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “Prompt ensembling.”

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Compacter: Efficient Low-Rank Hypercomplex Adapter Layers  GitHub Repo stars

    Neural Information Processing Systems

    Joe Davison (2021)

    TLDR Compacter is proposed, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work, and accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • LoRA: Low-Rank Adaptation of Large Language Models  GitHub Repo stars

    International Conference on Learning Representations

    J. E. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Weizhu Chen (2021)

    TLDR Low-Rank Adaptation, or LoRA, is proposed, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, J. Henderson (2021)

    TLDR This paper shows that one can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer 

    Conference on Empirical Methods in Natural Language Processing

    Alan Ansell, E. Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavas, Ivan Vulic, A. Korhonen (2021)

    TLDR MAD-G is proposed, which contextually generates language adapters from language representations based on typological features and remains competitive with more expensive methods for language-specific adapter training across the board.

    [Paper PDF]  [Semantic Scholar]

  • BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg (2021)

    TLDR BitFit is introduced, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified, which shows that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning  GitHub Repo stars

    Neural Information Processing Systems

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel (2022)

    TLDR This paper rigorously compares few-shot ICL and PEFT and demonstrates that the latter offers better accuracy as well as dramatically lower computational costs, and introduces a new PEFT method called (IA)$^3$ that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning  GitHub Repo stars

    arXiv.org

    Han Zhou, Xingchen Wan, Ivan Vulic, A. Korhonen (2023)

    TLDR Inspired by advances in neural architecture search, AutoPEFT is proposed for automatic PEFT configuration selection and it is shown that AutoPEFT-discovered configurations significantly outperform existing PEFT methods and are on par or better than FFT without incurring substantial training efficiency costs.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning  GitHub Repo stars

    Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao (2023)

    TLDR The proposed AdaLoRA adaptively allocates the parameter budget among weight matrices according to their importance score, which allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • QLoRA: Efficient Finetuning of Quantized LLMs  GitHub Repo stars

    Neural Information Processing Systems

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (2023)

    TLDR QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA, and current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Composable Sparse Fine-Tuning for Cross-Lingual Transfer  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Alan Ansell, E. Ponti, A. Korhonen, Ivan Vulic (2021)

    TLDR This work introduces a new fine-tuning method that outperforms adapters in zero-shot cross-lingual transfer by a large margin in a series of multilingual benchmarks, including Universal Dependencies, MasakhaNER, and AmericasNLI.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • VeRA: Vector-based Random Matrix Adaptation 

    arXiv.org

    Dawid Jan Kopiczko, Tijmen Blankevoort, Yuki Markus Asano (2023)

    TLDR Vector-based Random Matrix Adaptation (VeRA) is presented, which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead.

    [Paper PDF]  [Semantic Scholar]

  • DoRA: Weight-Decomposed Low-Rank Adaptation  GitHub Repo stars

    arXiv.org

    Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen (2024)

    TLDR Weight-Decomposed LowRank Adaptation (DoRA) is proposed, which decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • ReFT: Representation Finetuning for Language Models  GitHub Repo stars

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Daniel Jurafsky, Christopher D. Manning, Christopher Potts (2024)

    TLDR A strong instance of the ReFT family is defined, Low-rank Linear Subspace ReFT (LoReFT), which is a drop-in replacement for existing PEFTs and learns interventions that are 10x-50x more parameter-efficient than prior state-of-the-art PEFTs.

    [Paper PDF]  [Code]  [Semantic Scholar]

Composition Methods

  • MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder (2020)

    TLDR MAD-X is proposed, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations and introduces a novel invertible adapter architecture and a strong baseline method for adapting a pretrained multilingual model to a new language.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • AdapterFusion: Non-Destructive Task Composition for Transfer Learning  GitHub Repo stars

    Conference of the European Chapter of the Association for Computational Linguistics

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych (2020)

    TLDR This work proposes AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks by separating the two stages, i.e., knowledge extraction and knowledge composition, so that the classifier can effectively exploit the representations learned frommultiple tasks in a non-destructive manner.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Towards a Unified View of Parameter-Efficient Transfer Learning  GitHub Repo stars

    International Conference on Learning Representations

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig (2021)

    TLDR This paper re-frame state-of-the-art parameter-efficient transfer learning methods as modifications to specific hidden states in pre-trained models, and defines a set of design dimensions along which different methods vary, achieving comparable results to fine-tuning all parameters on all four tasks.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Jianfeng Gao (2022)

    TLDR AdaMix is proposed as a general PEFT method that tunes a mixture of adaptation modules – given the underlyingPEFT method of choice – introduced in each Transformer layer while keeping most of the PLM weights frozen, and outperforms SOTA parameter-efficient fine-tuning and full model fine- Tuning for both NLU and NLG tasks.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Composing Parameter-Efficient Modules with Arithmetic Operations  GitHub Repo stars

    Neural Information Processing Systems

    Jinghan Zhang, Shiqi Chen, Junteng Liu, Junxian He (2023)

    TLDR This paper proposes to compose parameter-efficient modules through linear arithmetic operations in the weight space, thereby integrating different module capabilities and extends this approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models 

    Findings

    Alexandra Chronopoulou, Matthew E. Peters, Alexander M. Fraser, Jesse Dodge (2023)

    TLDR This paper introduces AdapterSoup, an approach that performs weight-space averaging of adapters trained on different domains, and explores various approaches for choosing which adapters to combine, such as text clustering and semantic similarity.

    [Paper PDF]  [Semantic Scholar]

  • Combining Modular Skills in Multitask Learning  GitHub Repo stars

    arXiv.org

    E. Ponti, Alessandro Sordoni, Siva Reddy (2022)

    TLDR It is found that the modular design of a network significantly increases sample efficiency in reinforcement learning and few-shot generalisation in supervised learning, compared to baselines with fully shared, task-specific, or conditionally generated parameters where knowledge is entangled across tasks.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Multi-Head Adapter Routing for Data-Efficient Fine-Tuning  GitHub Repo stars

    arXiv.org

    Lucas Caccia, E. Ponti, Lu Liu, Matheus Pereira, Nicolas Le Roux, Alessandro Sordoni (2022)

    TLDR This paper investigates to what extent the ability to control which adapters are active for each task leads to sample-efficient generalization and proposes less expressive variants where the authors perform weighted averaging of the adapters before few-shot adaptation (Poly - µ ) instead of learning a routing function.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning  GitHub Repo stars

    arXiv.org

    Ted Zadouri, A. Ustun, Arash Ahmadian, Beyza Ermics, Acyr Locatelli, Sara Hooker (2023)

    TLDR This paper proposes extremely parameter-efficient MoE by uniquely combining MoE architecture with lightweight experts and is on par with full fine-tuning by only updating the lightweight experts -- less than 1% of an 11B parameters model.

    [Paper PDF]  [Code]  [Semantic Scholar]

Analysis and Evaluation

  • Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers  GitHub Repo stars

    Workshop on Knowledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out

    Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, N. Rozanov, Goran Glavavs (2020)

    TLDR A deeper analysis reveals that the adapter-based models substantially outperform BERT on inference tasks that require the type of conceptual knowledge explicitly present in ConceptNet and its corresponding Open Mind Common Sense corpus.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation 

    Annual Meeting of the Association for Computational Linguistics

    Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si (2021)

    TLDR It is demonstrated that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

    [Paper PDF]  [Semantic Scholar]

  • Robust Transfer Learning with Pretrained Language Models through Adapters 

    Annual Meeting of the Association for Computational Linguistics

    Wenjuan Han, Bo Pang, Y. Wu (2021)

    TLDR This work inserts small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, leading to improved stability and adversarial robustness in transfer learning to various downstream tasks.

    [Paper PDF]  [Semantic Scholar]

  • AdapterDrop: On the Efficiency of Adapters in Transformers 

    Conference on Empirical Methods in Natural Language Processing

    Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych (2020)

    TLDR This paper proposes AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions and can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances.

    [Paper PDF]  [Semantic Scholar]

  • What to Pre-Train on? Efficient Intermediate Task Selection  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Clifton A. Poth, Jonas Pfeiffer, Andreas Ruckl'e, Iryna Gurevych (2021)

    TLDR This work provides a comprehensive comparison of different methods for efficiently identifying beneficial tasks for intermediate transfer learning, focusing on parameter and computationally efficient adapter settings, highlight different data-availability scenarios, and provide expense estimates for each method.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer 

    arXiv.org

    M. Vidoni, Ivan Vulic, Goran Glavas (2020)

    TLDR This work proposes orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer that are trained to encode language- and task-specific information that is complementary to the knowledge already stored in the pretrained transformer's parameters.

    [Paper PDF]  [Semantic Scholar]

  • P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Xiao Liu, Kaixuan Ji, Yicheng Fu, W. Tam, Zhengxiao Du, Zhilin Yang, Jie Tang (2022)

    TLDR The method P-Tuning v2 is an implementation of Deep Prompt Tuning (CITATION) optimized and adapted for NLU and can serve as an alternative to finetuning and a strong baseline for future research.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models 

    arXiv.org

    Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, Maosong Sun (2022)

    TLDR The theoretical principles underlying the effectiveness of delta tuning are discussed and frameworks to interpret delta tuning from the perspective of optimization and optimal control are proposed, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches.

    [Paper PDF]  [Semantic Scholar]

  • UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, Madian Khabsa (2021)

    TLDR A unified framework, UniPELT, is proposed, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism, indicating that a mixture of multiple P ELT methods may be inherently more effective than single methods.

    [Paper PDF]  [Code]  [Semantic Scholar]

Applications

  • Simple, Scalable Adaptation for Neural Machine Translation 

    Conference on Empirical Methods in Natural Language Processing

    Ankur Bapna, N. Arivazhagan, Orhan Firat (2019)

    TLDR The proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model, which adapt the model to multiple individual tasks simultaneously, paving the way towards universal machine translation.

    [Paper PDF]  [Semantic Scholar]

  • Monolingual Adapters for Zero-Shot Neural Machine Translation 

    Conference on Empirical Methods in Natural Language Processing

    Jerin Philip, Alexandre Berard, Matthias Gallé, L. Besacier (2020)

    TLDR A novel adapter layer formalism for adapting multilingual models is proposed, which is more parameter-efficient than existing adapter layers while obtaining as good or better performance.

    [Paper PDF]  [Semantic Scholar]

  • UDapter: Language Adaptation for Truly Universal Dependency Parsing  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    A. Ustun, Arianna Bisazza, G. Bouma, Gertjan van Noord (2020)

    TLDR A novel multilingual task adaptation approach based on recent work in parameter-efficient transfer learning, which allows for an easy but effective integration of existing linguistic typology features into the parsing network, and consistently outperforms strong monolingual and multilingual baselines on both high-resource and low-resource languages.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Single-dataset Experts for Multi-dataset Question Answering  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Dan Friedman, Ben Dodge, Danqi Chen (2021)

    TLDR This work trains a collection of lightweight, dataset-specific adapter modules that share an underlying Transformer model, and finds that these Multi-Adapter Dataset Experts (MADE) outperform all the authors' baselines in terms of in-distribution accuracy, and simple methods based on parameter-averaging lead to better zero-shot generalization and few-shot transfer performance.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • UNKs Everywhere: Adapting Multilingual Language Models to New Scripts 

    Conference on Empirical Methods in Natural Language Processing

    Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder (2020)

    TLDR This work proposes a series of novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts and demonstrates that they can yield improvements for low- resource languages written in scripts covered by the pretrained model.

    [Paper PDF]  [Semantic Scholar]

  • Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters 

    Conference on Machine Translation

    Asa Cooper Stickland, Alexandre Berard, Vassilina Nikoulina (2021)

    TLDR This work study the compositionality of language and domain adapters in the context of Machine Translation, and aims to study parameter-efficient adaptation to multiple domains and languages simultaneously and cross-lingual transfer in domains where parallel data is unavailable for certain language pairs.

    [Paper PDF]  [Semantic Scholar]

  • Multilingual Unsupervised Neural Machine Translation with Denoising Adapters 

    Conference on Empirical Methods in Natural Language Processing

    A. Ustun, Alexandre Berard, L. Besacier, Matthias Gallé (2021)

    TLDR This paper proposes to use _denoising adapters_, adapter layers with a denoising objective, on top of pre-trained mBART-50, and shows that the resulting translations are on-par with back-translating as measured by BLEU, and furthermore it allows adding unseen languages incrementally.

    [Paper PDF]  [Semantic Scholar]

  • Efficient Test Time Adapter Ensembling for Low-resource Language Varieties  GitHub Repo stars

    Conference on Empirical Methods in Natural Language Processing

    Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, Graham Neubig (2021)

    TLDR This paper finds that ensembling multiple existing language adapters makes the fine-tuned model significantly more robust to other language varieties not included in these adapters, and proposes EMA, a method that optimizes the ensemble weights of the pretrained language adapters for each test sentence by minimizing the entropy of its predictions.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention  GitHub Repo stars

    arXiv.org

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, Y. Qiao (2023)

    TLDR A zero-initialized attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge on traditional vision and language tasks, demonstrating the superior generalization capacity of the approach.

    [Paper PDF]  [Code]  [Semantic Scholar]

Serving

  • Punica: Multi-Tenant LoRA Serving  GitHub Repo stars

    arXiv.org

    Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy University of Washington, Duke University (2023)

    TLDR Punica is a system to serve multiple LoRA models in a shared GPU cluster that contains a new CUDA kernel design that allows batching of GPU operations for differentLoRA models, significantly enhancing GPU efficiency in terms of both memory and computation.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • S-LoRA: Serving Thousands of Concurrent LoRA Adapters  GitHub Repo stars

    arXiv.org

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica (2023)

    TLDR S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine- Tuning services.

    [Paper PDF]  [Code]  [Semantic Scholar]

Computer Vision

Methods

  • Learning multiple visual domains with residual adapters  GitHub Repo stars

    Neural Information Processing Systems

    Sylvestre-Alvise Rebuffi, Hakan Bilen, A. Vedaldi (2017)

    TLDR This paper develops a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains and introduces the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very differentVisual domains and measures their ability to recognize well uniformly.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Efficient Parametrization of Multi-domain Deep Neural Networks 

    2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sylvestre-Alvise Rebuffi, Hakan Bilen, A. Vedaldi (2018)

    TLDR This paper proposes to consider universal parametric families of neural networks, which still contain specialized problem-specific models, but differing only by a small number of parameters, and shows that these universal parametrization are very effective for transfer learning, where they outperform traditional fine-tuning techniques.

    [Paper PDF]  [Semantic Scholar]

  • Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets 

    arXiv.org

    Hao Chen, R. Tao, Han Zhang, Yidong Wang, Weirong Ye, Jindong Wang, Guosheng Hu, M. Savvides (2022)

    TLDR Conv-Adapter, a PET module designed for ConvNets, is light-weight, domain-transferable, and architecture-agnostic with generalized performance on different tasks with comparable or surpasses the performance of full fine-tuning on 23 classification tasks of various domains.

    [Paper PDF]  [Semantic Scholar]

  • AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition  GitHub Repo stars

    Neural Information Processing Systems

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo (2022)

    TLDR AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks.

    [Paper PDF]  [Code]  [Semantic Scholar]

Audio Processing

Applications

  • Lightweight Adapter Tuning for Multilingual Speech Translation  GitHub Repo stars

    Annual Meeting of the Association for Computational Linguistics

    Hang Le, J. Pino, Changhan Wang, Jiatao Gu, D. Schwab, L. Besacier (2021)

    TLDR This paper proposes a comprehensive analysis of adapters for multilingual speech translation (ST) and shows that adapters can be used to efficiently specialize ST to specific language pairs with a low extra cost in terms of parameters.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition 

    IEEE International Conference on Acoustics, Speech, and Signal Processing

    Bethan Thomas, Samuel Kessler, S. Karout (2022)

    TLDR Adapters are applied to wav2vec 2.0 to reduce the number of parameters required for downstream ASR tasks, and increase scalability of the model to multiple tasks or languages, and support the theory that higher pre-trained layers encode more phonemic information.

    [Paper PDF]  [Semantic Scholar]

Multi-Modal

Methods

  • VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks  GitHub Repo stars

    Computer Vision and Pattern Recognition

    Yi-Lin Sung, Jaemin Cho, Mohit Bansal (2021)

    TLDR The results demonstrate that training the adapter with the weight-sharing technique can match the performance of fine-tuning the entire model, and enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning  GitHub Repo stars

    Neural Information Processing Systems

    Yi-Lin Sung, Jaemin Cho, Mohit Bansal (2022)

    TLDR LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections, and achieves higher accuracy than Adapter and LoRA in a low-memory regime.

    [Paper PDF]  [Code]  [Semantic Scholar]

  • Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference 

    Neural Information Processing Systems

    Tao Lei, Junwen Bai, Siddhartha Brahma, J. Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang (2023)

    TLDR This work proposes Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency and achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.

    [Paper PDF]  [Semantic Scholar]

  • VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control  GitHub Repo stars

    IEEE International Conference on Computer Vision

    Zi-Yuan Hu, Yanyang Li, M. Lyu, Liwei Wang (2023)

    TLDR A Vision-and-Language Parameter-Efficient Tuning (VL-PET) framework to impose effective control over modular modifications via a novel granularity-controlled mechanism and a variety of model-agnostic VL-PET modules can be instantiated from this framework for better efficiency and effective-ness trade-offs is proposed.

    [Paper PDF]  [Code]  [Website]  [Semantic Scholar]

Contributing

Contributions of new awesome adapter-related resources are very welcome! Before contributing, make sure to read this repository's contributing guide.

Acknowledgments

Paper metadata is partially retrieved via Semantic Scholar's API. Paper TLDRs are provided by Semantic Scholar's TLDR feature.