Adapters v1.2.0

Blog post: https://adapterhub.ml/blog/2025/05/adapters-for-any-transformer

This version is built for Hugging Face Transformers v4.51.x.

New

Adapter Model Plugin Interface (@calpt via #738; @lenglaender via #797)

The new adapter model interface makes it easy to plug most adapter features into any new or custom Transformer model. Check out our release blog post for details. Also see https://docs.adapterhub.ml/plugin_interface.html.

Multi-Task composition with MTL-LoRA (@FrLdy via #792)

MTL-LoRA (Yang et al., 2024) is a new adapter composition method leveraging LoRA for multi-task learning. See https://docs.adapterhub.ml/multi_task_methods.html#mtl-lora.

VeRA - parameter-efficient LoRA variant (@julian-fong via #763)

VeRA (Kopiczko et al., 2024) is a LoRA adapter variant that requires even less trainable parameters. See https://docs.adapterhub.ml/methods.html#vera.

New Models (via new interface)

A couple of new models are supported out-of-the-box via the new adapter model plugin interface:

• Gemma 2, Gemma 3
• ModernBERT
• Phi 1, Phi 2
• Qwen 2, Qwen 2.5, Qwen 3

More

• New init_weights_seed adapter config attribute to initialize adapters with identical weights (@TimoImhof via #786)
• Support defining custom forward method args via ForwardContext (@calpt via #789)

Changed

• Upgrade supported Transformers version (@calpt via #799; @TimoImhof via #805)

Adapters v1.1.1

This version is built for Hugging Face Transformers v4.48.x.

Changes

• Upgrade supported Transformers version to 4.48 (@TimoImhof via #782)

Fixed

• Fix BatchSplit for PrefixTuning (@FrLdy via #750)

Adapters v1.1.0

This version is built for Hugging Face Transformers v4.47.x.

New

Add AdapterPlus adapters (@julian-fong via #746, #775):

AdapterPlus (Steitz & Roth, 2024) is a new bottleneck adapter variant optimized for vision transformers. Check out our notebook for training AdapterPlus adapters for ViT models.

Easy saving, loading and pushing of full adapter compositions to the Hub (@calpt via #771):

The newly added save_adapter_setup(), load_adapter_setup() and push_adapter_setup_to_hub() methods allow saving, loading and uploading complex adapter compositions - including AdapterFusion setups - with one line of code. Read our documentation for more.

Enabling full gradient checkpointing support with adapters (@lenglaender via #759)

Gradient checkpointing is a technique for enabling fine-tuning in very memory-limited setttings that nicely complements efficient adapters. It is now supported across all integrated adapter methods. Check out our notebook for fine-tuning Llama with gradient checkpointing and adapters.

More

• Custom names for AdapterFusion layers (@calpt via #774):
  Allows distinguishing multiple fusions on the same adapter by name. See details.
• Allow specifying adapter dtype in AdapterConfig (@killershrimp via #767)

Changed

• Major refactoring of library test suite (@TimoImhof via #740):
  See tests readme for details.
• Upgrade supported Transformers version (@calpt via #757; @lenglaender via #776)

Fixed

• Fix Mistral flash attention with adapters (@divyanshuaggarwal via #758)
• Fix Bottleneck configs to work with ln_before = True and init_weights = "mam_adapter" (@julian-fong via #761)
• Fix default Lora/ (IA)^3 scaling in forward (@calpt via #770) Note: This restores compatibility with adapter-transformers, but changes logic compared to previous Adapters versions. See #770 for details.
• Fixes for sequence generation & orthogonal projection with ReFT (@calpt via #778)
• Various minor compatibility and warning fixes (@calpt via #780, #787)

Adapters v1.0.1

This version is built for Hugging Face Transformers v4.45.x.

New

• Add example notebook for ReFT training (@julian-fong via #741)

Changes

• Upgrade supported Transformers version (@calpt via #751)

Fixed

• Fix import error for Huggingface-hub version >=0.26.0 & Update Notebooks (@lenglaender via #750)
• Fix output_embeddings get/ set for empty active head (@calpt via #754)
• Fix LoRA parallel composition (@calpt via #752)

Adapters v1.0.0

Blog post: https://adapterhub.ml/blog/2024/08/adapters-update-reft-qlora-merging-models

This version is built for Hugging Face Transformers v4.43.x.

New Adapter Methods & Model Support

• Add Representation Fine-Tuning (ReFT) implementation (LoReFT, NoReFT, DiReFT) (@calpt via #705)
• Add LoRA weight merging with Task Arithmetics (@lenglaender via #698)
• Add Whisper model support + notebook (@TimoImhof via #693; @julian-fong via #717)
• Add Mistral model support (@KorventennFR via #609)
• Add PLBart model support (@FahadEbrahim via #709)

Breaking Changes & Deprecations

• Remove support for loading from archived Hub repository (@calpt via #724)
• Remove deprecated add_fusion() & train_fusion() methods (@calpt via #714)
• Remove deprecated arguments in push_adapter_to_hub() method (@calpt via #724)
• Deprecate support for passing Python lists to adapter activation (@calpt via #714)

Minor Fixes & Changes

• Upgrade supported Transformers version (@calpt & @lenglaender via #712, #719, #727)
• Fix SDPA/ Flash attention support for Llama (@calpt via #722)
• Fix gradient checkpointing for Llama and for Bottleneck adapters (@calpt via #730)

Adapters v0.2.2

This version is built for Hugging Face Transformers v4.40.x.

New

• Add example notebook for embeddings training & update docs (@hSterz via #706)

Changed

• Upgrade supported Transformers version (@calpt via #697)
• Add download redirect for AH adapters to HF (@calpt via #704)

Fixed

• Fix saving adapter model with custom heads (@hSterz via #700)
• Fix moving adapter head to device with adapter_to() (@calpt via #708)
• Fix importing encoder-decoder adapter classes (@calpt via #711)

Adapters v0.2.1

This version is built for Hugging Face Transformers v4.39.x.

New

• Support saving & loading via Safetensors and use_safetensors parameter (@calpt via #692)
• Add adapter_to() method for moving & converting adapter weights (@calpt via #699)

Fixed

• Fix reading model info in get_adapter_info() for HF (@calpt via #695)
• Fix load_best_model_at_end=True for AdapterTrainer with quantized training (@calpt via #699)

Adapters v0.2.0

This version is built for Hugging Face Transformers v4.39.x.

New

• Add support for QLoRA/ QAdapter training via bitsandbytes (@calpt via #663): Notebook Tutorial
• Add dropout to bottleneck adapters (@calpt via #667)

Changed

• Upgrade supported Transformers version (@lenglaender via #654; @calpt via #686)
• Deprecate Hub repo in docs (@calpt via #668)
• Switch resolving order if source not specified in load_adapter() (@calpt via #681)

Fixed

• Fix DataParallel training with adapters (@calpt via #658)
• Fix embedding Training Bug (@hSterz via #655)
• Fix fp16/ bf16 for Prefix Tuning (@calpt via #659)
• Fix Training Error with AdapterDrop and Prefix Tuning (@TimoImhof via #673)
• Fix default cache path for adapters loaded from AH repo (@calpt via #676)
• Fix skipping composition blocks in not applicable layers (@calpt via #665)
• Fix Unipelt Lora default config (@calpt via #682)
• Fix compatibility of adapters with HF Accelerate auto device-mapping (@calpt via #678)
• Use default head dropout prob if not provided by model (@calpt via #685)

Adapters v0.1.2

This version is built for Hugging Face Transformers v4.36.x.

New

• Add MT5 support (@sotwi via #629)

Changed

• Upgrade supported Transformers version (@calpt via #617)
• Simplify XAdapterModel implementations (@calpt via #641)

Fixed

• Fix prediction head loading for T5 (@calpt via #640)

Adapters v0.1.1

This version is built for Hugging Face Transformers v4.35.x.

New

• Add leave_out to LoRA and (IA)³ (@calpt via #608)

Fixed

• Fix error in push_adapter_to_hub() due to deprecated args (@calpt via #613)
• Fix Prefix-Tuning for T5 models where d_kv != d_model / num_heads (@calpt via #621)
• [Bart] Move CLS rep extraction from EOS tokens to head classes (@calpt via #624)
• Fix adapter activation with skip_layers/ AdapterDrop training (@calpt via #634)

Docs & Notebooks

• Update notebooks & add new complex configuration demo notebook (@hSterz & @calpt via #614)