- NeurIPS 2026
- Accepted
- Main conference · Poster
LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition
Conference on Neural Information Processing Systems (NeurIPS 2026)
Lead author: method, complete codebase and all experiments, on a single RTX 3090. Final-year undergraduate project, University of Moratuwa.
In one sentence. LoRASpace lets one model use a whole pool of separately trained LoRA adapters through a single composition mechanism, either with a fixed combination for every input (static) or with a mixture chosen per input (dynamic), without retraining the adapters together.
Status. Accepted at NeurIPS 2026 as a main-conference poster. The paper, full author list, code and citation will be posted on this page when the camera-ready version is public. Questions before then are welcome by email.
My role. Lead author. I developed the method, wrote the complete codebase, and ran every training, evaluation and ablation run.
Contribution
Low-rank adapters (LoRA) are small add-ons that specialise a large model cheaply, for example one adapter per style, subject or task. Many requests need several adapters at once. LoRASpace treats two decisions separately: which adapters an input needs, and how their effects are combined. One composition mechanism, which the paper calls a pool-wide shared substrate, serves every adapter in the pool. The adapters are used as they were trained.
Problem
Each adapter is trained on its own. Merging their weights is cheap, but the adapters interfere with each other. Running every adapter as a separate side path avoids that, at the cost of extra computation in every forward pass. Routing methods pick adapters per input, and each is usually built around one way of combining them. The paper asks whether choosing the adapters and combining them can be separated, so that one composition mechanism serves the entire pool.
Setting and constraints
All training, evaluation and ablation runs were done on a single RTX 3090 (24 GB). I owned the method and the complete codebase. With that budget, the method has to work on existing adapters as they are, with no joint retraining step.
Evaluation
- Diffusion. Stable Diffusion XL, composing across a pool of 20 style and content adapters, scored with a GPT-4o paired-comparison judge over hundreds of prompt and combination settings.
- Language models. The LoraRetriever benchmark on LLaMA-2-7B: 48 task adapters and about 2,400 samples across 10 domains. LoRASpace beats a retrieval-based routing baseline on macro task accuracy, and matches or beats the oracle router on several domains.
Supervision
Supervised by Dr. Sampath Perera and Dr. Ranga Rodrigo (University of Moratuwa), with Prof. Jason Mars (University of Michigan).