Executive Summary
Domain specialists fine-tuned from a shared generalist can, in principle, be composed at inference time according to what a query needs. This paper introduces constellation-indexed composition: capture the query's activation fingerprint in the generalist model, project it onto Gram-Schmidt-orthogonalized domain centroids to obtain per-domain relevance weights, and compose specialist parameters through node-level weighted mixing in which each hidden dimension receives its own specialist mixture.
Two methodological choices prove essential. First, generalist-space indexing: building the index from generalist rather than specialist activations eliminates generalist-specialist alignment failures and raises single-domain win rate against the generalist from 20.6% to 98.1% at Qwen 0.5B. Second, Gram-Schmidt orthogonalization: raw domain centroids have mean cosine similarity 0.91-0.97 (medical-legal 0.988-0.993), which makes naive projection meaningless; orthogonalization reduces this to about zero.
Across two architecture families and five scales (GPT-2 124M and Qwen 2.5 at 0.5B, 1.5B, 3B and 7B) with four specialists (medical, legal, code, science), all five models reach at least 93% cross-domain win rate against the generalist. Against task arithmetic the scale dependence is peaked and non-monotonic: 98.7% at 124M, 23.3% at 0.5B, 64.0% at 1.5B, 92.0% at 3B, and 60.7% at 7B. At 7B, injecting Gaussian noise at sigma = 0.020 times the centroid norm before orthogonalization lifts composition from 8.7% to 99.3% against task arithmetic (+90.7 points), while noise has null effects at the other four scales.
Key Contributions and Methodology
The method treats each specialist's contribution as a function of the query's position in generalist activation space. Specialists are fine-tuned from the generalist checkpoint, so the generalist provides a shared coordinate system for the index. Domain centroids are computed in that space and orthogonalized, so that each query decomposes into per-domain weights that drive parameter mixing.
Within the Qwen family, domain centroid collinearity rises monotonically with scale (0.906, 0.934, 0.969, 0.973), and constellation's advantage over task arithmetic tracks collinearity from 0.5B through 3B, then falls at 7B. The paper reads this as a collinearity sweet spot: constellation composition dominates when collinearity is high enough to make task arithmetic destructive (about 0.93-0.97), but above about 0.97 the orthogonal residuals fall below the effective precision threshold.
The stochastic resonance experiment shows that this signal attenuation is recoverable. Noise added to the centroids before orthogonalization produces an inverted-U at 7B, with the best effect at sigma = 0.020 times the centroid norm, making noise-assisted decomposition a targeted rescue for catastrophic collinearity collapse rather than a general improvement.
Key Findings
- Generalist-space indexing: Raises single-domain win rate against the generalist from 20.6% to 98.1% (Qwen 0.5B) by eliminating alignment failures
- Orthogonalization: Gram-Schmidt reduces domain centroid cosine similarity from 0.91-0.97 to about zero
- Non-monotonic scale dependence: Win rate against task arithmetic is 98.7% (124M), 23.3% (0.5B), 64.0% (1.5B), 92.0% (3B), 60.7% (7B)
- Stochastic resonance rescue: Noise at sigma = 0.020 times the centroid norm lifts 7B composition from 8.7% to 99.3% against task arithmetic (+90.7 points); null at other scales
- Architecture-invariant geometry: The rank order of domain similarities (code most distinct, medical-legal most entangled) is the same across all five models
Key References
Editing Models with Task Arithmetic. International Conference on Learning Representations.
TIES-Merging: Resolving Interference When Merging Models. Advances in Neural Information Processing Systems.
Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time. International Conference on Machine Learning.
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. Conference on Language Modeling.
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. International Conference on Learning Representations.