Executive Summary
This consolidated paper studies what happens to structural entanglement when models are fine-tuned with QLoRA across six conditions, three Qwen-2.5-Coder scales (7B, 14B, 32B), and four architecture families (Qwen, Mistral, DeepSeek, CodeLlama). Under condition B3 (code plus a natural-language companion), entanglement intensity falls 32% at 7B (0.641 ± 0.099, 3 seeds), 93% at 14B (0.063 ± 0.109, 3 seeds), and 100% at 32B (0.000 ± 0.000, 8 seeds). The paper places the transition from partial to complete disentanglement between 7B and 14B; at 32B the collapse is a sharp phase transition at training steps 2000–3500.
At ~7B the response depends on architecture: Qwen falls 32%, DeepSeek 24%, Mistral 10%, and CodeLlama rises 31%. Crosstalk structure partially predicts fine-tuning interference: the math companion retains 11.6 percentage points more HumanEval+ performance than the NL companion (48.2% vs. 36.6%, p = 0.016). But creative writing, the companion with the lowest measured crosstalk (CT = 0.0015), gives the worst coding performance among non-curriculum conditions (41.5%). The paper reports this as a falsification of naive crosstalk minimization.
Companion Selection and CSR
On Qwen-2.5-Coder-32B under strong intervention (rank 64, all linear layers, 5,000 steps), complement-subspace regularization (CSR) added to the math companion (B4) is the best fine-tuned condition at 49.4% HumanEval+, against a 58.5% baseline. CSR is robust to its hyperparameter: final EI varies by less than 0.01 across a 10x lambda range (0.03–0.3). Sequential math-then-code training (B5) is catastrophic at 10.4%. The paper's conclusion: low interference is necessary but not sufficient; a companion must also provide transferable inductive bias.
Scale and Architecture
B3 drives EI to zero in all 8 seeds of Qwen-32B while domain classification accuracy stays between 0.917 and 1.000; two of the eight seeds recover briefly before collapsing permanently. The non-degeneracy margin does not explain susceptibility: Qwen-32B has by far the largest margin (1.781) yet is the model that collapses completely, while every model at 14B and below has a near-zero margin. A preliminary reversibility test (N = 1) on Qwen-14B finds that continuing with the math companion after B3 drives EI from 0.279 to 0.000 within 1,000 steps rather than restoring it.
Key Findings
- Scale ladder: -32% at 7B, -93% at 14B, -100% at 32B under B3 within Qwen-2.5-Coder
- B3 drives EI to zero in Qwen-32B: All 8 seeds at EI = 0.000 by step 3,500
- Architecture-dependence at 7B: Qwen -32%, DeepSeek -24%, Mistral -10%, CodeLlama +31%
- Companion selection: Math beats NL on HumanEval+ (48.2% vs 36.6%, p = 0.016)
- Creative writing falsification: The lowest-crosstalk companion scores 41.5% on HumanEval+
- CSR: Best fine-tuned condition (49.4%) and insensitive to a 10x lambda range
- Possible one-way ratchet: A single reversibility run deepens rather than reverses disentanglement
Superseded Papers
This paper consolidates and supersedes:
- AI-28: Entanglement-Optimal Fine-Tuning — crosstalk-guided companion selection, CSR, cross-family replication
- AI-29: Selective Disentanglement — architecture-dependent collapse, scale ladder, mechanism tests
Key References
- McEntire (2026) — Universal Entanglement in Transformer Activation Space (AI-25/26): discovery, replication, and the discrimination-activation dissociation
- Dettmers et al. (2023) — QLoRA: Efficient Finetuning of Quantized Language Models (NeurIPS)
- Hu et al. (2022) — LoRA: Low-Rank Adaptation of Large Language Models (ICLR)
- Liu et al. (2024) — Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (NeurIPS)