Entangled Directions: Concept-Pure Discrimination Geometry Masks Universal Activation Entanglement

Executive Summary

Iterative Null-Space Projection (INLP) is the standard tool for finding and removing concept-specific directions in transformer activation space. This paper shows it has a systematic blind spot. In Qwen 2.5-7B, a single register direction (formal vs. informal) lies 82–88% outside the 36-dimensional domain INLP subspace, yet removing it drops domain classification accuracy by 52–70% across the layers studied (52.5% at layer 27). Removing any of 20 random unit vectors causes no domain damage. Thirty-six iterations of domain-focused INLP missed this direction entirely.

The entanglement is asymmetric: removing the register direction destroys domain information, but removing the 36 domain directions causes zero register damage. It is also learned rather than architectural: register directions computed from domain-neutral prompts carry no domain information and have low cosine similarity (at most 0.28) with the entangled direction. The paper reads this as compression under selection pressure: because register and domain co-occur in training data, a joint representation is efficient.

A dual-metric sigma sweep with a partial projection operator shows that partial extraction at sigma* below 1 preserves more joint information than full removal, with gains of +0.525 to +0.700. The Pareto frontier between extraction and preservation is L-shaped at layer 7 and convex at layers 14–27, placing the transition between layers 7 and 14. The paper proposes multi-concept INLP as a direct fix.

Key Findings

• INLP blind spot: One register direction drops domain accuracy 52.5% at layer 27; 36 iterations of domain INLP did not find it
• Asymmetric entanglement: Removing the 36 domain directions causes 0% register damage
• Specific, not dimensional: 20 random unit vectors cause 0.0% domain damage (z-score above 5000)
• Learned, not architectural: Register directions from domain-neutral prompts carry zero domain information (cosine at most 0.28 with the entangled direction)
• Partial extraction efficiency: Removal at sigma* below 1 gains +0.525 to +0.700 in joint information over full removal
• Developmental geometry: The Pareto frontier is L-shaped at layer 7 and convex at layers 14-27

Key References

• Ravfogel et al. (2020) — Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (ACL)
• Elhage et al. (2022) — Toy Models of Superposition (Transformer Circuits Thread)
• Belrose et al. (2023) — Eliciting Latent Predictions from Transformers with the Tuned Lens (ICLR)
• Gurnee and Tegmark (2024) — Language Models Represent Space and Time (ICLR)