Universal Entanglement in Transformer Activation Space

Executive Summary

This consolidated paper unifies two previously separate works. The discovery (from AI-25) comes from a single model. In Qwen 2.5-7B, a register direction (formal vs. informal) causes a 52.5% drop in domain classification accuracy when removed, despite lying 82% outside the 36-dimensional domain subspace found by Iterative Null-Space Projection (INLP). Thirty-six iterations of domain-focused INLP missed it. The entanglement is asymmetric: removing the 36 domain directions causes zero register damage. It is also learned: register directions computed from domain-neutral prompts carry no domain information.

The replication (from AI-26) uses factorial direction decomposition — multi-output ridge regression over three simultaneously varied concepts (domain, register, and reasoning shape), followed by SVD. The V-matrix shows clean concept separation; the damage matrix shows every informative direction causing 39–88% accuracy drops across all three concepts. Eight experiments across GPT-2, Qwen-0.5B, Qwen-7B, and Qwen-7B-Instruct show entanglement intensity (EI) above 1.0 at every terminal layer. The paper locates entanglement in the informative subspace rather than the full representation, and cites companion work showing that fine-tuning with sufficient capacity can reduce it to zero.

Discrimination-Activation Dissociation

Linear probing assumes that concept-separability in the classifier implies concept-separability in the activations. This paper shows that assumption fails. At layer 27 of Qwen 2.5-7B, each of the seven informative SVD directions loads primarily on a single concept in the V-matrix (primary loadings of 0.961 or higher). Yet removing any one of them damages all three concepts: the minimum drop across all 21 direction-concept pairs is 38.8%. The V-matrix shows how the classifier uses each direction; the damage matrix shows what each direction carries. The paper's methodological conclusion: any interpretability method claiming to isolate a concept should demonstrate isolation in the damage matrix, not just in classifier weights.

Dimension Dependence and PCA

Random Gaussian projections of the learned activations to 448 dimensions reproduce the learned EI in all three models tested (1.50 in Qwen-7B); projections to the 7-dimensional informative rank give near-baseline EI (0.16–0.36). PCA concentrates information into concept-pure directions: at 28 dimensions it reduces EI below 0.10 in every model, while random projections at the same dimension stay at 0.35–0.41. Shuffled labels and pure noise set the null baseline at about 0.37–0.51. Concept-type independence is tested by replacing the linguistic concepts with software engineering concepts (mean SE/original EI ratio 0.97). RLHF increases entanglement: the Instruct model crosses EI = 1.0 earlier in the forward pass (depth fraction 0.222 vs. 0.296) and reaches a higher peak.

Superlinear Amplification

When three concepts are probed simultaneously, the triple EI exceeds the mean pairwise EI by about 2x (GPT-2: 1.87x, Qwen-7B: 2.15x). Every pairwise EI is below 1.0; the triple EI is above it. Treating domain and register as a single nested concept reduces EI below the pairwise average, which the paper reads as showing that the number of independently tracked concept axes, not the informative subspace rank, drives entanglement.

Key Findings

• Concept-pure classifier weights: Each informative SVD direction's primary V-matrix loading is 0.961 or higher
• Minimum cross-concept damage 38.8%: Every direction carries every concept in its activations
• INLP blind spot: One register direction drops domain accuracy 52.5%; 36 iterations of domain INLP missed it
• Random projection reproduces entanglement: Gaussian projections to 448d match learned EI (1.50 in Qwen-7B)
• PCA disentangles: PCA to 28d reduces EI below 0.10 in all three models
• Superlinear amplification: Triple/pairwise EI ratio 1.87x-2.15x
• Cross-model consistency: All four models show EI greater than 1.0 at terminal layers
• RLHF accelerates crystallization: The Instruct model crosses EI = 1.0 earlier in the forward pass

Superseded Papers

This paper consolidates and supersedes:

• AI-25: Entangled Directions — the INLP blind spot and asymmetric feature coupling
• AI-26: Structural Entanglement in the Informative Subspace — eight experiments on cross-model replication, dimension dependence, and superlinear scaling

Key References