The Shape of the Problem: Domain-Invariant Structural Signatures in Activation Space Enable Cross-Domain Solution Transfer

Executive Summary

Problems have a shape. A diagnostic hierarchy in medicine, a statutory hierarchy in law, and a class hierarchy in code all instantiate the same structural pattern despite sharing no domain vocabulary. This paper shows that language models encode this structural signature in activation space independently of domain, and that the two components are linearly separable.

Using Iterative Null-space Projection (INLP) on Qwen 2.5 models at four scales (0.5B, 1.5B, 3B, 7B), domain classification accuracy falls from at least 97.5% to at most 37.5% (chance is 25%) while shape classification holds at 95.6% or higher, a shape survival rate of 98.7-100.0%. Three transfer tests show the surviving structural signal is functional: cross-domain shape classification on a held-out domain (90.6-93.8%), nearest-prototype matching after erasure (89.4-91.9%), and strip-and-rehydrate, which erases domain A, injects the domain B mean, and checks whether activations land closer to the correct shape in domain B (87.5-95.2%, chance 50%), with rehydrate accuracy improving monotonically with scale.

At 7B, domain encoding becomes distributed (36 INLP directions versus 9-16 at smaller scales) while structural encoding stays cleanly linear. Subspace-targeted stochastic resonance, with noise applied only inside the 36-dimensional INLP domain subspace, finds domain signal that standard INLP cannot reach at any iteration budget: an erasure floor of 0.269 versus 0.294. The paper attributes this to near-parallel direction accumulation at 7B hiding residual domain information from iterative removal.

Key Contributions and Methodology

The probe set crosses four domains (medical, legal, code, science) with four structural shapes (hierarchical decomposition, causal chain, constraint satisfaction, evidence aggregation), so that each shape appears in every domain with domain-specific vocabulary. Each probe's activation fingerprint is the mean-pooled final hidden state.

INLP iteratively trains linear classifiers on domain labels and projects activations onto the null space of each classifier until domain classification approaches chance. The critical test is whether shape classification survives this erasure; across all four scales it holds at 95.6% or higher. Representational Similarity Analysis gives a stable domain-to-shape variance ratio of 2.6-2.9x across scales: domain occupies more representational volume, but shape is more geometrically coherent.

A structural-subspace control confirms a double dissociation: noise in the domain subspace preserves shape accuracy (0.975), while noise in the structural subspace destroys it (0.338). The paper frames the decomposition as the activation-space analogue of reduced-space projection in the Automatically Scalable Computation (ASC) architecture: solve a problem's shape in one domain, rehydrate in another.

Key Findings

• Linear separability: INLP erases domain signal to near-chance (at most 37.5%) while shape classification holds at 95.6% or higher across Qwen 2.5 0.5B-7B
• Three transfer tests: Cross-domain shape classification (90.6-93.8%), nearest-prototype matching (89.4-91.9%), and strip-and-rehydrate (87.5-95.2%) show the structural signal is functional
• Transition at 7B: Domain encoding needs 36 INLP directions at 7B versus 9-16 at smaller scales; structural encoding stays linear
• Stochastic resonance capability gain: Subspace-targeted noise reaches a domain erasure floor of 0.269 versus 0.294 for standard INLP
• Double dissociation: Noise in the domain subspace preserves shape (0.975); noise in the structural subspace destroys it (0.338)

Key References