Rotor Compression for Transformer Weights

A 4x4 transform matrix stores 16 numbers. A transform built only from scale, rotation, and translation needs ten: three for translation, three for scale, four for a unit quaternion. Constraints fix the other six. I wanted to know whether the linear weights inside a pretrained language model hide similar structure, and whether rotations and scalings could store them in less space.

The test model was Qwen3.5-0.8B-Base, on one workstation: a Ryzen 5 7600X and an RX 7900 XTX under ROCm. Every run recorded source tensor checksums, seeds, probe fingerprints, and the host environment. Three representations went through that harness. Two failed. One worked to machine precision and saved nothing. Code and run reports are on GitHub.

Attempt	Training	Size vs dense	Accuracy	Decode speed	Verdict
Learned rotor cascade, all 24 FFNs	Gradient fit	25% of BF16 bytes	Weight error 0.73; 0/256 prompts matched	5.8 tok/s vs 37.8 dense	Generation collapsed
SVD to GA rotor angles, one projection	None	+0.06%	Error 8.2e-15	Not measured	Exact, no compression
Clifford block projection, q=4 / q=8	None	25% / 12.5%	Weight error 0.866 / 0.936	Not measured	No better than random weights

Fitted rotor cascades

The first representation replaced each feed-forward projection with a fixed cascade W ≈ D₀R₁D₁…RₖDₖ: 37 rotor blocks, each a set of butterfly-pattern Givens stages, interleaved with diagonal scalings and sized to 25% of the dense BF16 bytes. Gradient descent fit the angles and scales to each weight matrix independently, with 200 scale-only steps followed by 1,800 steps on angles and scales under cosine decay.

The fit stalled. Layer 0’s gate, up, and down projections finished at relative Frobenius errors of 0.73, 0.74, and 0.73. I converted all 24 FFNs anyway and ran a 256-prompt continuation corpus against the base model with greedy decoding. Zero of 256 outputs matched. 186 continuations contained only whitespace or end-of-text tokens. The other 70 were punctuation runs or one repeated word: commas, dashes, “and and and.” A fused Triton kernel for the rotor path decoded at 5.8 tokens/s against 37.8 for dense, so the representation lost on speed as well.

Locating the damage

Whole-model collapse can come from the math, the export, or the runtime. I isolated each one. Replacing only layer 0’s FFN on a fixed prompt produced an output cosine of 0.34 against the original, relative L2 error of 0.94, and 27% of the reference RMS. The FP32 master weights reproduced the same numbers, which ruled out BF16 serialization; storage rounding alone added 1.9 to 2.4% error per projection. The runtime matched dense execution of the stored rotations to a maximum absolute difference of 7.9e-5 in FP32.

The residual stream masked the failure. With the broken FFN in place, the block’s output RMS stayed at 96% of reference while its relative error sat at 0.50. Magnitude looked healthy. The direction was gone, and gain cannot restore a direction: the best possible scalar rescaling still leaves 0.94 error.

The fitted functions were the problem. The objective minimized weight-space error per matrix and never saw the SwiGLU composition or real activations. Truncated SVD at the same byte budget did worse, at 0.76 to 0.80 error, so a 25% budget is brutal for these matrices regardless of representation.

Exact conversion without training

Next I dropped the byte budget and removed training entirely. The target became an algebraic conversion that reproduces a real layer to numerical precision.

In the Euclidean Clifford algebra Cl(n,0), a rotor R = a + b·eᵢeⱼ applied as the sandwich x’ = R x R⁻¹ rotates the (i, j) plane:

c = (a² - b²) / (a² + b²)
s = -2ab / (a² + b²)
x_i' = c·x_i - s·x_j
x_j' = s·x_i + c·x_j

That is a Givens rotation. SVD factors any weight into W = U·diag(σ)·Vᵀ, and a sequence of Givens rotations reduces each orthonormal factor to the identity by zeroing every entry below the diagonal. The converter stores one angle per rotation, plus signs and singular values. Execution applies V’s rotations to the input, scales by σ, then applies U’s rotations in reverse. No dense matrix exists at runtime.

Rotations in one column depend on each other, so a naive implementation applies them one at a time. I scheduled elimination cell (j, i) into wave t = (m-1-i) + 2j. Rotations in the same wave touch disjoint row pairs and execute as one batched operation, with every prerequisite landing in an earlier wave.

On layer 0’s full 3584x1024 up projection, the converter reconstructed the weight with relative error 8.2e-15 in FP64. Direct execution on seeded probes matched the dense layer to 8.3e-15, and a serialized reload reproduced both numbers.

Then I counted parameters:

Representation	Stored scalars
Dense BF16	3,670,016
Paired rotor coefficients (a, b)	7,341,056
One angle per rotation	3,672,064

Angles halved the first version and still landed 2,048 scalars above dense. Those 2,048 are the sign vectors. Everything else cancels exactly. For an m×n matrix at full rank n:

U angles:         mn - n(n+1)/2
V angles:         n² - n(n+1)/2
singular values:  n
total:            mn

Full-rank SVD with Givens rotations is a change of coordinates. Orthogonality removes redundancy inside each factor, and storing two factors spends it back. The 4x4 transform saves space because it excludes shear and fixes the bottom row. Compression requires a restricted family, and the real question is whether pretrained weights sit close to one.

Clifford block projection

The restricted family I tested next groups features into multivectors. Cl(2,0) has 4 basis blades and Cl(3,0) has 8. Split the weight into q×q blocks and require each block to act as left multiplication by a multivector. Each block then stores q coefficients instead of q².

This projection needs no optimizer. The blade multiplication matrices L_b are orthogonal under the Frobenius inner product, with ⟨L_b, L_c⟩ = q·δ_bc, so the best fit for a block M is closed-form:

c_b = <M, L_b>_F / q

On the same 3584x1024 projection, with 64 shared Gaussian probes:

Arm	Coefficients	Weight error	Output error	Energy retained
Clifford, q=4	917,504	0.866	0.865	24.96%
Clifford, q=8	458,752	0.936	0.936	12.48%
SVD, q=4 budget	917,191	0.796	0.796	36.66%
SVD, q=8 budget	456,291	0.893	0.892	20.29%

Compressed execution matched its own reconstruction to 1.6e-15, so these errors measure approximation, not implementation.

The control settled it. Projecting an isotropic random matrix onto a q-dimensional subspace of a q²-dimensional space keeps 1/q of its energy: 25% for q=4, 12.5% for q=8. The pretrained weight kept 24.96% and 12.48%. Norm-matched Gaussian weights in the same harness kept 24.95 to 25.05% and 12.48 to 12.53%. Three seeded feature permutations moved the pretrained result by less than 0.04 percentage points. Qwen’s trained weights align with this Clifford structure no better than noise does, so this family does not scale to full-model conversion.

Next family

A single left product is the most restrictive Clifford layer available. The next candidate sums left and right products, y = Σ aₗ·x·bₗ, which adds capacity in steps of at most 2q coefficients per term and exposes the trade between size and error directly. Fitting aₗ and bₗ is bilinear, so the closed-form projection no longer applies. The gate stays the same: count every coefficient against dense, and run the norm-matched random weight beside the real one. A representation that cannot beat noise on retained energy does not get a full-model run.