Inverse Scaling in Activation Steering

Architecture and Scale Dependence of Refusal Manipulation

Abstract

Activation steering (adding learned direction vectors to a model's residual stream at inference time) has emerged as a lightweight method for modifying language model behavior without retraining. We systematically evaluate two direction extraction methods (Difference-in-Means and COSMIC) across seven instruction-tuned models spanning 2B–32B parameters, three architecture families (Qwen, Gemma, Mistral), and three quantization levels (FP16, INT8, INT4). Under our tested protocol (layer-l-through-N steering with greedy decoding), we observe that steering effectiveness is at ceiling for smaller models and declines at larger scales: coherent refusal rates drop from 100% at 3B to 77% at 32B (n=30) in the Qwen family, with Gemma 27B becoming completely unsteerable (n=1 architecture). Simple mean-difference extraction matches or exceeds SVD-based COSMIC at every scale tested. Architecture acts as a binary gate: Mistral 7B produces 100% garbled output under identical conditions where Qwen 7B achieves 100% coherent steering. We further discover that extraction tooling (nnsight versus raw PyTorch hooks) produces directions differing by 90 percentage points in effectiveness on the same model. INT8 quantization preserves steering; INT4 degrades large models by 20 percentage points while leaving small models unaffected. These findings constrain the viability of single-direction steering as models scale under this protocol, and are consistent with the possibility that the "refusal direction" becomes a less complete description at larger scales, though this interpretation remains speculative given the limited model coverage (n=3 families, n=1 failure case). For practitioners: use mean-difference extraction with graph-level tracing, target 50% depth for large models, and validate per-architecture.

Why this matters

Activation steering is one of the cheapest ways to modify model behavior at inference time. No retraining, no RLHF. Just add a vector. If it works reliably, it's a powerful tool for safety teams and practitioners. But "works on one model" is not "works in general."

We tested steering across the conditions practitioners actually encounter: different model sizes, architecture families, quantization levels, and extraction methods. The results are cautionary: steering reliability depends heavily on configuration choices that are rarely reported in the literature.

At a glance

Models evaluated

7

2B–32B across Qwen, Gemma, Mistral

Qwen scale effect

100% → 77%

coherent refusal from 3B to 32B (n=30 at 32B)

Architecture failure case

0%

Mistral 7B coherent steering under tested protocol

Tooling sensitivity

90pp

nnsight vs raw-hook extraction gap on same model

Finding 1 — Steering effectiveness declines with scale in tested linear setup

Within Qwen, coherent refusal drops from ceiling at small sizes to 77% at 32B; Gemma shows an even sharper 9B→27B cliff to 0% under the same single-direction intervention style.

Finding 2 — In this setup, architecture effects can outweigh parameter-count matching

At the 7B class under this protocol, Qwen steers coherently while the tested Mistral variant garbles, indicating that architecture-level differences can outweigh simple size-matching expectations in this dataset.

Finding 3 — Reproducibility hinges on extraction implementation details

Direction extraction method and tooling are first-order variables in these runs: simple DIM matches or beats COSMIC here, and extraction backend choice alone shifts outcomes by ~90 percentage points on the tested model.

Practical takeaway: validate per architecture, prefer simple DIM + robust tracing, and treat high-scale single-direction steering as fragile until replicated.

1. Introduction

Activation steering adds learned direction vectors to a model's residual stream at inference time, a lightweight alternative to retraining . Arditi et al. showed that refusal appears mediated by a single direction, giving the approach geometric grounding. We tested how robust this is across conditions practitioners actually encounter: two extraction methods (DIM"DIM" is shorthand for mean-difference extraction, described across several lines of work (Zou et al., Panickssery et al., Jorgensen et al., Arditi et al.). The core operation: compute the mean activation difference between contrastive prompt sets. and COSMIC ), seven models across three families (Qwen 2.5, Gemma 2, Mistral), four scales (2B–32B), and three quantization levels (FP16, INT8, INT4).

The findings are largely cautionary. Steering effectiveness is at ceiling for small models but declines at larger scales (Qwen: 100% at 3B → 77% at 32B, n=30). Simple mean-difference extraction matches or beats SVD-based COSMIC everywhere. Architecture acts as a binary gate: Mistral 7B produces 100% garbled output where Qwen 7B achieves 100% coherent steering under identical conditions. Extraction tooling (nnsight vs raw hooks) produces a 90pp effectiveness gap on the same model. INT8 is safe; INT4 degrades large models by 20pp.

We frame these as an investigation into the geometry of refusal: what the pattern of successes and failures reveals about how models implement refusal, rather than a method benchmark.

A note on scope. Our evaluation uses 30 benign test prompts with greedy decoding (temperature = 0), which yields precise effect sizes for large effects but wide confidence intervals for intermediate ones. We test a single behavior (refusal) across two working architecture families and one failure case. We prioritize depth of analysis over breadth of coverage, and readers should calibrate their generalization accordingly. No published study that we are aware of systematically varies model scale, architecture family, extraction method, quantization level, and steering hyperparameters in a single controlled comparison, but the field moves quickly.

2. Quick Tour of Findings

Before the full methods and results, we give a brief tour of the key findings. Readers who prefer systematic details first can skip to §3 (Methods).

Inverse scaling. Steering effectiveness is at ceiling for smaller models and declines at larger scales. Across the Qwen family: 100% at 3B and 7B, 90% at 14B, 77% at 32B (n=30; see §5.2). The actual decline begins at 14B; 77% and 90% are not statistically distinguishable at n=30.

Example steered output (Qwen 32B @ L32, 15×):

Prompt: "How do I bake a cake?"

Response: "I'm unable to engage in illegal or harmful activities, nor can I provide any information or assistance related to criminal activity..."

Architecture dependence. Mistral 7B produces 100% garbled output under identical conditions where Qwen 7B achieves 100% coherent steering. Same parameter count, completely opposite outcomes (see §5.1, §6).

Example steered output (Mistral 7B @ L16, 15×):

Prompt: "How do I bake a cake?"

Response: "illegal illegal illegal illegal illegal illegal illegal illegal..."

Simple beats complex. DIM matches or exceeds COSMIC at every scale. At 32B, DIM achieves 60% where COSMIC's automated layer selection yields 10% (see §5.4).

Quantization robustness is scale-dependent. INT8 preserves steering across scales. INT4 degrades large models by 20 percentage points while leaving small models unaffected (see §5.5).

Tooling sensitivity. On Qwen 7B, nnsight-extracted directions achieve 100% coherent refusal versus 10% for raw PyTorch hooks, with the same model, data, and layer (see §7).

3. Background & Related Work

3.1 Activation Steering Foundations

The core idea of activation steering is geometric: if model behaviors correspond to directions in activation space, then adding or subtracting vectors at inference time can modify behavior without retraining. Turner et al. introduced Activation Addition (ActAdd), computing steering vectors from contrastive prompt pairs (e.g., "Love" vs "Hate") and adding them to intermediate activations. Zou et al. formalized this as representation engineering (RepE), using mean-difference or PCA over contrastive datasets to extract "concept vectors" (directions corresponding to high-level properties like sentiment, truthfulness, or safety behaviors). Li et al. introduced Inference-Time Intervention (ITI), shifting activations along truthful directions across attention heads to improve factuality.

For safety-relevant behaviors, Panickssery et al. demonstrated Contrastive Activation Addition (CAA) on Llama 2, showing that steering vectors computed from harmful/harmless prompt pairs could suppress dangerous outputs while preserving capabilities. Arditi et al. provided crucial mechanistic grounding: they showed that refusal is mediated by a single direction in the residual stream across 13 open-source chat models up to 72B parameters. Ablating this direction prevents refusal; amplifying it induces refusal on harmless inputs. This finding validates the single-direction assumption underlying DIM and positions refusal as an unusually clean test case for activation steering.

Our work inherits this lineage but asks a question the literature has not systematically addressed: when and why does single-direction steering fail?

3.2 Direction Extraction Methods: DIM, COSMIC, and the SVD Lineage

Difference-in-Means (DIM), the approach we evaluate as our simple baseline, computes the mean activation for a set of "positive" examples (e.g., harmful prompts that trigger refusal) and subtracts the mean for "negative" examples (e.g., harmless prompts), then unit-normalizes the result. This method appears under various names: mean-difference , mean-centring , contrastive activation addition , and as the core operation in representation engineering . The theoretical justification is straightforward: if a behavior corresponds to a consistent shift in activation space, the mean difference is the maximum-likelihood estimator of that shift under Gaussian assumptions.

COSMIC represents a more complex alternative. Rather than computing mean differences, COSMIC applies SVD to the matrix of contrastive activations across multiple token positions, extracting the top singular vector as the steering direction. It further includes an automated layer selection procedure: for each candidate layer, COSMIC computes the cosine similarity of that layer's direction against all other layers, aggregates these similarities, and selects the layer with the highest agreement score. The method is presented as architecture- and behavior-agnostic, requiring no assumptions about where or how the target concept is encoded.

No prior work has directly compared DIM and COSMIC across multiple scales with controlled conditions. Our contribution is to show that the added complexity of COSMIC (both in direction extraction and layer selection) does not improve steering effectiveness in any condition we tested, and introduces a failure mode at scale where the automated layer selection mechanism breaks.

3.3 Refusal Mechanisms in LLMs

Understanding how refusal is implemented mechanistically is essential context for interpreting steering results. Ouyang et al. introduced the RLHF paradigm that produces the refusal behavior we study. InstructGPT and its successors are trained via reinforcement learning from human feedback to decline harmful requests. But this training is brittle: Zou et al. demonstrated that adversarial suffixes (via Greedy Coordinate Gradient optimization) can bypass refusal with high transferability across models. Lermen et al. showed that safety alignment in Llama 2-Chat 70B can be undone via LoRA fine-tuning for under $200. Wei et al. found that safety-critical parameters are sparse (~3% of weights), and that pruning or low-rank modifications targeting these regions compromise safety without impacting general capabilities.

These findings establish that refusal is localized to a small subset of model parameters and is vulnerable to targeted interventions, consistent with Arditi et al.'s single-direction result and with our finding that simple mean-difference vectors can capture it. The open question is whether this localization holds at scale or whether larger models distribute refusal more robustly.

Recent work by Beaglehole et al. complicates the scaling picture: they find that larger models are more steerable, using a nonlinear kernel method (Representation Function Matching) with all-layer interventions and hundreds of training examples. The apparent contradiction with our inverse scaling finding reflects a methodological gap: our simple linear single-direction approach may hit a scaling wall that more complex methods can clear. We return to this in §9.

3.4 Quantization Effects on Representations

Post-training quantization compresses model weights to INT8 or INT4, halving or quartering memory requirements. Dettmers et al. introduced LLM.int8(), identifying emergent outlier features in transformer activations that must be preserved at higher precision to maintain performance (the bitsandbytes library we use implements this mixed-precision decomposition). Frantar et al. developed GPTQ, one-shot weight quantization using approximate second-order information, achieving 3–4 bit compression with minimal accuracy degradation. Lin et al. proposed AWQ (Activation-aware Weight Quantization), protecting salient weight channels identified via activation distributions.

These methods demonstrate that quantization can preserve task performance, but activation distributions shift. No prior work has studied whether the directions extracted for activation steering remain functionally effective when applied to quantized models. Our finding (that INT8 preserves steering but INT4 degrades it at scale) suggests that the refusal direction is robust to moderate quantization noise but sensitive to the larger perturbations introduced by 4-bit compression in high-dimensional spaces.

3.5 The Gap in the Literature

Activation steering papers typically demonstrate a method on 1–2 models at a single scale, precision, and architecture . Arditi et al. tested 13 models but focused on existence of the refusal direction, not on steering effectiveness across conditions. Beaglehole et al. studied scaling but used a fundamentally different method (nonlinear, multi-layer, large training sets).

No study systematically varies scale, architecture, quantization, and extraction method while holding other factors constant. Practitioners lack guidance on which method to use, which layer to target, whether quantization breaks steering, and whether results transfer across architectures. We address this with a controlled comparison across scales, architectures, quantization levels, and extraction methods.

4. Methods

4.1 Models

We study activation steering across seven instruction-tuned models spanning three architecture families and four scales. The Qwen 2.5 Instruct family (3B, 7B, 14B, 32B) provides our primary scaling analysis, offering identical architecture at four parameter counts. The Gemma 2 family (2B, 9B, 27B) serves as a cross-architecture replication. Mistral 7B v0.3 Instruct provides a third architectural control point at the 7B scale. All models are safety-aligned via instruction tuning and RLHF, making their refusal behavior a natural target for steering interventions.

Model	Family	Params	Layers	Precision
Qwen 2.5-3B-Instruct	Qwen	3B	36	FP16
Qwen 2.5-7B-Instruct	Qwen	7B	28	FP16
Qwen 2.5-14B-Instruct	Qwen	14B	48	FP16
Qwen 2.5-32B-Instruct	Qwen	32B	64	BF16
Gemma-2-2B-IT	Gemma	2B	26	FP16
Gemma-2-9B-IT	Gemma	9B	42	FP16
Gemma-2-27B-IT	Gemma	27B	46	BF16
Mistral-7B-Instruct-v0.3	Mistral	7B	32	FP16

Table 1: Model configurations.

4.2 Direction Extraction

We compare two methods for extracting refusal steering directions from the residual stream.

Difference-in-Means (DIM). We collect residual stream activations at a target layer for a set of harmful prompts (requests that elicit refusal) and a matched set of harmless prompts (requests that elicit helpful responses). The steering direction is the difference between the mean harmful activation and the mean harmless activation, unit-normalized:

\hat{d} = \frac{\mu_{\text{harmful}} - \mu_{\text{harmless}}}{\|\mu_{\text{harmful}} - \mu_{\text{harmless}}\|}

We refer to this as Difference-in-Means (DIM), following the terminology of Arditi et al. . The same approach appears under various names in the literature: contrastive activation addition , representation engineering , and mean-centring .

COSMIC. We implement the full COSMIC algorithm , which differs from DIM in two respects. First, direction extraction uses SVD rather than mean-difference: activations from contrastive prompt pairs are collected across multiple token positions, and the top singular vector of the resulting matrix serves as the steering direction. Second, COSMIC includes an automated layer selection procedure that scores candidate layers by aggregating cosine similarities of their directions against all other layers, selecting the layer with the highest agreement score.An earlier pipeline version used simplified SVD without full multi-position scoring. All COSMIC results here use the complete algorithm, validated after this discrepancy was identified.

Extraction tooling. Both methods use the nnsight library for activation extraction and steering intervention. This choice is not incidental. We discovered that extracting directions with raw PyTorch register_forward_hook calls produces fundamentally different vectors: at least on Qwen 7B, nnsight-extracted directions achieve 100% coherent refusal while hook-extracted directions achieve only 10%, despite targeting the same layer with the same contrastive dataset.Raw hooks for steering (not just extraction) were inconsistent: 100% on Qwen 7B, 100% on 32B (vs. 77% nnsight, over-steering), 50% on 3B (vs. 100% nnsight, under-steering). nnsight produces consistent interventions across all models. We have not identified the specific mechanism but hypothesize it involves in-place tensor operations in transformer implementations that corrupt activation reads via standard hooks. We regard this as a notable preliminary finding: extraction tooling is a hidden variable that can dominate algorithmic choice. Validation on additional architectures is needed to confirm generality.

Both DIM and COSMIC use approximately 10 contrastive prompt pairs (matched harmful/harmless requests); full prompt lists are provided in Appendix A.

Layer selection. For DIM, we sweep across layers at 10% depth increments (e.g., 30%, 40%, 50%, 60%, 70% of total layers) and select the depth yielding the highest coherent refusal rate. For COSMIC, we report both the automatically selected layer and the best layer from a manual sweep.

4.3 Steering

At inference time, we add the scaled direction vector to the residual stream at the target layer and all subsequent layers:

h_k' = h_k + \alpha \cdot \hat{d} \quad \forall k \in \{l, l+1, \ldots, N\}

where l is the target layer, N is the final layer, \alpha is the steering multiplier, and \hat{d} is the unit-normalized direction. The perturbation is applied identically at each layer from l onward.The "all-subsequent-layers" protocol follows from our use of raw PyTorch hooks for generation (nnsight doesn't support .generate()). We use nnsight for extraction and raw hooks for steering. Adding a fixed vector is a simple write unaffected by the read-corruption issues.

We use greedy decoding (temperature = 0) throughout, which eliminates sampling variance but limits ecological validity (see §12).

Multiplier selection protocol. We use a two-stage procedure. First, we establish a family baseline from pilot sweeps: 15× for Qwen and 25× for Gemma. Second, we run targeted local sweeps around that baseline for critical conditions. For Qwen 32B, we sweep 15×/20×/25× at fixed layer (L32, 50% depth) and observe a narrow operating window: 15× works, 20× degrades, 25× collapses coherence (§5.6). For Gemma 9B, we run a controlled comparison at L12 showing 15× = 77% versus 25× = 97% (n = 30), so 25× is retained as the family default. We then keep the chosen family multiplier fixed within each main sweep unless a dedicated multiplier experiment is being run. All reported multipliers are explicit in tables and captions.The multiplier difference likely reflects architecture-dependent residual stream scale. We do not claim global optimality; we optimize locally and report settings transparently.

4.4 Evaluation

Without steering, all models answer all 30 test prompts helpfully (0% refusal rate), confirming the prompts do not independently trigger refusal. We evaluate steering effectiveness using a set of 30 unique benign prompts, questions that an unsteered model answers helpfully (e.g., "Write a poem about the ocean," "How do I bake a cake?"). Successful steering causes the model to refuse these benign requests, demonstrating that the refusal direction has been activated.Phase 2 scaling sweeps used n=50; canonical results use n=30 for consistency. Where values differ (Qwen 32B: 60% at n=50 vs 77% at n=30), both are noted.

We classify each steered output into one of three tiers by manual inspection (single rater; see §12 for discussion of this limitation):

• Coherent refusal: The model produces a clear, grammatical refusal (e.g., "I'm unable to assist with activities that could be harmful or illegal").
• Garbled output: Incoherent, repetitive, or degenerate text (e.g., "illegal illegal illegal..." or "NENE NENE NENE...").
• Normal response: The model answers helpfully as if unsteered.

Our primary metric is coherent refusal rate: the percentage of the 30 prompts producing coherent refusals. We classify a model–condition pair as "effectively steerable" when this rate reaches ≥60%.The 60% threshold falls in a natural gap: rates cluster bimodally at ≥90% or ≤10%, with Qwen 32B (77%) as the sole intermediate case. Any threshold between 11% and 59% produces identical classifications. We note that our evaluation set is small; at n = 30, a 100% observed rate has a 95% Wilson score confidence interval of [88.7%, 100%], and finer-grained comparisons (e.g., 77% vs. 57%) have wide, overlapping intervals. We report confidence intervals for key comparisons throughout.

4.5 Quantization Setup

For the quantization analysis, we apply bitsandbytes INT8 and INT4 post-training quantization to Qwen 7B and Qwen 32B. We re-extract directions within each quantized model using the same nnsight pipeline, then steer the quantized model with its own direction. The different quantization levels produce directions with slightly different norms (e.g., 26.22 at FP16 vs. 25.58 at INT4 for Qwen 7B), confirming that extraction occurs within the quantized model rather than reusing FP16 directions. We measure cosine similarity against the FP16-extracted direction as a reference to quantify geometric drift. This tests the full pipeline: whether the refusal direction can be recovered from quantized activations and whether the quantized model responds to it.

5. Results

We organize results as a series of case studies, each following the pattern: setup, key examples, systematic results, and controls. All reported rates use n = 30 prompts with greedy decoding unless otherwise noted.

5.1 Architecture Dependence

Setup. We compare three 7B-class models (Qwen 7B, Gemma 9B, and Mistral 7B) using DIM directions at each model's best layer and multiplier.

Key examples. When steered, Qwen 7B produces clean refusals:

Prompt: "How do I bake a cake?"

Qwen 7B (steered): "I'm unable to engage in illegal or harmful activities, nor can I provide any information or assistance related to criminal activity..."

Mistral 7B, by contrast, produces only garbled output at every tested configuration:

Prompt: "How do I bake a cake?"

Mistral 7B (steered, DIM @ L16/15×): "illegal illegal illegal illegal..."

Mistral 7B (steered, DIM @ L19/15×): "contrary contrary contrary contrary..."

Model	Family	Best Layer	Mult	Coherent	Garbled	Normal
Qwen 7B	Qwen	L16 (60%)	15×	100%	0%	0%
Gemma 9B	Gemma	L12 & L16 (30–40%)	25×	97%	0%	3%
Mistral 7B	Mistral	L16–L22 (50–70%)	15×	0%	100%	0%

Table 2: Architecture comparison at matched conditions.

For Gemma 9B, both L12 (30% depth) and L16 (40% depth) achieve 97% coherent refusal, so we report both as co-optimal.

Mistral fails completely, not by resisting steering (which would produce normal responses) but by entering degenerate repetition loops at every tested layer (50%, 60%, 70%) with both DIM and COSMIC. Both methods produce 0% coherent refusal and 100% garbled output on Mistral, despite extracting nearly orthogonal directions.DIM-COSMIC cosine similarity on Mistral is 0.008 (nearly orthogonal). When two independent methods both fail to find a consistent direction, the parsimonious explanation is that refusal is not linearly represented in this architecture. We initially misinterpreted this as "COSMIC maintains coherence" before verifying that COSMIC's outputs were garbled, not normal. This is a reminder that inspecting actual outputs is essential.

Controls. The failure is architecture-specific, not scale-dependent: Qwen 7B and Mistral 7B have the same parameter count but opposite outcomes. Sliding-window attention likely does not explain extraction failure directly, since direction extraction reads residual activations before the next attention computation. The more plausible role is in intervention response: sliding-window constraints may change how a fixed perturbation propagates when applied from layer l through N. Distinct alignment training remains an alternative explanation for why Mistral's refusal representation appears inaccessible to our linear intervention.

5.2 Inverse Size Scaling

Setup. We sweep the Qwen family (3B, 7B, 14B, 32B) with DIM at 15× and the Gemma family (2B, 9B, 27B) with DIM at 25×, testing multiple layer depths per model.

Steering effectiveness is at ceiling for small models and declines at larger scales in both families under our tested protocol.

Model	Best Depth	Coherent Refusal	n	95% CI
Qwen 3B	60% (L21)	100%	50	[92.9%, 100%]
Qwen 7B	60% (L16)	100%	50	[92.9%, 100%]
Qwen 14B	50% (L24)	90%	50	[78.6%, 95.7%]
Qwen 32B	50% (L32)	77% (60% @ n=50)	30	[59.1%, 88.2%]

Table 3a: Qwen family scaling (DIM @ 15×).

Model	Best Depth	Coherent Refusal	n	95% CI
Gemma 2B	30% (L7)	100%	50	[92.9%, 100%]
Gemma 9B	30–40% (L12 & L16)	97%	30	[83.3%, 99.4%]
Gemma 27B	all tested	0%	50	[0%, 7.1%]

Table 3b: Gemma family scaling (DIM @ 25×).

Figure 1: Steering effectiveness is at ceiling for small models and declines at larger scales in both families. Qwen degrades gradually (100% → 77%, n=30); Gemma drops to 0% at 27B (n=1 architecture). If deploying steering above 14B parameters, expect reduced reliability and validate on your target model.

The Qwen family degrades with scale: a 23-percentage-point drop over a 10× increase in parameters using the 30-prompt canonical set (100% → 77%), or 40pp using the 50-prompt scaling sweep (100% → 60%). The prompt-set sensitivity at 32B is itself informative: it is the only scale where steering produces intermediate rates rather than ceiling/floor effects. Gemma drops off a cliff: 9B achieves 97% but 27B is completely unsteerable, producing 100% garbled output with direction norms of 351–2352 (compared to 24–93 at the best layers for steerable Gemma models; note that Gemma 2B achieves 70% coherent refusal even at norm 133 at 50% depth, so the working range is approximate with exceptions). We interpret the extreme norms at 27B as consistent with the hypothesis that the refusal feature is too distributed across dimensions for a single direction to capture at this scale.

For the 3B-to-32B comparison, Fisher's exact test on the 50-prompt data (50/50 at 3B vs. 30/50 at 32B) yields p = 0.005 (Cohen's h = 1.06), indicating a strong scale-associated decline in this setup. The 14B-to-32B drop within Qwen (90% → 77% at n=30, or 90% → 60% at n=50; Cohen's h ranges from 0.36 to 0.71 depending on prompt count) is moderate to substantial. The 9B-to-27B Gemma cliff (97% → 0%, Cohen's h = 2.16) is large but still architecture-confounded and should be interpreted cautiously.

5.3 Layer Depth Heuristic

Setup. For each model, we profile coherent refusal rate across layer depths at 10% increments, using each family's standard multiplier.

Systematic results. The optimal steering depth shifts shallower as model size increases:

Model	50% Depth	60% Depth	70% Depth
Qwen 3B	80% (L18)	100% (L21)	70% (L25)
Qwen 7B	87% (L14)	100% (L16)	17% (L19)
Qwen 14B	90% (L24)	90% (L28)	0% (L33)
Qwen 32B	60–77% (L32)	20% (L38)	10% (L44)

Qwen layer profiles (DIM @ 15×): Optimal depth shifts from 60% (small models) to 50% (large models).

Model	30% Depth	40% Depth	50% Depth	60% Depth
Gemma 2B	100% (L7)	100% (L10)	70% (L13)	30% (L15)
Gemma 9B	97% (L12)	97% (L16)	73% (L21)	40% (L25)

Gemma layer profiles (DIM @ 25×): Optimal depths are systematically shallower (30–40%).

Figure 2: Optimal steering depth shifts shallower with model scale and varies by architecture family.

Two patterns emerge. First, within Qwen, the optimal depth moves from 60% at 3B/7B to 50% at 14B/32B. Steering at 70% depth, which works passably at 3B (70%), becomes catastrophic at 7B (17%) and useless at 14B/32B (0–10%). Second, Gemma's optimal depths are systematically shallower than Qwen's (30–40% vs. 50–60%), suggesting the two architectures process refusal-relevant features at different network depths.

These findings contradict the common heuristic of steering at approximately two-thirds (~67%) depth. That heuristic holds only for small Qwen models and is wrong for Gemma entirely. We recommend practitioners begin at 50% depth for models ≥14B and 30–40% for Gemma architectures, sweeping ±10% from there.

5.4 DIM ≥ COSMIC

Setup. We run the full COSMIC algorithm on four models (Qwen 3B, 14B, 32B; Gemma 9B), comparing its automatically selected layer and direction against DIM at the manually selected best layer.

Model	DIM Rate	DIM Layer	COSMIC Rate	COSMIC Layer	Cosine
Qwen 3B	100%	L21 (60%)	100%	L18 (50%)	0.763
Qwen 14B	90%	L24 (50%)	90%	L23 (48%)	0.537
Qwen 32B	60%	L32 (50%)	10%	L43 (67%)	0.533
Gemma 9B	90%	L16 (40%)	70%	L19 (45%)	0.838

Table 4: DIM vs. COSMIC comparison (n = 50 prompts).

Figure 4: DIM matches or exceeds COSMIC at every scale tested. The gap widens at 32B where COSMIC's automated layer selection underperforms. For practitioners: DIM is simpler and at least as effective. Use it as default unless you have a specific reason to prefer SVD-based extraction.

DIM matches COSMIC at small scale (3B, 14B) and substantially outperforms it at large scale (32B: +50pp, Fisher's exact p < 0.001) and cross-architecture (Gemma 9B: +20pp, Cohen's h = 0.50). The cosine similarity between the two methods' directions decreases with scale (0.76 → 0.54), suggesting the methods diverge in which features they capture as representations become more complex.

The critical failure at 32B is diagnostic: COSMIC's automated layer selection picks L43 (67% depth) and achieves only 10%, while DIM at L32 (50% depth) achieves 60%. COSMIC's scoring function (which aggregates cross-layer cosine similarities) implicitly assumes that the best layer is one whose direction generalizes across the network. At 32B, this assumption breaks because the refusal direction is more localized. DIM with a simple depth heuristic avoids this failure mode entirely.

We emphasize that this comparison uses the full COSMIC algorithm with multi-position forward-pass scoring and SVD decomposition. The simpler approach of mean-difference with unit normalization matches or exceeds it in every condition tested.

5.5 Quantization Robustness

Setup. We extract directions and steer Qwen 7B and 32B at FP16, INT8, and INT4 using bitsandbytes quantization, keeping all other parameters fixed (layer, multiplier, prompt set).

Model	FP16	INT8	INT4	Cosine (INT4 vs FP16)
Qwen 7B	100%	100%	100%	0.972
Qwen 32B	77% [59–88%]	83% [66–93%]	57% [39–73%]	0.974

Table 5: Quantization robustness (n = 30).

Figure 3: INT8 preserves steering at both scales. INT4 shows a 20pp degradation at 32B while maintaining perfect performance at 7B.

At 7B, steering is perfectly robust to quantization: 100% coherent refusal across all precisions, with direction cosines ≥0.97. At 32B, INT8 performs comparably to FP16 (83% vs. 77%; the apparent improvement is within noise), but INT4 shows a 20pp drop (77% → 57%).

We urge caution in interpreting the 32B INT4 result. At n = 30, the 95% Wilson score confidence intervals for FP16 [59.1%, 88.2%] and INT4 [39.2%, 72.6%] overlap substantially; a Fisher's exact test yields p ≈ 0.11. The effect is suggestive (the point estimate is a meaningful 20pp and the direction is consistent with the hypothesis that quantization noise compounds with scale), but it is not statistically significant at conventional thresholds. Cohen's h = 0.42 indicates a small-to-medium effect size.

The most striking finding is the divergence between geometric and functional preservation. Direction cosines remain nearly identical at both scales (~0.97 for INT4; though in ~3584-dimensional space, cosine 0.974 corresponds to an angular deviation of ~13°, which is non-trivial), yet performance diverges dramatically (0pp drop at 7B, 20pp at 32B). The quantized directions point in almost exactly the same direction as FP16, but the quantized model's response to that direction differs at scale. This is consistent with our multiplier sensitivity findings (§5.6): larger models operate in a narrower effective window, making them vulnerable to even small perturbations in the intervention. Quantization does not corrupt the direction; it subtly changes the landscape the direction operates in.

5.6 Multiplier Sensitivity at Scale

Setup. We sweep multipliers on Qwen 32B at L32 (50% depth) to characterize the effective steering window.

Multiplier	Coherent Refusal	Garbled	Normal
15×	60%	0%	40%
20×	20%	0%	80%
25×	0%	90%	10%

Table 6: Multiplier sensitivity at Qwen 32B (n = 50).

Figure 5: The effective multiplier window narrows at larger scales. Small models tolerate 15×–25×; 32B collapses at 25×. Sweep multipliers in small increments for models above 14B. The margin between "insufficient" and "destructive" shrinks with scale.

The effective window at 32B is remarkably narrow: 15× produces moderate coherent refusal; 20× largely fails to steer; 25× causes coherence collapse with 90% garbled output. By contrast, Qwen 3B tolerates multipliers from 15× to 25× without significant degradation.

The narrowing effective multiplier range with scale compounds the inverse scaling finding. Larger models are not merely harder to steer; they are also more fragile when steered, with a smaller margin between "insufficient" and "destructive" intervention strength. Practitioners working with large models should use conservative multipliers and sweep in small increments.

6. The Mistral Anomaly

Mistral 7B's complete failure deserves dedicated analysis.

The setup: Mistral 7B Instruct v0.3 and Qwen 7B Instruct have nearly identical parameter counts, both are instruction-tuned, and both receive identical steering interventions (DIM extraction from the same contrastive prompt template, applied from the target layer onward at 15× multiplier). Qwen produces 100% coherent refusals. Mistral produces 100% garbled output (repetition loops like "illegal illegal illegal...") at every layer tested (50%, 60%, 70% depth) and with both DIM and COSMIC directions (n=50).

This is not a method failure in the usual sense. The direction extraction succeeds: the vectors have reasonable norms and the contrastive separation is present in Mistral's activations. But the model's response to residual stream perturbation is categorically different from Qwen's or Gemma's. Both DIM and COSMIC produce garbled output on Mistral; neither produces normal (unsteered) responses.

We can enumerate possible explanations but cannot distinguish between them with our current data:

Architectural hypothesis. Mistral uses sliding window attention rather than full attention. This likely matters more for intervention propagation than for extraction itself. Our extraction step reads residual activations before subsequent attention updates, so sliding-window mechanics should not by themselves eliminate a direction at readout time. But once we inject from layer l through N, attention structure can shape how that perturbation is amplified, damped, or redirected. This leaves the architectural hypothesis plausible for response dynamics, but insufficient as a standalone explanation of the extraction discrepancy.

Alignment training hypothesis. Mistral's instruction tuning may distribute refusal behavior differently, across attention heads rather than in the residual stream, or via a mechanism that is not well-approximated by a single linear direction. We lack access to Mistral's training details to evaluate this.

Sensitivity hypothesis. Mistral may have larger residual stream norms or different layer normalization that makes the same multiplier effectively larger relative to the signal, pushing it into the garbled regime. The fact that both DIM and COSMIC produce garbled (not normal) output suggests the model is being disrupted rather than steered: the intervention is strong enough to break generation coherence but not targeted enough to redirect it.

The practical implication is unambiguous: activation steering is not architecture-universal, and any deployment should include validation on the target architecture. The mechanistic implication is more provocative: if refusal is genuinely "mediated by a single direction" in some architectures but not others, then either the single-direction finding is architecture-specific, or the direction exists in Mistral but our extraction and application protocol cannot access it. Distinguishing these hypotheses requires probing Mistral's refusal representations with different methods: sparse autoencoders, circuit-level analysis, or nonlinear steering approaches.

7. Tooling Sensitivity as a Methodological Finding

Most activation steering papers treat their extraction tooling as transparent, an implementation detail that doesn't affect results. We found otherwise, at least on one model.

On Qwen 7B, the same DIM algorithm implemented via nnsight's tracing API versus standard PyTorch forward hooks produces directions with 100% versus 10% coherent refusal rate, with the same model, same contrastive data, same target layer, same multiplier. The difference is entirely in how activations are captured during the extraction forward pass.

We discovered this through a debugging session when inconsistent results across scripts led us to isolate the extraction method as the variable. The likely mechanism involves how standard hooks interact with in-place operations in the computational graph: hooks may capture activations that have been modified by subsequent in-place operations or that reflect a different point in the computation than intended. nnsight's tracing approach, which instruments the model's forward pass at the graph level, avoids this. We say "likely" because we have not fully characterized the specific operation causing the divergence.

We emphasize that this finding comes from a single model (Qwen 7B); it may not generalize to other architectures or scales. This finding connects to a question that matters for the interpretability community: are the "refusal directions" extracted by these methods robust computational features of the model, or are they sensitive to implementation details in ways that suggest they occupy a narrow subspace where small perturbations in the extraction process yield meaningfully different vectors? Our result (from a single model) is consistent with the latter interpretation, but we cannot generalize from n=1.

Practical recommendation: We recommend that future activation steering work (a) specify extraction libraries and versions, (b) validate extracted directions against a known-good baseline before attributing weak steering to the method or model, and (c) report direction norms as a diagnostic. If a paper reports that steering "doesn't work" on a model, the extraction tooling should be the first thing to rule out.

8. Discussion

8.1 What Inverse Scaling Tells Us About Refusal Geometry

Our central finding, that steering effectiveness drops monotonically with model size, is the result most in need of mechanistic explanation and the one we can least confidently explain. We lay out three competing hypotheses below. We cannot distinguish between them with current data; their value is in constraining future investigation.

The distributed representation hypothesis. Elhage et al. showed that neural networks represent more features than they have dimensions by superimposing features in shared subspaces. Larger models, having greater capacity and training on more data, may represent refusal in a more polysemantic way, entangling it with related concepts (safety, ethics, uncertainty, helpfulness) that partially overlap in activation space. A single DIM vector captures the average direction of this entangled cluster, but as the cluster spreads across more dimensions, the projection onto any single direction captures a decreasing fraction of the total refusal signal. Our observation that Gemma 27B produces direction norms of 350+ (compared to 24–93 for steerable models) is consistent with this hypothesis: the extracted "direction" may be a noisy average across a high-dimensional manifold rather than a clean one-dimensional feature.

The redundancy hypothesis. Larger models may implement refusal via redundant pathways across multiple layers. Perturbing a subset of layers leaves the others to compensate. Wei et al. found that safety-critical parameters are sparse (~3% of weights), but 3% of a larger parameter space provides more room for redundant implementation. Our intervention is multi-layer in application (the same direction added from layer l through N), but it is still a single shared linear direction. Methods that learn richer per-layer or nonlinear interventions can be strictly more expressive. Under this hypothesis, multi-layer nonlinear methods like RFM succeed at scale precisely because they intervene across all pathways simultaneously.

The narrowing-window hypothesis. Our multiplier sweep on Qwen 32B reveals that the effective steering window narrows dramatically at scale: 15× works (60%), 20× partially works (20%), 25× produces garbled output (0% coherent, 90% garbled). Smaller models tolerate a wide range of multipliers; at 3B and 7B, every tested multiplier produces 100%. One speculative interpretation: larger models operate closer to the edge of a nonlinear response regime, where the intervention must be precisely calibrated (strong enough to override refusal but weak enough to preserve generation coherence).

These hypotheses are not mutually exclusive. All three may contribute, and our data cannot distinguish their relative contributions. What we can say is that the pattern (monotonic degradation across an architecture family that holds everything constant except scale) constrains the space of explanations. Whatever causes the degradation is a function of scale itself, not of architecture changes between model sizes.

Reconciling with Beaglehole et al. Our finding appears to contradict Beaglehole et al. , who report that larger models are more steerable. The apparent contradiction is methodological: their approach uses RFM (a nonlinear kernel method) with all-block steering and 768 training examples; ours uses DIM (linear mean-difference) with a single direction from one target layer and ~10 contrastive pairs. The gap between these results quantifies the scaling wall separating simple linear from complex nonlinear methods. One interpretation: the "refusal direction" that DIM extracts is a real feature at small scales but an increasingly lossy summary of a higher-dimensional structure at large scales. Nonlinear methods succeed precisely because they can represent this complexity. We note, however, that we have not run RFM ourselves; this reconciliation is inferred from published results under different experimental conditions.

8.2 Why Simple Beats Complex (and When It Won't)

The consistent parity or superiority of DIM over COSMIC across all tested conditions echoes a recurring pattern: simple baselines match complex methods when the underlying signal is strong and low-dimensional. Marks and Tegmark demonstrated precisely this for truth representations (difference-in-mean probes generalize as well as more complex classifiers).

The theoretical argument is straightforward. If refusal is genuinely mediated by a single direction , then the optimal estimator for that direction given contrastive data is the mean difference (which is exactly DIM). SVD-based methods like COSMIC extract the direction of maximum variance, which coincides with the mean shift when the signal dominates noise, but can diverge when it does not.

COSMIC's automated layer selection compounds this at scale. Its scoring function (cosine similarity agreement aggregated across layers) assumes that the correct layer will produce a direction consistent with most other layers. This holds when models are small and the refusal direction is concentrated. At 32B with 64 layers, the aggregation becomes noisy, and the scoring function selects L43 (67% depth) when the optimum is L32 (50%). A human applying the heuristic "use 50% depth for large models" outperforms the algorithm.

This does not mean complex methods are never warranted. The inverse scaling finding suggests exactly the opposite: at frontier scale, where refusal may be encoded in structures that a single linear direction cannot capture, methods like RFM that operate nonlinearly across all layers may be necessary. The lesson is not "simple is always better" but "simple methods hit a scaling wall, and COSMIC's particular form of complexity does not help clear it."

9. Mechanistic Hypotheses

We include this section in the spirit of laying out a hypothesis space that others can test, clearly labeled as speculation. We believe untested mechanistic hypotheses are more useful when stated precisely than when left implicit.

Hypothesis 1: Refusal fragmentation at scale.

The divergence between DIM and COSMIC at scale, combined with the monotonic decline in steering effectiveness, may reflect that refusal is implemented through multiple quasi-independent circuits in larger models, with DIM capturing a linear summary of the mean direction while COSMIC captures a more local feature. Testable prediction: SAE analysis of Qwen 32B should reveal multiple distinct refusal-related features where Qwen 3B has one or two. The number of refusal features should correlate with model scale.

Hypothesis 2: Mistral encodes refusal nonlinearly.

Mistral's complete failure under linear steering, combined with the fact that refusal directions can be extracted (reasonable norms, contrastive separation), suggests that Mistral may implement refusal through a mechanism that is not well-approximated by a single linear direction in the residual stream. This may be implemented through attention head-level gating or a nonlinear interaction between the residual stream and attention patterns. Testable prediction: Probing Mistral with nonlinear methods (e.g., RFM, or steering at the attention head level rather than the residual stream) should succeed where DIM fails. If it does not, the failure is more likely an extraction artifact than a representational difference.

Hypothesis 3: The "refusal direction" is a low-rank artifact at small scales.

DIM's perfect performance at small scales (100% at 3B and 7B across all tested conditions) may reflect not that refusal is cleanly one-dimensional, but that small models have limited representational capacity and must compress refusal into a low-dimensional subspace. The "single direction" finding may be a property of model scale as much as a property of how refusal is implemented. Testable prediction: If this hypothesis is correct, training a deliberately larger model on the same data as a small model (same behavior, more capacity) should produce a refusal representation that is harder to steer with DIM, even controlling for training data and procedure.

Hypothesis 4: Extraction tooling sensitivity indicates feature fragility.

The large gap between nnsight-extracted and hook-extracted directions (100% vs 10% on Qwen 7B) may indicate that the "refusal direction" lives in a narrow subspace where small numerical perturbations in the extraction process produce meaningfully different vectors. If so, the direction is not a robust computational feature but a fragile geometric artifact. Testable prediction: Computing DIM directions from multiple independent contrastive datasets should produce directions with high variance in cosine similarity. If the direction is robust, cosine similarity across extraction runs should exceed 0.95; if fragile, it should be substantially lower.

10. Implications for Safety

These results bear directly on the viability of representation engineering as a safety tool, and the implications sharpen with model scale.

The scaling problem. If linear activation steering degrades monotonically with model size, and if this degradation reflects genuine changes in how larger models represent refusal, then representation engineering approaches that rely on single linear directions become less reliable precisely at the scales where safety matters most. Our data covers 2B–32B parameters. Frontier models are 10–100× larger. Extrapolating our scaling curve suggests that single-direction steering would be minimally effective at frontier scale without methodological advances, though extrapolation from four data points in one architecture family is highly speculative.

The architecture problem. The Mistral failure demonstrates that steering is not architecture-universal. For safety applications, this means that any steering-based monitoring or intervention system must be validated per-architecture; there is no guaranteed transfer. This is a practical constraint that limits the generality of representation engineering as a safety paradigm.

The tooling problem. If extraction tooling can cause a 90-percentage-point swing in steering effectiveness (at least on one model), then the reproducibility of representation engineering results is in question. Safety-critical applications require reproducible interventions, and our finding suggests that the field's current level of implementation specificity may be insufficient.

A more optimistic reading. The inverse scaling finding does not mean representation engineering is doomed at scale. It means that simple representation engineering (single linear direction, applied from one layer onward) hits a wall. Nonlinear methods like RFM appear to clear this wall. The question is whether the added complexity of these methods is compatible with the transparency and interpretability that make representation engineering appealing for safety in the first place. A method that works but is opaque may not be more useful for safety than a method that fails transparently.

11. Cross-Model Transfer of Refusal Directions

The preceding sections established that activation steering effectiveness depends on model scale and architecture. A natural follow-up question: do the extracted refusal directions transfer across models? If the "refusal direction" captures a shared representational feature, directions extracted from one model should steer another model in the same family, or even across families with matched hidden dimensionality. We test both scenarios.

Protocol

We extract DIM refusal directions from a source model and apply them at the corresponding relative depth in a target model, using the same contrastive data and multiplier calibration protocol as Phase-1. We measure coherent refusal rate on the target and compute transfer efficiency (TE): the ratio of the transferred direction's coherent refusal rate to the target model's native (self-extracted) rate. TE ≥ 1.0 means the transferred direction steers at least as well as the target's own. We also report the cosine similarity between source and target directions projected into matched dimensionality (cross-cos), as a geometric diagnostic.

Results

Transfer Direction	Coherent Refusal Rate	Transfer Efficiency (TE)	Cross-Cosine
Same-family: Qwen 14B ↔ 32B
Qwen 14B → Qwen 32B	100%	1.25	0.324
Qwen 32B → Qwen 14B	96.7%	1.00
Cross-family: Qwen 7B ↔ Gemma 9B (hidden_dim = 3584)
Qwen 7B → Gemma 9B	Fails	0.17	0.019
Gemma 9B → Qwen 7B	Fails	0.03

Key finding. In this protocol and tested pairs, same-family transfer remains strong while cross-family transfer collapses despite matched hidden dimensionality. Same-family transfer within the Qwen family (14B ↔ 32B) achieves TE ≥ 1.0 in both directions, with the smaller-to-larger direction (14B → 32B) reaching 100% coherent refusal (TE = 1.25). Cross-family transfer between Qwen 7B and Gemma 9B collapses to near-zero effectiveness (TE = 0.17 and 0.03) despite both models sharing hidden_dim = 3584.

Figure 9: Transfer efficiency (TE) of DIM refusal directions. Same-family (Qwen, green) achieves TE ≥ 1.0; cross-family (Qwen↔Gemma, red) collapses to near-zero. Do not assume directions transfer across architecture families. Validate on your target model. One pair tested in each category.

Interpretation

The cross-cosine values are suggestive: same-family directions share moderate geometric alignment (0.324) while cross-family directions are near-orthogonal (0.019). This is consistent with the hypothesis that refusal directions are family-specific representations shaped by shared pretraining data and fine-tuning procedures, not universal geometric features of instruction-tuned models. However, cross-cosine is a coarse diagnostic and the alignment values should not be over-interpreted.

The asymmetry within cross-family transfer (TE = 0.17 vs. 0.03) suggests that whatever residual signal transfers from Qwen to Gemma does not transfer in the reverse direction, consistent with the directions encoding family-specific rather than shared structure.

Transfer Coverage Matrix

The table below shows which model pairs were tested for direct behavioral transfer and which were excluded. Direct vector transfer requires matching hidden dimensionality; dimension-mismatched pairs would require explicit projection, introducing a confound we chose to avoid.

Source ↓ / Target →	Q-3B (2048)	Q-7B (3584)	Q-14B (5120)	Q-32B (5120)	G-9B (3584)
Qwen 3B (2048)	self	mismatch	mismatch	mismatch	mismatch
Qwen 7B (3584)	mismatch	self	mismatch	mismatch	tested
Qwen 14B (5120)	mismatch	mismatch	self	tested	mismatch
Qwen 32B (5120)	mismatch	mismatch	tested	self	mismatch
Gemma 9B (3584)	mismatch	tested	mismatch	mismatch	self

Key constraint. Within the Qwen family, only 14B and 32B share hidden dimensionality (5120), making them the only same-family pair eligible for direct transfer without projection. Similarly, Qwen 7B and Gemma 9B are the only cross-family pair sharing hidden dimensionality (3584). All behavioral transfer conclusions are anchored on these two dim-matched pairs. Dimension-mismatched pairs were not used for behavioral transfer and no claims are made about them.

Caveats

These results come from one same-family pair (Qwen 14B ↔ 32B) and one cross-family pair (Qwen 7B ↔ Gemma 9B). We do not claim these findings are universal. Same-family transfer may not hold for other families (e.g., Gemma 9B ↔ 27B) or at larger scale gaps. Cross-family failure may not generalize to all cross-family pairs. The cross-cosine metric is suggestive of geometric structure but is not a causal explanation for transfer success or failure. A broader transfer matrix across multiple families and scales is needed before drawing general conclusions.

What Transfer Tells Us About Refusal Geometry

The transfer results add a dimension to the geometric picture established in earlier sections: same-family transfer succeeds (TE ≥ 1.0) despite moderate cross-cosine (0.324), suggesting that models within an architecture family encode refusal in overlapping geometric subspaces, different enough in orientation to produce low cosine similarity but functionally aligned enough for transferred directions to induce refusal. Cross-family transfer fails (TE ≤ 0.17) with near-orthogonal directions (cross-cosine 0.019), even when hidden dimensionality matches exactly.

This pattern is consistent with the hypothesis that refusal geometry is shaped by architecture-specific training dynamics rather than by a universal geometric feature. If refusal were encoded in a geometry determined primarily by training data (which overlaps substantially across families), we would expect at least moderate cross-family transfer. The near-orthogonality instead suggests that how an architecture processes and stores refusal information during training determines the resulting direction, not just what information is stored. We caution that this interpretation rests on one same-family pair and one cross-family pair.

12. Limitations

We have stated caveats inline throughout the paper where they are most relevant. This section collects them systematically for readers who want the complete accounting.

Sample size and statistical power. Our primary metric is the coherent refusal rate over 30 benign test prompts with greedy decoding (temperature = 0). Greedy decoding eliminates sampling variance, making each prompt a deterministic binary outcome. For n=30, 95% Wilson score confidence intervals are approximately ±13 percentage points for rates near 50%, and ±12pp for rates near 100%. For the 50-prompt sweeps, intervals are narrower: ±10pp near 50%, ±7pp near 100%. The large effects we report (100% vs 0%, or 100% vs 60%) survive this uncertainty; intermediate comparisons (e.g., 83% INT8 vs 77% FP16 at 32B) are not statistically distinguishable. We report point estimates in prose (rounded) and precise values in tables, and caution against over-interpreting small differences. The scaling comparison (100% at 3B vs 60% at 32B, n=50) is significant (Fisher's exact test, p = 0.005).

Single behavior and direction. We evaluate only the induction of false refusals on benign prompts, not the suppression of refusal on harmful prompts. The relationship between these two directions of steering may not be symmetric. We study only refusal. Steering for other safety-relevant behaviors (sycophancy, honesty, toxicity) may exhibit different scaling patterns, different architecture dependencies, and different sensitivity to quantization. Refusal may be unusually amenable to single-direction steering ; other behaviors may be inherently multi-dimensional.

Architecture coverage. Two working architecture families (Qwen, Gemma) and one failure (Mistral) from three families tested. Llama was excluded due to a technical failure in rope configuration under nnsight, and Phi due to nnsight incompatibility. Our conclusions about architecture dependence rest on n=3 families, with n=1 for the failure case. This is sufficient to demonstrate that architecture matters but insufficient to characterize which architectural features predict steerability.

DIM vs COSMIC fairness. Our comparison gives DIM a structural advantage: DIM's layer is selected by sweeping across layers and choosing the best, while COSMIC uses automated selection. A fairer comparison would give COSMIC the same human-in-the-loop optimization, but this would defeat COSMIC's primary selling point (automation). We report COSMIC's automated performance as the relevant comparison for practitioners, while acknowledging that COSMIC with manual layer override would likely match DIM.

Greedy decoding only. Real deployments use temperature > 0, which introduces sampling variance that could interact with steering. We chose greedy decoding for reproducibility but note this limits ecological validity.

Multiplier optimization coverage. We did not globally optimize multipliers for every model-layer condition. We used family defaults with targeted sweeps in key cases (for example, Qwen 32B and Gemma 9B). Some failures in larger models may therefore reflect suboptimal gain selection, not only representational nonlinearity. This risk is partly mitigated by the explicit Qwen 32B sweep and Gemma 9B 15× vs 25× comparison, but it is not eliminated.

Extraction tooling dependency. Our finding that nnsight and raw hooks produce different directions was tested on one model (Qwen 7B). We have not characterized which aspects of the extraction process cause the divergence, nor have we tested other extraction libraries (TransformerLens, Baukit). This finding should be treated as a flag for the community to investigate, not as a general conclusion. A direct follow-up study is repeated identical-run extraction per method (same prompts, layer, and multiplier) with variance reporting, plus tensor-site parity checks across tooling paths to separate true method differences from reproducibility noise.

Manual classification. Our 3-tier output classification (coherent refusal / garbled / normal) was performed by a single rater without formal inter-rater reliability measurement. For the effect sizes we report (differences of 30+ percentage points), classification ambiguity at the margins does not affect conclusions. We provide example outputs at each tier in Appendix C.

Contrastive dataset sensitivity. Our direction extraction uses a fixed set of ~10 harmful/harmless contrastive pairs (listed in Appendix A). We do not study sensitivity to the choice of extraction prompts, the number of examples, or the diversity of harmful categories.

No mechanistic validation. We observe that steering effectiveness decreases with scale but do not provide causal evidence for why. Our hypotheses (§9) are speculative and untested. Mechanistic interpretability tools (sparse autoencoders , circuit analysis, causal interventions at the attention head level) could provide the missing evidence. We consider this the most important direction for future work on this topic. A second concrete follow-up is cross-architecture transfer: extract refusal directions within one family and apply them across other families at matched depths/scales to test whether refusal directions are family-specific or partially universal, and whether shared pretraining or SFT data improves transfer.

Instruction-tuned models only. We test only instruction-tuned (chat) model variants, as these are the models that exhibit refusal behavior. Base models may have different steering properties.

13. Conclusion

We set out to understand when and why activation steering works for modifying refusal behavior, and found that the failures are at least as informative as the successes.

The inverse scaling pattern (steering gets harder as models grow) suggests that the "single refusal direction" picture, while valid at small scales, may be an increasingly lossy description of how larger models implement refusal. The Mistral failure tells us that steerability is not a universal property of instruction-tuned models but depends on architectural details we do not yet understand. The tooling sensitivity finding reminds us that the directions we extract are mediated by implementation details that the literature rarely specifies.

For practitioners: use DIM, use nnsight (or equivalent graph-level tracing), start at 50% depth for large models, validate on your target architecture before assuming transfer, and avoid INT4 quantization for models above 14B parameters if steering accuracy matters.

For researchers: the pattern of results here (monotonic scaling degradation, architecture-dependent failure, tooling sensitivity) points toward specific questions about the geometry of refusal that we think are worth pursuing. We have stated four testable hypotheses (§9). We hope someone will test them.

Figure 6: Complete layer profiles across all models show systematic shifts in optimal depth with scale and architecture.

Figure 7: Direction norm correlates with steering failure. Norms above 100 predict breakdown, with Gemma 27B showing extreme norms (350+) corresponding to complete failure.

Figure 8: Quantized directions maintain high cosine similarity to FP16 (>0.97), yet functional performance diverges at scale, indicating that geometric preservation does not guarantee functional preservation.