Internal LLM analysis separates risk recognition from refusal generation

Distinguishing between recognizing harm and writing the refusal

Safety-aligned large language models remain vulnerable to jailbreaks and can over-refuse benign requests. To understand these failures, it is necessary to distinguish between the recognition of a request's harmfulness and the computation that generates the refusal response.

The authors of the study reframe this problem as an internal pathway question within the model. They propose a measurement in the residual-stream updates of decoder-only Transformers, separating the harmfulness information that a module update exposes from the change it produces in readable refusal evidence.

This distinction is critical because conflating the internal recognition of risk with the final output generation obscures the specific mechanisms that lead to safety failures. By isolating these two processes, the study provides a clearer framework for understanding why a model might correctly identify a harmful intent but fail to generate an appropriate refusal, or conversely, why it might refuse a benign request without a clear internal signal of harm. This separation allows for more precise diagnostics of model behavior, moving beyond simple input-output analysis to examine the internal computational pathways that drive safety-aligned responses.

The reframing of the problem as an internal pathway question shifts the focus from external prompt engineering to the internal architecture of the model. This approach acknowledges that the safety alignment process does not simply add a layer of refusal on top of the base model, but rather modifies the internal representations and computations in specific ways. Understanding these modifications is essential for developing more robust safety mechanisms that can distinguish between genuine threats and benign requests, thereby reducing both jailbreak vulnerabilities and over-refusal issues.

Internal Signals: Attention and MLP Layers

The analysis focuses on attention updates and Multi-Layer Perceptron (MLP) layers. The results indicate that information about harmfulness is more easily decoded from attention updates, especially in the model's early layers.

On the other hand, MLP updates contribute more strongly to the selective writing of refusals in aligned models. This suggests a functional division where attention acts as a sensor and MLP layers as the execution mechanism for the safety response.

The functional division between attention and MLP layers suggests a hierarchical processing model where early layers are responsible for feature extraction and risk assessment, while deeper layers handle the decision-making and response generation. This division of labor is consistent with the general understanding of transformer architectures, where attention mechanisms facilitate the integration of contextual information, and MLP layers perform non-linear transformations that can encode complex decision boundaries. In the context of safety alignment, this means that the model's ability to recognize harm is primarily a function of its attention mechanisms, while its ability to refuse is a function of its MLP layers.

The concentration of harmfulness decoding in early attention layers implies that the model's initial assessment of a request's safety is formed quickly and is relatively stable throughout the processing pipeline. This early assessment is then passed to deeper layers, where it is combined with other contextual information to produce a final decision. The fact that MLP layers are more strongly associated with refusal writing suggests that the decision to refuse is not simply a direct output of the harmfulness assessment, but rather a more complex process that involves additional computations and transformations.

Sample Scope: 18 Model Pairs

The study evaluates 18 Base-Instruct model pairs spanning the Qwen, Llama, and Gemma families. This selection allows for the observation of common patterns in widely used decoder-only architectures.

It is observed that post-training strengthens refusal writing in MLP layers across all 18 pairs, without systematically improving harmfulness decodability. Additionally, positive MLP writing becomes more concentrated in deeper layers after post-training.

The evaluation of 18 Base-Instruct model pairs across the Qwen, Llama, and Gemma families provides a broad and representative sample of widely used decoder-only architectures. This diversity in model families and sizes allows the study to identify common patterns in the internal pathways of safety-aligned models, suggesting that the observed functional division between attention and MLP layers is not an artifact of a specific model architecture or training procedure. The consistency of these patterns across different model families increases the generalizability of the findings and their applicability to other decoder-only models.

The observation that post-training strengthens refusal writing in MLP layers without systematically improving harmfulness decodability highlights the specific impact of safety alignment on the model's internal representations. This suggests that the safety alignment process primarily modifies the decision-making components of the model, rather than its risk assessment capabilities. This finding has important implications for the development of future safety alignment techniques, as it suggests that improvements in harmfulness recognition may require different approaches than those used to enhance refusal writing.

Causal Interventions and the Sentinel Method

Causal interventions support this pathway: resetting downstream MLP updates to their clean values substantially reduces the refusal shift caused by upstream attention interventions. This confirms the functional dependency between both modules.

Based on these findings, the authors introduce Sentinel, a method that combines an attention-derived perception score (P) with an MLP-derived decision score (D). Conditional routing using both signals achieves a lower aggregate error rate than single-signal routing across eight evaluated models.

The causal interventions performed in the study provide strong evidence for the functional dependency between attention and MLP layers in the safety pathway. By resetting downstream MLP updates to their clean values and observing the reduction in refusal shift caused by upstream attention interventions, the study demonstrates that the MLP layers are not merely passive recipients of information from the attention layers, but active participants in the generation of the refusal response. This causal relationship is crucial for understanding the internal dynamics of safety-aligned models and for developing more effective safety mechanisms.

The introduction of the Sentinel method represents a practical application of the findings from the study. By combining an attention-derived perception score with an MLP-derived decision score, Sentinel provides a more comprehensive and accurate assessment of a model's safety behavior than single-signal routing. The lower aggregate error rate achieved by Sentinel across eight evaluated models demonstrates the effectiveness of this approach and its potential for use in real-world safety evaluation and monitoring. The conditional routing based on the joint profile of P and D allows for more nuanced and context-sensitive safety decisions, reducing both false positives and false negatives.

Implications and Study Limitations

In six models with reasoning capabilities, harmfulness remains decodable during reasoning, while strong refusal writing concentrates at the onset of the answer. This pattern persists even with reasoning disabled.

It is important to note that this is a preprint and not a peer-reviewed work. The results do not demonstrate that jailbreaks are resolved, but rather offer a tool to diagnose failures. Full reproducibility and robustness against all prompt variants require additional verification beyond the preprint abstract.

The findings regarding models with reasoning capabilities suggest that the internal pathways for safety alignment are robust to changes in the model's processing mode. The fact that harmfulness remains decodable during reasoning, while strong refusal writing concentrates at the onset of the answer, indicates that the model's safety mechanisms are not dependent on the specific reasoning process, but rather are integrated into the core architecture of the model. This robustness is an important feature for safety-aligned models, as it ensures that the safety mechanisms remain effective even when the model is engaged in complex reasoning tasks.

The limitations of the study, including its status as a preprint and the lack of peer review, highlight the need for further verification and replication of the findings. While the study provides valuable insights into the internal pathways of safety-aligned models, the conclusions should be interpreted as hypotheses supported by preliminary data. Full reproducibility and robustness against all prompt variants require additional verification beyond the preprint abstract, and independent replication of the results is essential for establishing the validity and generalizability of the findings. The study's focus on a specific set of model families and architectures also limits the generalizability of the findings to other types of models and architectures.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.