Long-Transduction: A Controlled Diagnostic for Measuring Drift in Long-Horizon AI Agents

The Reliability Problem in Agentic Workflows

Long-horizon agentic workflows require language models to sustain repeated actions that depend on the current state of the system. As context grows, sub-task complexity changes, and new data arrives, each situation represents an independent axis along which an agent may fail.

An illustrative example is an agent reconciling a long ledger: it must repeatedly read its state, update the correct record, and preserve alignment across thousands of outputs. A model may accept the entire ledger yet lose its place or stop applying the operation consistently as generation proceeds.

The central difficulty is not simply processing a large input once. The agent must preserve a usable representation of the current state while generating a sequence of outputs, each of which can affect what should happen next. A mistake can therefore arise even when the relevant information appeared earlier in the context: the challenge is continuing to apply the right operation to the right state as the sequence unfolds.

The three failure axes described here are related but distinct. More context increases the amount of information accumulated; changes in data format alter how that information is presented; and greater local complexity raises the demands of an individual operation. Keeping these factors separate matters because a failure associated with one axis need not establish that the others caused it.

Constructing Long-Transduction: Controlled Variables

Long-Transduction is a controlled diagnostic that tests a model's ability to stay on task during long generation. The test involves continuously reading, mutating, and outputting input-context dependent operations, such as arithmetic, sorting, variable lookups, and table transformations.

The evaluation independently varies three key variables: local task complexity, input data formatting, and context length. This design allows failures to be isolated along each axis separately, rather than measuring a diffuse global performance.

In this setup, the model is not being asked only to produce a single answer from a fixed prompt. It repeatedly reads input, changes or acts on it, and emits an output whose correctness depends on the context available at that point. The diagnostic consequently targets continuity across a sequence of state-dependent operations, rather than treating each operation as an isolated question.

Varying one dimension independently provides a way to interpret observed errors more narrowly. For example, a comparison across context lengths is intended to assess degradation associated with information accumulation, while a change in presentation tests robustness to format and a harder sub-task tests immediate task demand. This separation supports diagnosis, but does not by itself establish why a particular model fails or how it would behave outside the tested conditions.

Results Across Seven Open-Weight Models

The authors evaluated seven open-weight models using this diagnostic framework. The results show a 62.8% decrease in performance when scaling context length from 4K to 128K tokens. This figure suggests that the mere accumulation of context is a critical factor in degradation.

Additionally, a 36.5% decrease was observed when varying the input format, and a 39.9% drop when increasing local task complexity. Together, these failures represent critical liabilities in long-horizon agentic workflows, indicating that reliability is not linear with model size or window length.

The reported percentages describe decreases in performance associated with changes to the test conditions; they should not be read as absolute accuracy levels. In particular, the 62.8% figure concerns the change from a 4K-token context to a 128K-token context, while the other reported decreases correspond to input-format variation and increased local complexity. The abstract-based figures provide a summary of the reported results, not a complete account of how each measure was calculated.

The example that starts from 90% accuracy is illustrative rather than a reported result for one of the seven models. A percentage decrease does not, on its own, specify the resulting accuracy unless the baseline and the precise calculation are known. It therefore supports the narrower point that performance may deteriorate substantially as context length grows, without establishing an exact effective accuracy or proving that every such task becomes infeasible.

Distinguishing Diagnostic from Production Demonstration

It is fundamental to distinguish this controlled diagnostic from a performance demonstration in production environments. Long-Transduction isolates variables to understand the mechanics of failure, but it does not simulate the complete interaction with external tools, dynamic databases, or human users.

The results do not guarantee that a model failing this test will fail in a real application, nor that one passing the test is reliable in production. The test is an engineering tool to identify specific vulnerabilities, not a certification of fitness for deployment.

This distinction sets the scope of what the diagnostic can establish. By controlling context length, presentation, and local difficulty, it can expose weaknesses under defined test conditions. It does not recreate the complete environment in which an agent would interact with tools, changing databases, or people, and it cannot substitute for evaluating those interactions directly.

Passing or failing should therefore be interpreted as evidence about performance on this particular diagnostic, not as a general verdict about deployment readiness. A failure identifies a vulnerability worth investigating; a pass does not rule out failures under conditions the test does not cover. The evaluation flow is useful precisely as a way to organize targeted checks, while leaving broader reliability questions open.

Limitations and Requirements for Generalization

The preprint, published on arXiv, does not detail in the abstract the specific success metrics, comparative baselines, or the availability of code and data for replication. Full verification requires accessing the complete document to evaluate the detailed methodology.

To evaluate the generalization of these findings, execution trace data, test code, and validation in agents with tools would be needed. The authors implicitly acknowledge that these failures are critical, but the extension to real environments with complex tools remains a hypothesis pending empirical verification.

The available description leaves important parts of the evaluation unresolved. Without the specific success metrics and comparative baselines, it is difficult to determine how performance was summarized or to compare the reported decreases with other measurements. Likewise, without confirmed access to the test code, data, and execution traces, an independent reader cannot assume that the reported setup can be reproduced in full.

The error rate mentioned in the evaluation flow is a way to measure outcomes for particular operation and variable combinations, but the description does not specify in detail how the broader idea of staying on task is operationalized. That uncertainty limits how precisely the figures can be interpreted. Further verification would require the full methodology and replication materials, followed by testing in tool-using agents; until then, extending these diagnostic findings to complex real environments remains an empirical question rather than an established conclusion.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.