What evolves when we talk about harness evolution?

Section 1

An agent system can be viewed as an LLM paired with a harness: the prompts, tools, memory, skills, orchestration code, and control flow that shape how a model interacts with a task.

Recent work^{[1], [2], [3]} automates harness engineering through harness evolution, a form of text-space optimization that iteratively updates the non-parametric system around a fixed model using task feedback. In each iteration, an executor model attempts the tasks with the current harness. A proposer agent then inspects the trajectories and evaluation results to propose edits to the harness. Recent systems following this paradigm have reported intriguing gains from evolved harness code^[1], skill files^[4], and updated tool-use or orchestration patterns^[5].

However, a higher aggregate score does not reveal what changed in the system or whether the gains will persist beyond the search set (training split). The same score increase can hide different mechanisms:

These mechanisms have different implications for cost and generalization, even when they produce the same benchmark gain.

We therefore separate harness-evolution gains into three mechanisms: Overfitting, Test-Time Scaling, and Generalizable Improvement.

We define Overfitting as gains that rely on patterns specific to the optimization setting and therefore do not persist under an appropriate distribution shift. This includes exploiting dataset artifacts, or distilling task-specific knowledge from the proposer model. For example, a harness may string-match a recurring answer option if it is always correct when appears in the dataset. Similarly, on well-studied synthetic tasks, the proposer may write deterministic code tailored to the current setting without calling the executor LLM at all. Such gains raise the search-split score without improving how the executor handles new tasks. Among the three mechanisms, overfitting is therefore the most concerning and should be interpreted with the greatest caution.

Test-Time Scaling sits in the middle ground. It happens when the evolved harness spend more inference compute through, for example, retries, verification, or parallel sampling^[6]. The resulting gains may be matched by simply scaling the baseline to the same compute budget. While gains bought by extra compute may transfer to held-out tasks, whether they are worthwhile depends on the use case and the sensitivity to cost.

We define Generalize Improvement as gains that remain after accounting for overfitting and test-time scaling. These gains may come from reusable skills, better task decomposition, or methods that address common executor errors. They are most desirable because they are more likely to work on new tasks without using more compute.

Figure 1 previews our main and most striking finding: across representative benchmarks in math, coding, creativity, and agentic tasks, most search split gains are explained by Overfitting or Test-Time Scaling; and the residual generalizable improvement is often small. The transferability of evolved harnesses to held-out test sets is even more limited, as shown in the transfer analysis.

In the following sections, we will explain how we attribute the gains with Harness-Delta Attribution, directly evaluate the generalizability on held-out tests, and show that preventing overfitting remains non-trivial even with validation gating. We further discuss how harness evolution interacts with the executor model’s base capabilities and other task-specific details in the appendix.

Section 2

Harness-Delta Attribution: How Much Does Each Mechanism Contribute?

We introduce Harness-Delta Attribution (HDA) (method spec), a procedure that decomposes the score difference between a baseline and an evolved harness and quantifies the contributions of overfitting, test-time scaling, and generalizable improvement.

Let B denote the baseline harness and E the evolved harness. HDA evaluates two controlled variants: B_cc, which matches the baseline to the evolved harness's inference budget, and E_neutral, which neutralizes detected overfitting mechanisms in the evolved harness. Compute matching gives B the same per-example budget of executor calls, tool calls, or samples as E. Neutralization removes identified shortcuts, such as executor-bypass rules, hardcoded task knowledge, or recurring dataset artifacts, while preserving the rest of the evolved harness.

Let S(h) denote the score of harness h. We decompose the observed gain into three components:

Here, T is the gain recovered by matching the inference budget, O is the gain removed by neutralizing detected overfitting mechanisms, and G is the remaining gain. By construction,

Note that the estimated contribution of overfitting is necessarily a lower bound, since some may appear as implicit or entangled. Correspondingly, the residual is an upper-bound estimate of generalizable improvement. Even with this conservative attribution, HDA can reveal when identifiable shortcuts or additional compute explain a large share of the reported gain. We also evaluate transfer on held-out tasks directly in the transfer analysis.

Benchmarks and Experiment Setup

We evaluate harness evolution on representative benchmarks spanning single-turn reasoning (LiveMath^[7], CREATE^[8]) and long-horizon agentic tasks (ALFWorld^[9], SWE-Bench Verified^[10]), covering math, coding, planning, and creativity. We use Claude Code (Opus 4.8) as the proposer agent and evaluate various executor models with different sizes and capabilities, including Qwen-family models and Claude Haiku 4.5. The exact executor set varies by benchmark; see the appendix for details.

We largely follow the evolution framework introduced by Meta-Harness^[1]. Given an executor, the proposer iteratively revises the harness while keeping the executor weights fixed. Following Meta-Harness, candidate code, evaluation scores, execution traces, and proposer logs are written to a shared filesystem, which serves as the proposer’s memory across iterations: the proposer can inspect prior artifacts, diagnose failures, and use them to propose the next set of harnesses.

We additionally add validation feedback as an intuitive attempt to encourage transfer. Training evaluations write detailed per-example results and full trajectories to the shared filesystem, while validation evaluations write only aggregate scores. We retain both the best training-selected and validation-selected harnesses and evaluate them after evolution on a held-out test set. The former lets us analyze what the usual search-set gain is made of, while the latter tests whether validation feedback improves transfer.

Figure 3 · Evolution Loop and Split Usage

Pseudocode pseudocode

input:  splits {TRAIN, VAL, TEST}, executor M, proposer P, iterations N
init:   baseline B,  filesystem D ← ∅   # D stores code, scores, traces

D ← D ∪ {(B, eval(B, TRAIN), eval(B, VAL))}   # seed-eval

for t = 1 … N:
  P inspects filesystem D (prior harness code, scores, traces),
    then proposes k new harnesses {H₁, …, Hₖ}

  for H in {H₁, …, Hₖ}:
    if H passes interface validation:
      e_tr = eval(H, TRAIN)  # on train set: full trajectories + per-example results
      e_vl = eval(H, VAL)    # on val set: aggregate score only
      D ← D ∪ {(H, e_tr, e_vl)}       # written to D, visible to P next round

H_train* = argmax train score over D    # train-selected
H_val*   = argmax val score   over D    # val-selected

return eval({B, H_train*, H_val*}, TEST)  # held-out audit, never fed back

Section 3

Most gains come from overfitting or test-time scaling

In many settings, a large share of the observed gain reflects overfitting or added test-time compute rather than a more capable executor. The mix varies: ALFWorld and LiveMath are dominated by overfitting, CREATE by test-time scaling, while SWE-Bench Verified shows the clearest residual improvement.

LOCATION_PRIORS = {
 'apple':['fridge',
   'countertop',...],
 ... # 46 objects
}
if 'heat'  in t: go('microwave')
if 'clean' in t: go('sinkbasin')
# 0 LLM calls

MARKERS = [
 "a stronger result
   can be proven", ...
]
for c in choices:
 if any(m in c.text
      for m in MARKERS):
   return c.label  # 0 calls

for f in TWENTY
        _FRAMES:
 paths += llm(f)
# every path is a
# real LLM call;
# call-matched
# baseline recovers
# most of the gain

ADD_T,DEL_T = 12, 8
if adds>=ADD_T or \
   dels>=DEL_T:
 warn("patch is
  LARGE for a bug
  fix") # shrink

In ALFWorld, evolution largely replaces the executor. All five models reach 100% training success, but HDA attributes 98% of the aggregate gain to overfitting. The smallest-model harnesses become rule-based controllers that make no LLM calls, while larger-model harnesses encode similar benchmark-specific knowledge through prompts, templates, and deterministic rules. Appendix A1 →

In LiveMath, a dynamic multiple-choice benchmark for research-level mathematical reasoning^[7], evolution exploits a recurring dataset shortcut. The option “a stronger result can be proven” appears in 21 of 35 training questions and is always correct. HDA attributes 79% of the aggregate gain to this pattern. For example, the 0.8B harness improves from 5/35 to 27/35 correct, but 20 of the 22 added correct answers come from meta-option questions. Appendix A2 →

In CREATE, an associative-creativity benchmark where each prompt can have many valid answers^[8], much of the gain comes from additional inference compute. A call-matched baseline recovers 49% of the gain for Claude Haiku 4.5 and for Qwen3.6-35B-A3B, it even outperforms the evolved harness. Appendix A3 →

SWE-Bench Verified, grounded in real GitHub issues^[10], is where evolution produces the most faithful gains — but mainly for the weaker executors, which have the most headroom. The validation-selected Qwen3-4B-Instruct-2507 harness improves held-out test from 4% to 22% (HDA: 62% residual, 15% overfitting, 23% test-time scaling), and Qwen3-30B-A3B-Instruct-2507 is cleaner still — its train-selected harness lifts held-out test from 12% to 26% with almost no overfitting or added compute (O = 0%, T = 6%, G = 94%), and validation selection transfers further (12% to 36%).

The stronger executors barely improve on held-out test despite little measured overfitting or test-time scaling. Qwen3.6-27B rises from 77% to 85% on training but from 66% to 64% on test; Qwen3-Coder-30B-A3B-Instruct rises from 56% to 69% on training but its train-selected harness drops from 40% to 34% on test (validation selection recovers only part, to 44%). Here the proposer used honest scaffolding with no artifact to neutralize, yet the gain still does not transfer — a small overfitting/compute share is not the same as generalization; near the capability ceiling, the training gain is largely fit to the selection set. Appendix A4 →

Training gains often fail to transfer

We test generalizability directly on held-out test sets. As shown in Figure 5, test gains are substantially smaller than training gains across models and benchmarks in general. In 4 of the 16 settings, the train-selected harness even performs worse than the baseline on the held-out set. This confirms that large training gains often come from overfitting rather than generalizable improvement.

Validation-based selection offers limited protection against overfitting

At each iteration, we give the proposer an aggregate validation score for each candidate harness to encourage more generalizable improvement. We then evaluate the harness with the highest validation score on the held-out test set.

Compared with the train-selected harness, the val-selected harness improves test performance by +8.0% (ALFWorld), +0.8% (LiveMath), +7.3% (SWE-Bench Verified), and −0.4 CU (CREATE) on the four benchmarks. However, the gains remain 5.9%, 11.7%, 9.3%, and 9.3 CU (on ALFWorld, LiveMath, SWE-Bench Verified, and CREATE) below the corresponding training gains. Validation feedback therefore reduces overfitting, but avoiding overfitting in harness evolution remains non-trivial.

Section 4

Harness optimization may appear symmetric to updating model weights: both modify part of the agent system to improve overall performance. But they optimize very different things. Model weights shape the executor’s capabilities, while a harness changes how those capabilities are used or supplemented through prompts, tools, control flow, and additional inference. This flexibility creates distinct evaluation challenges that an aggregate score alone cannot resolve.

Our analysis suggests several key takeaways for future research on harness evolution:

• Measure Overfitting and report held-out performance. Performance on the search set alone does not show whether the gain will generalize to unseen tasks. When the same artifacts may appear across splits, stronger distribution shifts or stronger benchmarks are needed for fair method assessment.
• Account for Test-Time Scaling. Users need to understand the additional costs to interpret the utility in practice. When cost or latency matters, practitioners should also specify constraints on costs before evolution.
• Choose benchmarks carefully for harness evolution research. Observed gains may boil down to distilling the proposer's parametric knowledge of a well known benchmark. For example, an evolved overfitting-driven harness can perform well even on held-out tasks if the proposer already knows the benchmark beyond the training split. Such gains may not transfer to new tasks or unfamiliar benchmarks.
• We need benchmarks that enable Generalizable Improvement. Easy artifacts can short-circuit the search process, pushing any system toward overfitting and low-hanging gains. This can hide meaningful differences across proposers or evolution frameworks. To support the development of better harness evolution methods, we need rich, low-artifact benchmarks and meaningful distribution shifts that make transferable improvements easier to discover and reward.

Cite

Please cite this work as:

Ding, Wenxuan. “What Evolves When We Talk About Harness Evolution?” Blog post, 2026. https://wenwen-d.github.io/blog/harness-delta-attribution/

Or use the BibTeX citation:

@article{ding2026harnessdelta,
  title = {What Evolves When We Talk About Harness Evolution?},
  author = {Ding, Wenxuan},
  journal = {wenwen-d.github.io},
  year = {2026},
  month = {August},
  url = "https://wenwen-d.github.io/blog/harness-delta-attribution/"
}

Bibliography

1. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv preprint, 2026.
   arXiv:2603.28052
2. Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, et al. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. arXiv preprint, 2026.
   arXiv:2604.25850
3. Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, et al. Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents. arXiv preprint, 2026.
   arXiv:2605.30621
4. Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv preprint, 2026.
   arXiv:2605.23904
5. Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, et al. HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry. arXiv preprint, 2026.
   arXiv:2606.14249
6. Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the Evaluation of Harness Evolution for Agents. arXiv preprint, 2026.
   arXiv:2607.12227
7. Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, and Nima Mesgarani. LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches. arXiv preprint, 2026.
   arXiv:2604.01754
8. Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li, and Greg Durrett. CREATE: Testing LLMs for Associative Creativity. arXiv preprint, 2026.
   arXiv:2603.09970
9. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. OpenReview.
   openreview.net/forum?id=0IOX0YcCdTn
10. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can Language Models Resolve Real-world Github Issues? The Twelfth International Conference on Learning Representations, 2024.
    openreview.net/forum?id=VTF8yNQM66