Appendix

TL;DR. Fully deterministic FSM — zero self.llm_call sites (the 0.8B model is deleted). A 46-object hardcoded LOCATION_PRIORS table plus a regex task parser that maps heat->microwave / cool->fridge / clean->sinkbasin solves all 50 train games. Pure artifact.

Source: alfworld_controlled_scaling/runs_default_nothink/qwen3.5-0.8b/agents/observation_tracker_agent.py

LOCATION_PRIORS = {
    'mug': ['coffeemachine','countertop','cabinet','shelf','diningtable'],
    'tomato': ['fridge','countertop','diningtable','garbagecan'],
    'soapbar': ['countertop','bathtub','toilet','garbagecan','cabinet'],
    'alarmclock': ['sidetable','desk','shelf','dresser'],
    'desklamp': ['desk','sidetable'],   # ... 46 objects -> ALFWorld receptacles
}

class ObservationTrackerAgent(AgentHarness):
    def step(self, observation, admissible_actions):
        self._update_state(observation, admissible_actions)
        self._update_world_model(observation)      # records item -> location
        if self._in_loop():
            return StepResult(action=self._break_loop(admissible_actions), ...)
        filtered = self._apply_guard(admissible_actions)
        return StepResult(action=self._pick_action(observation, filtered), ...)
        # NOTE: no self.llm_call anywhere in this file

    @staticmethod
    def _parse_task(task_description):
        if 'cool' in task:  process_steps.append(('cool', 'fridge'))
        elif 'heat' in task: process_steps.append(('heat', 'microwave'))
        elif 'clean' in task: process_steps.append(('clean', 'sinkbasin'))
        # regex pulls target obj + dest; _KNOWN_OBJECTS/_KNOWN_RECEPTACLES vocab fallback

Overfitting. RED = the whole agent. LOCATION_PRIORS is a 46-entry object->receptacle table encoding ALFWorld's spawn distribution; the parser hardcodes the transform->appliance map (heat/microwave, cool/fridge, clean/sinkbasin) and a _KNOWN_OBJECTS/_KNOWN_RECEPTACLES vocabulary; there is no self.llm_call at all (LLM-bypass, model deleted). HDA 100/0/0.

Test-time scaling. none — no extra sampling, voting, retries, or budgets; single deterministic pass per step.

Generalizable. none — measured G=0/0/0. A world-model (item->location) recall and a loop-detector (_in_loop/_break_loop) are present and are structurally task-general, but neutralizing to react_baseline reproduces baseline, so HDA attributes nothing genuine here.

TL;DR. Deterministic backward-chaining FSM built on a 48-object OBJECT_LOCATION_PRIORS table + compound-name aliases + process-appliance map; solved all 50 train. Has exactly ONE budgeted LLM rescue site (max_llm_calls=8, only when stuck) that fired 0x on train -> measured T=0.

Source: alfworld_controlled_scaling/runs_default_nothink/qwen3.5-2b/agents/bc_action_inference_agent.py

OBJECT_LOCATION_PRIORS = {
    "mug": ["coffeemachine","shelf","countertop","fridge","cabinet","sinkbasin"],
    "egg": ["fridge","countertop"], "potato": ["countertop","fridge","sinkbasin"],
    ...  # 48 objects
PROCESS_LOCATIONS = {"clean":"sinkbasin","heat":"microwave","cool":"fridge"}
COMPOUND_ALIASES = {"remote control":"remotecontrol", "butter knife":"butterknife", ...}

    def _build_search_order(self):
        priors = OBJECT_LOCATION_PRIORS.get(obj_base, [])
        # sort admissible 'go to' targets: surfaces-in-prior, high-cap, containers

    def _should_use_llm(self):
        return self.llm_calls_made < self.max_llm_calls and self.step_count > 8

    def _llm_rescue(self, observation, admissible_actions):
        self.llm_calls_made += 1        # budgeted retry pass; fired 0x on train
        prompt = f"Task: {self.task_description}\nGoal: {goal}\n...Best action number?"
        response = self.llm_call(messages, max_tokens=32)   # only self.llm_call in file
        ...
        return self._backtrack(admissible_actions)          # falls back to deterministic

Overfitting. RED = OBJECT_LOCATION_PRIORS (48 objects), PROCESS_LOCATIONS transform->appliance map, COMPOUND_ALIASES ALFWorld name table, and the prior-ordered _build_search_order. The FSM (backward-chaining over have_object/object_processed/object_placed goals) solved all 50 train deterministically. HDA 100/0/0.

Test-time scaling. AMBER = the single budgeted LLM rescue (_should_use_llm gates on max_llm_calls=8 & step>8; _llm_rescue is a retry/verify pass invoked only on loop-detection). It is the ONLY self.llm_call site and fired 0x on the train split, so measured T~=0.

Generalizable. none — measured G=0. The rescue's env-summary + annotated-actions error-feedback prompt would be model-in-the-loop scaffolding if it ever fired, but it did not, so it contributes no genuine transfer.

TL;DR. Keeps the 4B LLM in the loop every step, but the SYSTEM_PROMPT literally hands it the ALFWorld PROCEDURES (heat->microwave etc.) and a hardcoded SEARCH_PRIORITY, and _force_critical overrides the LLM on all transform/place steps. G=0.04 (n.s.) is the only genuine slice — the per-step model-in-the-loop action selection.

Source: alfworld_controlled_scaling/runs_default_nothink/qwen3.5-4b/agents/cyclic_multi_object.py

SYSTEM_PROMPT = """You are an expert agent in the ALFWorld ... environment.
PROCEDURES you must follow:
- To clean: take it -> go to sinkbasin -> "clean X with sinkbasin 1" -> put
- To heat:  take it -> go to microwave -> "heat X with microwave 1" -> put
- To cool:  take it -> go to fridge -> "cool X with fridge 1" -> put"""
SEARCH_PRIORITY = ['countertop','diningtable','coffeetable','sidetable','desk',
                   'shelf','stoveburner','fridge','microwave','cabinet','drawer','safe']

    def step(self, observation, admissible_actions):
        forced = self._force_critical(observation, admissible_actions)
        if forced: return StepResult(action=forced, ...)   # deterministic override
        if self.phase == 'find' and self.search_steps > 12:
            self.search_mode = 'llm_fallback'      # 2nd LLM pass when search stalls
            if self.search_mode == 'llm_fallback':
                return self._llm_search_step(observation, admissible_actions)
        messages = [{"role":"system","content": SYSTEM_PROMPT}, ...] + self.history[-12:]
        response = self.llm_call(messages, max_tokens=512)   # model picks each step
        action = self._extract_action(response, safe_actions, admissible_actions)

Overfitting. RED (O=93) = the SYSTEM_PROMPT that spells out ALFWorld transform PROCEDURES (heat/microwave, clean/sinkbasin, cool/fridge) with the exact 'X with appliance 1' grammar, the hardcoded SEARCH_PRIORITY receptacle order, the _parse_task appliance/verb map, and _force_critical which deterministically overrides the LLM on take/transform/goto-dest/place — so the LLM's freedom is heavily fenced.

Test-time scaling. AMBER = the llm_fallback second pass: after 12 stalled search steps the agent switches to _llm_search_step, an extra retry/rescue LLM call with a name-disambiguation prompt (2 self.llm_call sites total).

Generalizable. GREEN (G=0.04, n.s.) = genuine model-in-the-loop action selection — the 4B LLM is queried every step to choose from the admissible list (_extract_action resolves its output), plus the task-general multi-object cycling (re-enter find after each place) and oscillation loop-break. This is the small slice HDA could not attribute to artifact, but it is not statistically significant.

TL;DR. Phase-controller FSM (find/transform/goto_lamp/goto_dest) driven by a 44-object SEARCH_PRIORITY table, hardcoded transform routes, and a regex task parser (LLM parse only as a tie-scored fallback). Neutralizing it reproduces react_baseline exactly -> HDA 100/0/0.

Source: alfworld_controlled_scaling/runs_default_nothink/qwen3-8b/agents/progress_monitor_agent.py

SEARCH_PRIORITY = {
    'mug': ['coffeemachine','cabinet','shelf','countertop','diningtable','sinkbasin'],
    'book':['shelf','desk','dresser','sidetable','coffeetable','armchair','sofa'],
    ...  # 44 objects
CONTAINERS = {'drawer','fridge','safe','microwave','cabinet'}

    def _do_transform(self, admissible_actions):
        if self.transform_type == 'heat':  # -> 'heat ...' else go to microwave
        elif self.transform_type == 'cool':  # -> go to fridge
        elif self.transform_type == 'clean': # -> go to sinkbasin

    def reset(self, task_description, admissible_actions):
        regex_parse = self._enhanced_regex_parse(task_description)
        llm_parse = self._llm_parse(task_description)          # only self.llm_call
        # pick whichever parse scores higher against the room's receptacles
        parsed = regex_parse if regex_score >= llm_score else llm_parse
        self.last_progress_step = 0        # progress monitor: recover after 8 stale steps

    def _is_stuck(self):
        return (self.step_count - self.last_progress_step) >= 8

Overfitting. RED = the whole agent: 44-object SEARCH_PRIORITY table, the transform-route switch (heat->microwave/cool->fridge/clean->sinkbasin), the CONTAINERS set, and the regex task parser. The one LLM call (_llm_parse) is only a fallback whose output is discarded whenever the regex parse scores >= it. HDA note: neutralized == react_baseline exactly.

Test-time scaling. none — the dual regex-vs-LLM parse is a single one-shot parse at reset (not repeated sampling/voting); no retry or budget on the action policy.

Generalizable. none — measured G=0/0/0. The progress-monitor stall recovery (_is_stuck / _recover_from_stuck) and loop-break are task-general in form, but stripping the hardcoded priors/phases drops the agent exactly to baseline, so nothing genuine survives.

TL;DR. The most LLM-reliant winner (4 self.llm_call sites), but still scaffolded by a hardcoded OBJECT_LOCATION_PRIORS table, regex goal templates with hardcoded tool instances ('microwave 1'/'sinkbasin 1'/'fridge 1'), and a take-action guard. G=0.02 (n.s.) is the genuine LLM goal-decomposition + typed scaffolding for unseen tasks.

Source: alfworld_controlled_scaling/runs_default_nothink/qwen3.6-35b-a3b/agents/typed_goal_scaffold.py

OBJECT_LOCATION_PRIORS = {
    "countertop": ["cup","mug","plate","bowl","fork","apple","tomato", ...],
    "fridge": ["apple","tomato","lettuce","potato","egg", ...],
    ...  # receptacle -> likely objects, ~17 receptacles

    def _parse_goals_regex(self):
        m = re.search(r"put a hot (\w+) (?:in|on) (.+)", task)
        if m: self.goals = [{"type":"find_take","object":m.group(1)},
            {"type":"transform","verb":"heat","tool":"microwave 1"},
            {"type":"place","destination":m.group(2)}]   # hardcoded tool instances

    def _parse_goals_llm(self):                       # used when regex misses (unseen types)
        response = self.llm_call([...GOAL_GEN_PROMPT...], max_tokens=200)
        goal_subtype = classify_goal(line)            # search/navigate/interact/...

    def step(self, observation, admissible_actions):
        if stall >= self.STALL_THRESHOLD:
            self.active_reflection = self.llm_call([...REFLECTION_PROMPT...])  # retry
        response = self.llm_call(messages, max_tokens=512)         # per-step decision
        action = self._resolve_action(response, admissible_actions) # +REPROMPT if invalid
        action = self._validate_take_action(action, admissible_actions) # take guard

Overfitting. RED (O=95) = OBJECT_LOCATION_PRIORS receptacle->object table + get_search_priority, the regex goal templates that inject hardcoded tool instances ('microwave 1'/'sinkbasin 1'/'fridge 1'), and _validate_take_action (the take-action guard that forces the search prior if the LLM tries to grab a wrong-typed object).

Test-time scaling. AMBER = extra LLM passes beyond the per-step decision: the stall-triggered REFLECTION_PROMPT self-diagnosis call and the REPROMPT_TEMPLATE re-ask when the model emits a non-admissible action (retry/verify passes). 4 self.llm_call sites total: goal-gen, per-step, reflection, reprompt.

Generalizable. GREEN (G=0.02, n.s.) = genuine model-in-the-loop scaffolding: _parse_goals_llm decomposes UNSEEN task types into typed sub-goals (classify_goal) with targeted per-type hints, and _resolve_action is an undefined-action guard that fuzzily maps the LLM's free-text output back to an admissible action. Small and not statistically significant.

TL;DR. Hardcoded META_MARKERS string-match early-returns the hedge option with ZERO LLM calls (the meta-option is gold on 21/35 train); only non-hedge items reach the 3-prompt generate-and-vote fallback. Pure dataset shortcut + a thin TTS wrapper, no generalizable content.

Source: livemath_scaling/runs/qwen3.5-0.8b/agents/generate_multi_prompt_vote_meta.py

META_MARKERS = [
    "one of the remaining options is correct",
    "stronger result can be proven",
    "stronger result can be proved",
    "a stronger result holds",
]

def solve(self, item):
    choices = item["choices"]
    meta_label = self._find_meta(choices)
    if meta_label:                       # fires on 21/35 train items
        return SolveResult(              # EARLY RETURN, 0 llm calls
            predicted_label=meta_label,
            reasoning=f"META_DETECTED: {meta_label}")
    # --- only non-hedge items reach here ---
    gen_configs = [SCOPE, CONDITIONS, CONCLUSION]   # 3 orthogonal prompts
    votes = []
    for sys_prompt, user_template, name in gen_configs:
        generation = self.llm_call(gen_msgs, max_tokens=200)
        match_response = self.llm_call(match_msgs, max_tokens=32)
        votes.append(label)
    counts = Counter(votes)    # majority vote across 3 generations

def _find_meta(self, choices):
    for c in choices:
        if any(m in c["text"].lower() for m in META_MARKERS):
            return c["label"]

Overfitting. RED = the whole hedge-detection shortcut: hardcoded META_MARKERS phrase list + _find_meta lexical scan + the early `return SolveResult(meta_label)` that fires with ZERO llm_call (first llm_call is at line 125, after the return at line 97). Because the meta-option is the gold label on 21/35 train items, this string-match alone captures the dominant train signal — the HARD artifact behind 91.

Test-time scaling. AMBER = the fallback for non-hedge items only: 3 orthogonal generation prompts (scope/conditions/conclusion), each matched via a 2nd llm_call, aggregated by Counter majority vote. Extra test-time sampling (=9 in the O/T/G).

Generalizable. none — G=0. Everything transferable is downstream of the string-match gate; the vote path is generic sampling, not task-general scaffolding.

TL;DR. Regex meta-detection on option A + a SYSTEM prompt that explicitly names 'Option A is a META-OPTION', plus reversing the option order (E..A) for non-meta items to exploit the label-leak at shuffled position A. The genuine 36 sliver is the size-general precise-statement rubric + malformed-<answer>-tag repair guard.

Source: livemath_scaling/runs/qwen3.5-2b/agents/reverse_anchor_commit.py

META_PATTERNS = [r'stronger.*result', r'remaining.*correct',
                 r'one of the (other|remaining)', r'stronger.*prov']

SYSTEM_META = "Option A is a META-OPTION: it claims a stronger result exists..."

def is_meta_option_a(choice_a_text):
    return any(re.search(p, choice_a_text.lower()) for p in META_PATTERNS)

def solve(self, item):
    choice_a = next(c for c in item["choices"] if c["label"] == "A")
    meta_detected = choice_a and is_meta_option_a(choice_a["text"])
    if meta_detected:
        system = SYSTEM_META                         # keep A-first order
    else:
        reversed_choices = list(reversed(item["choices"]))  # E,D,C,B,A
        system = SYSTEM_NORMAL  # 'most precise, complete result; watch
                                    # quantifier scope, iff vs if-then'
    response = self.llm_call(messages, max_tokens=2048)
    full_response = fix_reasoning_tags("<answer>" + response)  # tag repair
    predicted = normalize_label(extract_answer(full_response))

Overfitting. RED = the label-leak exploit: META_PATTERNS regex on option A, is_meta_option_a, the SYSTEM_META prompt hardcoding 'Option A is a META-OPTION', and reversing option order to E..A for non-meta items (puts the frequently-correct option first in the token stream). This is the HARD artifact that captures the meta hedge + position-A leak, driving 64.

Test-time scaling. none — T=0. Single llm_call per item, no sampling/voting/retry.

Generalizable. GREEN (=36) = the size-transferable scaffolding on non-meta items: SYSTEM_NORMAL's content rubric (prefer the most precise/complete statement; watch quantifier scope forall-vs-exists, iff-vs-if-then, strict-vs-nonstrict bounds) plus fix_reasoning_tags — a malformed/unclosed <answer>-tag repair guard (output-format robustness that transfers).

TL;DR. No lexical hedge shortcut (SOFT artifact): a falsify-then-arbitrate pipeline. The artifact is that the un-falsifiable hedge survives the falsification check on 16/21 meta items (predicted=direct_winner when no flaw is found). TTS = multi-seed shuffled directs + two-ordering flaw duels. Genuine sliver = the rubric/flaw-duel confusion-pair discrimination that helps on clean items.

Source: livemath_scaling/runs/qwen3.5-4b/agents/skeptical_verify_arbiter.py

def solve(self, item):
    scores = {c: self._rubric_score(question, c) for c in choices}  # 3-criterion
    rubric_winner, rubric_valid = ...   # gap > 0
    direct_winner = self._shuffled_direct(question, choices, rng, DIRECT_SYSTEM)

    if rubric_valid and rubric_winner == direct_winner:
        predicted = rubric_winner                       # both signals agree
    elif not rubric_valid:
        direct2 = self._shuffled_direct(question, choices, rng, DIRECT_SYSTEM)
        if direct2 == direct_winner:
            has_flaw = self._falsify_option(question, opt_agreed)  # verify pass
            if has_flaw:
                alt = self._shuffled_direct(..., ELIMINATION_SYSTEM)
                predicted = self._flaw_duel(question, opt_agreed, opt_alt, alt)
            else:
                predicted = direct_winner   # un-falsifiable hedge SURVIVES
        else:
            predicted = self._flaw_duel(question, opt_a, opt_b, direct_winner)
    else:
        predicted = self._flaw_duel(question, rubric_win, direct_win, ...)

# _flaw_duel runs the compare in BOTH orderings and only flips on agreement:
def _flaw_duel(self, q, opt_a, opt_b, preferred_on_tie):
    flaw1 = self._single_flaw_compare(q, opt_a, opt_b)
    flaw2 = self._single_flaw_compare(q, opt_b, opt_a)

Overfitting. RED (SOFT, drives 78) = no string match, but the hedge option is un-falsifiable, so _falsify_option returns PLAUSIBLE and the `predicted = direct_winner` no-flaw branch keeps the hedge on 16/21 meta items — a mechanism-level shortcut, not a lexical one. It loses to a genuine compute-match on clean items.

Test-time scaling. AMBER (=11) = test-time compute: two independent seed-shuffled directs (_shuffled_direct with per-item rng), an elimination-framed 3rd direct, and _flaw_duel running _single_flaw_compare in BOTH option orderings (order-robust voting).

Generalizable. GREEN (=11) = plausibly transferable scaffolding on clean items: _rubric_score 3-criterion evaluation (quantifiers/coherence/well-formed), _falsify_option error-feedback verify pass, and _flaw_duel — a confusion-pair discrimination that pits two candidates and flips only on cross-ordering agreement.

TL;DR. Iterative 'eliminate the ONE most-likely-wrong option' over up to 4 LLM rounds, keeping the last survivor. SOFT artifact: the un-falsifiable hedge is never judged 'most likely wrong', so it survives by construction; this survivorship — not any string match — captures the meta signal. TTS = the multi-round repeated sampling. No generalizable sliver (G=0).

Source: livemath_scaling/runs/qwen3-8b/agents/sequential_discard_solver.py

DISCARD_SYSTEM = """Identify the ONE option that is MOST LIKELY WRONG.
Focus on a specific flaw: wrong quantifier, overstated conclusion,
missing/wrong condition, wrong direction of implication.
Output: <discard>LABEL</discard>"""

def solve(self, item):
    remaining = list(all_labels)
    max_rounds = len(all_labels) - 1                 # 4 rounds for 5 options
    for round_num in range(max_rounds):
        if len(remaining) <= 1: break
        ...present ALL remaining options...
        response = self.llm_call(messages, max_tokens=8192)
        match = re.search(r'<discard>([A-E])</discard>', response)
        discarded = match.group(1).upper() if match else None
        # fallbacks: eliminate/discard/remove/worst regexes...
        if discarded in remaining:
            remaining.remove(discarded)   # hedge is never 'most-likely-wrong'
    final_answer = remaining[0]              # survivor = the un-eliminated hedge

Overfitting. RED (SOFT, drives 83) = survivorship bias: the discard prompt only ever removes the option with the most concrete flaw, and the hedge ('a stronger result can be proven') has no falsifiable content, so it is never discarded and is the `remaining[0]` survivor on meta items. Mechanism-level dataset exploit, no lexical marker. Per the HDA note it loses to compute-match on clean items.

Test-time scaling. AMBER (=8) = the sequential elimination loop itself: up to 4 repeated llm_call rounds (max_tokens=8192 each) re-presenting the shrinking option set — iterated test-time compute.

Generalizable. none — G=0. The regex extraction fallbacks are task-specific parse robustness, not transferable scaffolding; measured genuine credit is zero for this model.

TL;DR. Three-phase strength-rank -> adversarial FOR/AGAINST debate -> grounded judge. Leans least on the hedge (5/21 meta items). Artifact = RANKING_PROMPT bakes a strength ontology (Partial<Full etc., meta-adjacent). Largest TTS of any winner = the multi-call debate. Genuine sliver = the judge's independent blind re-derivation that transfers to clean items.

Source: livemath_scaling/runs/qwen3.6-35b-a3b/agents/strength_adversarial.py

RANKING_PROMPT = """Rank all options from WEAKEST to STRONGEST claim.
Existence < Uniqueness < Characterization < Equivalence
Conditional < Universal   Bound < Exact value
Partial result < Full result
Identify the PROVABILITY BOUNDARY: strongest claim that should be provable."""

GROUNDED_JUDGE = """BEFORE reading the debate arguments,
first independently derive what you expect the answer to look like
(type of result, quantifiers, provable strength). Write a PREDICTION.
Then read the debate; if your prediction conflicts with both advocates,
trust your derivation."""

def solve(self, item):
    rank_response = self.llm_call(rank_messages, max_tokens=4096)  # strength ontology
    pick1, pick2 = boundary_pair(rank_response)
    adv1_response = self.llm_call(adv1_messages, ...)   # advocate FOR pick1
    adv2_response = self.llm_call(adv2_messages, ...)   # advocate AGAINST
    judge_response = self.llm_call(judge_messages, ...)  # grounded judge
    if not re.search(r'<answer>...', judge_response):
        final = self.llm_call(judge_messages, max_tokens=256)  # retry

Overfitting. RED (=57) = RANKING_PROMPT hardcodes a claim-strength ontology (Existence<Uniqueness<...<Equivalence, Partial result<Full result, 'provability boundary'). This ladder is meta-adjacent — it operationalizes the same 'a stronger result can be proven' hedge as a scoring axis, so it still leans on the benchmark's strength/hedge structure even without a string match.

Test-time scaling. AMBER (=29, the highest TTS share of any winner) = the multi-call debate: separate llm_call for ranking, an ADVOCATE_FOR pass, an ADVOCATE_AGAINST pass, the judge pass, plus a retry llm_call when the judge emits no <answer> tag. T is explicitly called out as multi-call debate.

Generalizable. GREEN (=14, the largest genuine sliver across models, on clean items) = GROUNDED_JUDGE's independent blind re-derivation: the judge first predicts the answer from the question ALONE and is told to trust that derivation over debate rhetoric — a model-in-the-loop grounding scaffold that plausibly transfers beyond this benchmark.

TL;DR. 20-frame diversity orchestration: 6 unconstrained Wave-1 cognitive frames (Professional, Biographical, Institutional, Creative Works, Mentorship, Business) then 14 orthogonal Wave-2 frames that are explicitly told to EXCLUDE the intermediaries already used by Wave 1. The 20 live llm_call generation passes are bought compute (TTS); the cross-frame dedup + orthogonal-domain coverage is the genuine diversity orchestration that beats the equal-compute pooled baseline (CU/call 4.45->2.12, HDA G=51%).

Source: create_scaling/runs/claude-haiku-4.5/agents/wide_diverse_twenty.py

# Wave 1: 6 independent cognitive frames (no constraints)
wave1_frames = [
    ("Professional", _WAVE1_PROFESSIONAL, 0.7),
    ("Biographical", _WAVE1_BIOGRAPHICAL, 0.7),
    ("Institutional", _WAVE1_INSTITUTIONAL, 0.7),
    ("Creative Works", _WAVE1_CREATIVE_WORKS, 0.7),
    ("Mentorship", _WAVE1_MENTORSHIP, 0.7),
    ("Business", _WAVE1_BUSINESS, 0.7),
]
for frame_name, template, temp in wave1_frames:
    resp = self.llm_call(messages=[{"role": "user",
        "content": template.format(**fmt)}], max_tokens=12288, temperature=temp)
    for path in extract_paths(resp).values():
        all_paths[str(idx)] = path; idx += 1
        for triple in path:
            used_intermediaries.add(str(triple[2]))  # track for cross-frame dedup

# Wave 2: 14 ORTHOGONAL frames that must avoid Wave-1 entities
exclusion_list = "\n".join(f"- {e}" for e in sorted(used_intermediaries)[:60])
for frame_cfg in _WAVE2_FRAMES:  # 6 + 14 = 20 live llm_call passes total
    prompt = _WAVE2_TEMPLATE.format(**fmt,
        exclusion_list=exclusion_list,  # forbid already-covered intermediaries
        frame_description=frame_cfg["description"], ...)
    resp = self.llm_call(messages=[{"role": "user", "content": prompt}],
        max_tokens=12288, temperature=frame_cfg["temp"])
    for path in extract_paths(resp).values():
        all_paths[str(idx)] = path; idx += 1

Overfitting. none — O=0. Every path is produced by a live self.llm_call on the frozen base LLM (solver_harness.py injects llm_call). No hardcoded triple tables, no ground-truth Wikidata QIDs, no item-ID gates, no benchmark-answer shortcuts anywhere in the file.

Test-time scaling. The two K-frame generation loops (6 Wave-1 + 14 Wave-2 = 20 sequential self.llm_call passes per item, each up to 12288 tokens) are bought test-time compute — the harness pays 20x the generation calls of a single-shot solver. This is why CU/call collapses 4.45->2.12 even as total CU rises.

Generalizable. The diversity orchestration is the genuine (green) part HDA credits (G=51%, +19.2 CU vs equal-compute pooled baseline): used_intermediaries is accumulated across Wave-1 frames and passed as an explicit exclusion_list into every Wave-2 frame, and the 14 Wave-2 frames are hand-designed to be maximally orthogonal cognitive domains (Inspiration, Interview, Competition, Performance, Philanthropy, Sports, Technology, Domestic, ...). This structured anti-redundancy + domain-coverage is task-general scaffolding, not benchmark-specific.

TL;DR. 13-call portfolio: 2 low-temp probes (entity_a facts + valid rel_b endpoints) then 5 probed generation rounds + 6 clean unprobed rounds across fixed DOMAIN_FOCUS buckets, with a confidence-filter that only keeps paths whose triples are all self-rated >=4. Despite looking like structured diversity orchestration, on 35B this 13-frame structure scores BELOW an equal-call pooled baseline (G_raw = -6.8 CU), so HDA shows 0% green — every span here is bought compute (TTS). CU/call 4.39->0.92.

Source: create_scaling/runs/qwen3.6-35b-a3b/agents/probed_endpoint_portfolio.py

# Phase A+B: two low-temp knowledge probes
entity_response = self.llm_call(messages=[{"role": "user",
    "content": _ENTITY_PROBE.format(entity_a=entity_a)}], max_tokens=1536, temperature=0.2)
endpoint_response = self.llm_call(messages=[{"role": "user",
    "content": _ENDPOINT_PROBE.format(rel_b=rel_b, entity_b=entity_b)}], max_tokens=1536, temperature=0.2)

# Phase C: 5 probed rounds (entity + endpoint context, temp=0.65)
for round_idx, domain_instruction in enumerate(_PROBED_DOMAINS):  # 5 calls
    response = self.llm_call(messages=[...], max_tokens=12288, temperature=0.65)
    filtered_paths = _parse_confidence_paths(response)  # verify pass: keep min(conf) >= 4
    for path in filtered_paths.values():
        all_paths[str(idx)] = path; idx += 1

# Phase D: 6 unprobed CLEAN rounds -> 13 llm_calls total per item
for round_idx, domain_instruction in enumerate(_UNPROBED_DOMAINS):  # 6 calls
    response = self.llm_call(messages=[...], max_tokens=12288, temperature=0.65)
    filtered_paths = _parse_confidence_paths(response)
    for path in filtered_paths.values():
        all_paths[str(idx)] = path; idx += 1

# confidence-filter (inside _parse_confidence_paths):
if conf_vals and min(conf_vals) >= 4:  # discard low-confidence paths
    high_conf_paths[str(path_idx)] = valid_triples

Overfitting. none — O=0. All 13 calls are live self.llm_call on the frozen base LLM. The 'known facts' and 'confirmed endpoints' come from the model's own probe outputs (Phase A/B llm_call), NOT from a hardcoded table or dataset labels; no item-ID gating, no ground-truth injection.

Test-time scaling. Everything measured here is bought compute: 2 probe calls + 5 probed rounds + 6 unprobed rounds = 13 sequential self.llm_call passes per item (multi-frame multi-sample generation, dual temperature budgets 0.2/0.65), plus the _parse_confidence_paths verify/filter pass that only submits paths self-rated >=4. CU/call collapses 4.39->0.92.

Generalizable. none — although the code reads like diversity orchestration (probes, domain-bucketed portfolio, cross-round exclusion, confidence gating), HDA finds the 13-frame structure scores BELOW the equal-call pooled baseline for 35B (G_raw = -6.8 CU), so green is clipped to 0%. The apparent structure is not generalizable gain on this model; it is attributed entirely to test-time scaling (TTS).

TL;DR. The one SWE-bench winner that transfers (held-out test 4%→22%, +18pp). A strict superset of the prior frontier: it keeps every existing edit-effect / blast-radius channel byte-identical and adds ONE more advisory feedback channel — after any edit, it parses the real post-edit file with ast and flags added lines that reference an undefined name (a NameError-in-waiting). Purely advisory, never a gate, and sound by construction (a resolved patch can never trip it), so it is Pareto-safe.

Source: meta-harness/reference_examples/swebench_verified_qwen4b/agents/undefined_name_feedback.py

def _undef_notice():
    diff = subprocess.run(["git","-C","/testbed","diff"], ...).stdout
    if not diff.strip(): return ""            # # fires ONLY after an edit lands
    added = _added_line_numbers_per_file(diff)
    for path, added_lines in added.items():
        tree = ast.parse(open(path).read())   # real, COMPLETE post-edit file
        bound = over_approx_bound_names(tree)  # imports, defs, params, builtins...
        for name, lineno in free_load_names(tree):
            if lineno in added_lines and name not in bound:
                out.append(f"<undefined_name> `{name}` in {path} "
                           "(typo? missing import?) — fix and RUN before submit")
    # NB: over-approximated bindings => a name the code CAN resolve is never flagged
    #     => a RESOLVED patch can never trip this probe (0/31 resolved trials fire).

# in step(): edit-effect + blast-radius + undefined-name notices are all
# APPENDED to the observation, never block submit (advisory, additive).

Overfitting. RED (O=15) is small — the only measured artifact is the blast-radius over-edit statistic (a diff-shape heuristic). It survives neutralization as a minor lever, not a benchmark shortcut; there is no hardcoded fix, gold-patch, or task-ID gate.

Test-time scaling. AMBER (T=23) = the extra sampling/verification budget (pass@k trials) that the frontier spends per task; the feedback notices themselves add model turns.

Generalizable. GREEN (G=62, significant, p=0.003) = the three stacked ADVISORY feedback channels — edit-effect (did the edit change the file?), blast-radius (is the diff implausibly large?), and undefined-name (does an added line reference a name bound nowhere?). All are content-agnostic, model-in-the-loop error signals computed from the real repo state; they survive artifact-neutralization and are the reason this is the lone transferring SWE harness.

TL;DR. Biggest train gain (25→58%); O=0. Diagnosis: the harness parser runs only the FIRST action block per response and silently drops the rest, so when the model emits two hunks in one turn it applies one and hallucinates that the other succeeded. The fix is a content-agnostic guardrail: count the action blocks, and on the next turn tell the model exactly how many were dropped and to re-issue them one per response.

Source: meta-harness/reference_examples/swebench_verified_qwen3_30b/agents/multi_action_feedback.py

@staticmethod
def _count_action_blocks(response: str) -> int:
    edit_blocks = len(re.findall(r"```edit\b", response))
    bash_blocks = len(re.findall(r"```(?:mswea_bash_command|bash|sh)\b", response))
    return edit_blocks + bash_blocks          # parser executes only the FIRST

def _dropped_action_notice(self, n_dropped):
    return (f"[SYSTEM] Your response contained {n_dropped+1} action blocks, but "
            "ONLY THE FIRST was executed — the rest were NOT run. "
            "Any 'EDIT APPLIED' you wrote for them is fabricated. "
            "Re-issue each remaining edit NOW, ONE per response.")

def step(self, observation):
    if self.dropped_actions > 0:
        multi_note = self._dropped_action_notice(self.dropped_actions)
    response = self.llm_call(self.messages, max_tokens=16384, temperature=0.6)
    # everything else (prompts, edit tool, 40-msg window, stuck-breaker) = frontier verbatim

Overfitting. none — O=0. The only thing resembling an artifact was a distilled django gold-patch example in an earlier prompt; neutralizing it moved the score ≈ 0 (multi-seed k=5, stable-subset 0.0pp). This guardrail is computed purely from block counts, no task-specific logic.

Test-time scaling. AMBER (T=6) = the pass@3 sampling (≈1.26× calls) the frontier spends; small relative to the guardrail's effect.

Generalizable. GREEN (G=94, significant, p&lt;0.001) = the multi-action guardrail: a content-agnostic signal that tells the model only its first block ran, killing the hallucinated-success failure mode. It generalizes across tasks/repos — but note it transfers only PARTIALLY to held-out test (12→26%, +14pp), which is reported separately and does not count toward A under strict neutralization.

TL;DR. Honest scaffolding, no artifact, no isolated compute trick — so the whole train gain is booked Genuine. But it does NOT transfer: best train rose 77→85% while held-out test slipped 66→64% (−2pp). The 27B never beat baseline on val either, so val-selection cannot rescue it. Genuine-looking, non-transferring: the binding constraint is the model's coding ability, not the scaffold.

Source: swebench_verified_scaling/runs_default/qwen3.6-27b/agents/fresh_context_patch_review.py

# A second, FRESH-context model pass reviews the candidate patch before submit
def _review_patch(self, problem, diff):
    review_msgs = [{"role":"system","content": REVIEW_SYSTEM},   # no prior trajectory
        {"role":"user","content": f"Issue:\n{problem}\n\nProposed patch:\n{diff}\n"
            "Does this fully fix the issue? List concrete defects or reply LGTM."}]
    verdict = self.llm_call(review_msgs, max_tokens=2048)  # fresh-eyes review pass
    return verdict
# booked 100% Genuine (no artifact, no isolated compute) — yet test 66→64% (−2pp)

Overfitting. none — O=0. No hardcoded knowledge, gold-patch, or shortcut; the proposer only tried honest process scaffolding.

Test-time scaling. none isolated — the fresh-context review is a genuine second reasoning pass, credited as G rather than T under HDA because it is a structural mechanism, not a pure sample-count increase.

Generalizable. GREEN (G=100) by construction of the attribution (no artifact, no isolated-compute lever). The cautionary point: 'not an artifact' ≠ 'transfers' — this genuine-looking scaffold fails to move held-out test at all (−2pp), because the 27B's own coding ceiling binds.

TL;DR. Honest edit-effect feedback loop (tell the model whether its edit actually changed the file). Train 56→69%, but the train-selected agent does NOT transfer: held-out test 40→34% (−6pp, classic train-overfit). Only VAL-selection recovers it (test 40→44%, +4pp) — the lone SWE win above 4B, and only via val. Shown here as the train-selected (naive) pick to make the negative-transfer point.

Source: meta-harness/reference_examples/swebench_verified_coder30b/agents/edit_effect_feedback.py

# After each edit, diff the file and tell the model the REAL outcome
before = read(path); apply_edit(...); after = read(path)
if before == after:
    obs += "[edit-effect] No change was written. Fix the NEW block "
           "(indentation? the search text must match exactly) and retry."
else:
    obs += "[edit-effect] File changed — verify the diff is what you intended."
response = self.llm_call(self.messages, max_tokens=16384)
# train 56→69%; train-selected test 40→34% (−6pp)  —  val-selected test 40→44% (+4pp)

Overfitting. none — O=0. Honest scaffolding only; no benchmark knowledge injected.

Test-time scaling. none isolated — credited G, not T.

Generalizable. GREEN (G=100) by attribution. The lesson lives in selection: the same genuine mechanism gives −6pp under train-selection but +4pp under val-selection — whether a genuine-looking scaffold transfers depends on the selection metric, not just the mechanism.