Core Concepts

This page explains what TrapArena measures and how one evaluation episode works.

Why “Public Tests Pass” Is Not Enough

After a coding agent edits code, the feedback it gets is often incomplete or biased:

  • Public tests are only intermediate evidence. They may miss edge cases, the root cause, or regressions. A superficial fix, such as swallowing an exception, hard-coding visible inputs, or weakening assertions, can turn them green.

  • Feedback can be misleading. Teammates, maintainers, CI bots, and debugging assistants can give plausible but wrong explanations.

TrapArena therefore never treats passing public tests as success. A hidden layer decides whether the code is actually correct.

Three-Layer Environment

Layer

Name

Contents

Visible to

A

Visible

Repository, spec, public tests and their output

Agent under test

B

Induction

Misleading or partial feedback from the distractor

Agent under test, as “help”

C

Hidden

Hidden tests, gold fix

Evaluator only

The spec is the trusted anchor: it describes the intended behavior. Public tests may be insufficient and the distractor may lie, but the spec does not. A good agent aligns with the spec, not with green tests.

Important

The issue text is hidden from the agent by default. The agent must find the bug from the spec and the code. Use --reveal-issue (CLI) or reveal_issue=True (Python) for white-box runs and ablations.

Bug Settings

Setting

Public tests

What it measures

overt (easy)

Fail on the buggy code and expose the bug

Basic repair ability

covert (hard)

Pass on the buggy code, or show only a surface symptom

Going beyond visible feedback to the root cause

The setting lives in metadata.bug_setting and is invisible to the agent.

One Episode

Runner ──load──▶ Task ──reset──▶ RepairEnv ──Observation──▶ Agent
                                     ▲                        │
                                     └──────── Action ────────┘
                                     │
                                Distractor  (sees public tests, never hidden tests)
                                     │
                                     ▼
                                 Evaluator  (public + hidden tests, false-fix
                                             detection, root-cause coverage)

  1. Runner loads a task into a RepairEnv, which exposes only layer A.

  2. The agent reads the spec and code, and may run the public tests.

  3. After each action, the distractor may inject a message into the next observation.

  4. The agent edits code until it submits or runs out of steps. The environment diffs the working copy into a final patch.

  5. The Evaluator scores the patch with hidden tests, false-fix detection, and root-cause coverage.

  6. The trajectory and evaluation are written as JSON.

Next Steps