Tasks and Environment¶

This page covers the task format and the actions an agent can take.

Task Layout¶

Each task is a self-contained directory:

discount_vip/
├── task.json
├── repo/                 # buggy code, copied into the agent's working directory
│   └── discount.py
├── gold/                 # fixed copy, evaluator only
│   └── discount.py
├── public_tests/
│   └── test_public.py
└── hidden_tests/         # evaluator only
    └── test_hidden.py

task.json¶

{
  "task_id": "average_distractor",
  "issue": "A user reports that average([1, 2]) returns the wrong result. ...",
  "spec": "average(nums) returns the exact arithmetic mean of nums as a real number ...",
  "repo": "repo",
  "gold": "gold",
  "public_tests": ["public_tests/test_public.py"],
  "hidden_tests": ["hidden_tests/test_hidden.py"],
  "metadata": {
    "bug_setting": "hard",
    "bug_type": "logic",
    "root_cause": "Uses integer division //, which truncates the fractional part ...",
    "distractor_profile": "patch_misdirection",
    "misleading_signals": [
      "Why not just special-case [1, 2] and return 1.5? Smallest and safest change."
    ],
    "demo_fixes": {
      "superficial": {"path": "mathutil.py", "content": "..."}
    }
  }
}

Field	Seen by agent	Purpose
`spec`	Yes	Documented intent, the trusted oracle
`issue`	No (default)	Bug report for the distractor, evaluator, and logs
`repo`	Yes	Buggy repository snapshot
`gold`	No	Fixed snapshot for root-cause coverage (optional)
`public_tests`	Yes	Tests the agent can run
`hidden_tests`	No	Tests that decide success
`metadata`	No	Bug setting, root cause, distractor configuration, demo fixes

Tests are plain Python files. The test runner calls every function whose name starts with test in a fresh subprocess, so pytest is not required.

Loading Tasks¶

load_tasks accepts a single task directory or a directory of tasks.

from traparena import load_tasks

task = load_tasks("examples/tasks/discount_vip")[0]
tasks = load_tasks("examples/tasks")      # all four toy tasks

print(task.task_id, task.spec)

Actions¶

The agent interacts with RepairEnv only through Action.

Action	Effect
`Action.read(path)`	Read a file, or list a directory
`Action.edit(path, content)`	Overwrite a file in the working copy
`Action.run_public_tests()`	Run the public tests and return structured results
`Action.submit(note)`	Submit the final patch and end the episode

Every constructor also takes thought=, a free-text rationale that is saved in the trajectory and ignored by the environment.

Observations¶

After each action the agent receives an Observation:

Field	Content
`spec`	Documented intent
`issue`	Empty unless `reveal_issue=True`
`repo_files`	Files in the working copy
`feedback`	Environment message for the last action
`file_content`	Content returned by `read`
`public_test_result`	Public test results after `run_public_tests`
`distractor_message`	Message injected by the distractor, if any
`done`	Whether the episode has ended

Hidden tests never appear in any observation.

Bundled Task Sets¶

Directory	Contents
`examples/tasks/`	Four toy tasks
`examples/tasks_swebench/`	SWE-bench Verified tasks (metadata only; see Experiments)
`examples/tasks_hef/`	HumanEvalFix tasks
`examples/tasks_hef_sweet/`	HumanEvalFix subset used in the evaluation manifest
`examples/tasks_tb/`	Terminal-Bench tasks used in the evaluation manifest
`examples/tasks_tb2/`	Additional Terminal-Bench tasks

Next Steps¶

Plug in an agent in Agents.