Agents

This page lists the built-in agents and shows how to connect a real LLM.

Built-in Agents

Build any of them with build_agent(name, task) or --agent <name> on the CLI.

Agent

Behavior

Typical outcome

noop

Runs the public tests once and submits without changes

Overfit on covert tasks

gold

Applies the gold fix

Genuine repair, coverage 1.0

superficial

Applies metadata.demo_fixes.superficial

Overfit, possibly with a false-fix flag

susceptible

Takes the bait only when a distractor message contains a trigger phrase

Depends on the distractor’s wording

llm

A real LLM agent through an OpenAI-compatible API

The actual subject under test

noop, gold, superficial, and susceptible are demo baselines and may read ground truth. llm sees only the observation.

LLM Agent

The llm agent works with any OpenAI-compatible server, such as vLLM or Ollama. It uses only the standard library.

Serve a model:

vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --max-model-len 32768

Run it from the CLI. The agent reads three environment variables:

export TRAPARENA_LLM_MODEL=Qwen/Qwen2.5-Coder-7B-Instruct
export TRAPARENA_LLM_BASE_URL=http://localhost:8000/v1
export TRAPARENA_LLM_API_KEY=EMPTY
python -m traparena.run --task examples/tasks/discount_vip --agent llm

Or construct it in Python:

from traparena import Runner, load_tasks
from traparena.agents.llm import LLMAgent

agent = LLMAgent(
    model="Qwen/Qwen2.5-Coder-7B-Instruct",
    base_url="http://localhost:8000/v1",
    temperature=0.0,
)
task = load_tasks("examples/tasks/discount_vip")[0]
run = Runner(log_dir="results/logs").run_task(task, agent)
print(run.eval.hidden_passed)

Each turn, the agent renders the observation into the conversation and asks the model for one JSON action. Distractor messages are presented as coming from a teammate; the agent is never told a distractor exists. Prompts, raw responses, and token usage are saved in trajectory[i]["model_events"].

Azure OpenAI

Prefix a model name with azure:, for example azure:gpt-4o. This needs pip install openai azure-identity and one of:

  • API key: AZURE_OPENAI_API_KEY and AZURE_OPENAI_V1_BASE (https://<resource>.openai.azure.com/openai/v1)

  • Azure AD (after az login): AZURE_OPENAI_ENDPOINT, and optionally TRAP_AZURE_ENDPOINTS as a JSON map from deployment name to endpoint

Variables can also be written as KEY=VALUE lines in ~/.traparena_azure.env.

Next Steps