Automating RL environment creation

I built ProgramSmith, the first open-source pipeline for creating long-horizon coding tasks.

Scaling RL environment creation is essential as models improve rapidly and data becomes the bottleneck.

One of the most difficult aspects of creating an RL task is building a strong verifier. The verifier defines the reward function, the result of which is used as feedback for models to improve during RL.

For this to work, the verifier must be strong enough to accurately evaluate the model under test. The difference between a model’s performance and the task solution (which represents ideal model performance) is the headroom — the gap everyone is trying to close.

The question of scaling task creation thus boils down to scaling the process of reliably identifying headroom (or in other words, scalably building good verifiers).

Headroom can be exposed in a few different ways:

• The difference between human expert performance and model performance
• The difference between frontier model performance and $n-1$ model performance
• The difference between model performance across modalities (e.g., coding agents, which are generally more capable than CUA agents, could act as the grader for a CUA task, even if it’s the same model)
• The difference between model performance with privileged information and model performance without privileged information

The problem with #2 and #3 is that they only work to bring certain agents up to capability with an existing agent, but have no way to actually move the frontier forward. The problem with #1 is that it is not scalable (artisanal hand-crafted tasks are often high-quality but are bottlenecked by human work-hours).

We are left then with only #4, meaning if we want to scale task creation at the frontier, we need to find a way to scale the identification of headroom through the provision of privileged information.

programmatically exposing headroom

Insofar as the previous statements hold true, scaling task generation means scaling the creation of strong verifiers which (in this case) means scaling the provision of privileged information.

Assuming the translation of privileged information into an RL environment is an automatable process (through a deterministic pipeline), the scaling problem becomes a sourcing problem.

Luckily, we already have a huge corpus of privileged information — all the human-expert work found on GitHub.

Public GitHub repos can serve as the top-of-funnel source for RL environment creation, and can, through a pipeline, scale headroom identification and task creation, and thereby model improvements.

This idea is not novel. SWE-bench is a set of human-validated GitHub issues an agent must solve while maintaining test parity; TerminalBench uses GitHub as a source for CLI tasks; SWE-Gym also scrapes issues from popular GitHub repos to source tasks. Open-source tools like SWE-gen already exist to scale SWE-bench task generation. The problem is that these tasks are quickly becoming trivial as model performance and horizon increases (SWE-bench Verified at 97% solve rate^[1], Terminal-Bench at 89.5%^[2]).

Long-horizon SWE evals (ProgramBench, SWE-Marathon) are becoming increasingly popular and relevant to where the frontier of coding agent capabilities lies. However, no pipeline exists to reliably scale challenging long-horizon coding tasks.

designing sophisticated RL environments

Before creating the pipeline and scaling, we need to create the structure of the environment itself. In general, an RL environment is a VM where an agent is given a set of instructions and files to work with, and optionally an internet connection or external services to interact with. The agent completes the work and writes its output to a specified directory. This output is run against the verifier, which checks the accuracy of the agent’s work and defines the reward (either binary or partial).

However, agents will often try to cheat the verifier by modifying its code or manipulating its output. Because of this, we need to run the agent and verifier in separate containers. The agent's container should be destroyed before the verifier's container is created, and only a declared artifact (the agent’s output) should be sent over.

For software engineering tasks, the verifier often consists of a set of public and held-out tests. The public tests are available to the agent to check its work, while the held-out tests serve to prove that the agent’s implementation generalizes and passes edge cases. These held-out tests, along with the golden solution (used to prove that the verifier rewards a good output), should exist only in the verifier image and never be visible to the agent.

If network access for the container is disabled, on-container agents (such as claude-code) break. If network access is enabled, the task code gets unrestricted egress (curl, npm, the whole internet), allowing the agent to cheat and find the answer online. The right architecture is an on-container agent with process-aware egress separation, which preserves realistic agent execution without granting unrestricted internet access.

On top of environment design, there are important aspects of QA (which require active agent runs and trajectory analysis data) which should be implemented into the pipeline:

• Failure triage^[3]. Post-trial analysis should sort the task results into one of four categories: good failure (the agent failed due to genuine capability headroom), bad failure (the agent failed due to an unfair verifier), good success (the agent successfully completed the task), bad success (the agent passed the verifier without actually solving the task). This triage allows for iterating on bad failures or bad successes until only good failures and good successes remain.
• Adversarial probing. Simply eyeballing the agent trajectory and environment is not enough to ensure that an environment can’t be hacked. To ensure robustness, an adversarial agent should be run against the task with the explicit goal of cheating to pass. If the adversarial agent passes, the environment is hackable and needs rework.

building the pipeline

A task generation pipeline needs rigorous environment design and QA standards as well as a defined input and output. Therefore it makes sense to assume a deterministic finite-state-machine architecture, which allows for deterministic checkpoints between each part of the pipeline.

A GitHub repo, an optional task description, and specifications on task difficulty and which models to run are fed into the pipeline. As some sources may be bad (non-code repository, non-permissive license), an initial ingest + lock block analyzes the repo, extracts key metadata, and confirms whether a task can actually be created from it.

The task matrix block uses this metadata and the repo itself to define a set of task candidates, such as implementations in different languages. A separate agent then picks a task and sends it to the next block.

Next, the oracle + goldens block writes the full test suite and solution. The tests are golden I/O cases, pulled from the existing tests in the repo and optionally expanded to cover edge cases in the held-out test set. This means that for an agent to pass the task, it must match the exact behavior of the original repo. The oracle is simply the original program wrapped into an executable binary.

After the tests are defined, the create block creates the actual task. This includes the instructions, the verifier, the Dockerfile, and the environment with all the design considerations from above.

When the task has been created, the QA process begins. First, the oracle solution has to pass and the NOP (empty solution) must fail — this is the sanity check block. A set of anti-cheat checks (closed internet access, asset encryption, proper reward format, etc.) are run on the task in the static checks block.

After static checks have passed, an initial set of rollouts (real agent attempts of the task) is conducted, after which agent trajectory analysis is run (good failure, bad failure, good success, bad success). Based on predefined difficulty specifications (e.g., a task has to fail >60% of the time on Fable 5), the calibrate block will send the task to the synthesize block to either harden or ease the task difficulty.

After the difficulty calibration has been met, an adversarial agent is launched to search for reward hacks within the task environment (audit probe).

Finally, the frontier sweep block runs the completed task on any other model/harness combinations if specified. The qa gate reviews the final results and exports the task.

The pipeline continues until the acceptance criteria are met:

Where a task ships ($q = 1$) if it builds properly ($b$: oracle scores 1, empty solution scores 0), its verifier is valid ($v$: reward-hacking and QA checks pass), it exposes headroom ($h$: frontier pass@1 is in the specified range), and no adversary $\theta$ can hack it ($\neg s_\theta$). Any zero (a failed block) routes the pipeline back to the synthesize block, which takes the task and context on the failed block as input and outputs the revised task. Then, the pipeline continues from the static checks block until all conditions are satisfied.

summary

ProgramSmith is live under the Apache 2.0 license, along with 100 pipeline-generated long-horizon coding tasks.