OpenForgeRL: Train Harness-native Agents in Any Environment

The ambition to deploy sophisticated AI agents capable of complex, multi-step reasoning and interaction with diverse external systems hinges on their ability to learn effectively in realistic environments. Yet, a fundamental tension persists: the intricate, stateful inference harnesses (like Claude Code, OpenClaw, or Codex) that empower these agents in deployment are notoriously difficult to integrate with standard open-source Reinforcement Learning (RL) training pipelines. This disconnect has historically forced researchers into compromises, either simplifying the agent’s operational environment or relying on less direct training methods.

Executive Summary

The paper introduces OpenForgeRL: Train Harness-native Agents in Any Environment, an open-source framework that directly addresses this critical gap. It enables end-to-end training of sophisticated AI agents, complete with their elaborate inference harnesses, in the very environments they are designed to operate within. This is not merely an incremental improvement; it’s a foundational shift. By decoupling the complexities of harness inference from the RL training loop, OpenForgeRL allows for direct, scalable learning in real-world scenarios, promising a faster, more robust path to truly intelligent AI agents. For anyone invested in the practical deployment and advancement of LLM-powered agents, this framework offers a compelling solution to a long-standing challenge.

Technical Deep Dive

At its core, OpenForgeRL solves the problem of integrating multi-process, stateful harness inference with stateless RL training stacks. The framework achieves this through a dual-component architecture:

1. Lightweight Proxy: This intelligent proxy sits between the agent’s harness and the underlying LLM calls. During inference, it transparently serves the model calls while simultaneously recording all interactions—inputs, outputs, states, and decisions—as structured training data. This effectively transforms real-time agent execution into a stream of samples suitable for Machine Learning.
2. Kubernetes Orchestrator: To handle the computational demands and diverse environments, OpenForgeRL leverages Kubernetes. Each agent rollout, complete with its specific harness and environment, is encapsulated and executed within its own remote container. This orchestrator ensures scalability, isolating rollouts and enabling parallel execution across a distributed infrastructure.

The brilliance of this design lies in its simplicity and generality. By treating the harness as a black box during training data generation and abstracting environment execution, OpenForgeRL allows any harness to be trained with any standard RL codebase (e.g., veRL) in virtually any environment. This decoupling fundamentally changes how we can study and improve complex agent behaviors, moving beyond simplified simulations to direct, on-the-job learning.

Real-World Applications

The impact of OpenForgeRL is immediately apparent across several challenging domains where AI agents are poised to revolutionize workflows:

• Advanced Tool-Use and API Interaction: For agents designed to interact with complex APIs or execute code (like those built on OpenClaw or Codex), OpenForgeRL allows for direct training within the actual tool ecosystems. On benchmarks like ClawEval and QwenClawBench, OpenForgeClaw achieved pass@3 scores of 31.7 and 55.9 respectively, and 33.7 on QwenClawBench, significantly outperforming open baselines. This means more reliable code generation, better API orchestration, and more robust automated workflows.
• Multimodal GUI and Computer Automation: The framework excels in training agents that navigate and interact with graphical user interfaces (GUIs) and operating systems. OpenForgeGUI, for instance, demonstrated strong performance on OSWorld-Verified (37.7), Online-Mind2Web (63.0), and WebVoyager (72.3). These results not only surpass open-source competitors of similar size but often match or even exceed models several times larger. This capability is critical for automating enterprise software tasks, sophisticated web browsing, and general computer control, bridging the gap between human instruction and digital execution.

The ability to train these agents directly within their operational environments, with their full harness capabilities, means moving closer to truly autonomous and highly reliable AI agents for diverse industry applications.

Future Outlook

OpenForgeRL paves the way for a new era of agent development. Looking 2-3 years out, we can anticipate several key developments:

Firstly, the framework provides an unparalleled platform for deeper research into LLM harness design. The paper’s analysis already highlights that some harnesses are substantially harder to learn than others. This insight will drive the creation of more “learnable” and effective harness architectures, optimizing the synergy between the base LLM and its agentic wrapper.

Secondly, the improvements in agentic reliability observed with RL—such as enhanced self-verification, broader tool coverage, and more successful multi-step plan completion—will become standard. This means fewer brittle agents and more robust systems, accelerating the deployment of AI agents in critical business processes.

However, the analysis also pointed to a persistent weakness: error recovery. This will likely become a major focus of future research. Developing agents that can gracefully handle unexpected failures, diagnose issues, and recover autonomously in complex, real-world environments is the next frontier. OpenForgeRL offers the perfect testbed for this, allowing iterative improvements directly in the deployment context.

Ultimately, this framework brings us closer to a future where AI agents can continuously learn and adapt in their operational settings, reducing the chasm between laboratory research and real-world intelligence.

Key Takeaways

• OpenForgeRL is an open-source framework enabling end-to-end training of LLM-powered AI agents within their native, complex inference harnesses and real-world environments.
• It utilizes a lightweight proxy to record training data from live harness interactions and a Kubernetes orchestrator for scalable, containerized rollouts, effectively decoupling training from inference complexities.
• The framework achieves superior performance in demanding tasks like tool-use (OpenForgeClaw) and multimodal GUI/computer automation (OpenForgeGUI), outperforming open baselines and matching larger models.
• It provides a crucial platform for studying the impact of harness design and Machine Learning strategies on agent behavior, improving agentic reliability in areas like self-verification and multi-step planning.
• While significantly advancing agent capabilities, the research highlights error recovery as a key area for future improvement, underscoring the ongoing journey towards truly robust and autonomous AI agents.