Instead of humans hand-building every test scenario for AI agents, Polymath's system automatically generates thousands of realistic digital workplaces—complete with tools, data, and tasks—so agents can learn faster and cheaper.

Polymath employs ML-driven procedural generation to automatically create rich, varied digital environments that simulate real-world enterprise toolchains. The system uses learned priors about tool relationships, task dependencies, and realistic data distributions to compose environments featuring interconnected tools like GitHub repositories, Slack channels, Linear boards, email inboxes, and web browsers. A human-in-the-loop feedback mechanism allows researchers to steer generation toward specific complexity profiles or domain requirements. This dramatically reduces the engineering bottleneck of manual environment authoring—traditionally the most time-consuming part of RL research—while ensuring statistical diversity that prevents agent overfitting. The generator can produce thousands of unique world configurations, each with internally consistent state, realistic data artifacts, and verifiable task objectives, enabling scalable agent training pipelines that would be impossible to build by hand.

It's like having an AI dungeon master who instantly conjures up thousands of unique office buildings—each with different teams, tools, and problems—so your AI intern can practice working at all of them before its first real day on the job.

Polymath built an automated grading system that tests whether AI coding agents can actually do a software engineer's full job—from reading a ticket to shipping and monitoring code—not just answer trivia questions about programming.

Horizon-SWE is Polymath's flagship benchmark product, simulating a complete software company environment where AI agents must perform end-to-end software engineering: interpreting product requirements from Linear tickets, navigating codebases on GitHub, writing and testing code, deploying via CI/CD pipelines, and monitoring production systems. ML-powered verifiers automatically assess agent performance across multiple dimensions—correctness, efficiency, code quality, tool usage patterns, and task completion—without requiring human graders. These verifiers use learned models trained on expert-annotated trajectories to distinguish between superficially correct and genuinely robust agent behavior. Current frontier models (GPT-4-class and beyond) achieve only approximately 25% on Horizon-SWE, demonstrating the benchmark's difficulty and its value as a discriminating evaluation instrument. The benchmark serves as both a product differentiator for Polymath and a community resource that drives adoption of their simulation platform by AI research labs seeking to measure real-world agent capability.

It's like building a full-scale simulated restaurant kitchen where you test whether a robot chef can handle an entire dinner service—not just see if it can chop an onion.

Every time an AI agent practices in Polymath's simulated worlds, it automatically creates labeled training data showing what worked and what didn't—like a flight simulator that writes its own textbook after every session.

As AI agents interact with Polymath's procedurally generated environments, every action, observation, tool invocation, and outcome is captured as structured trajectory data. This synthetic interaction data is then processed through Polymath's automated verifiers to produce richly labeled datasets—annotating successful strategies, failure modes, tool usage patterns, and decision quality at each step. These datasets serve as high-quality training signal for reinforcement learning from environment feedback (RLEF), enabling iterative agent improvement without costly human annotation. The data pipeline supports filtering by task type, difficulty, tool combination, and outcome, allowing researchers to construct targeted training curricula. Because the environments are procedurally generated and internally consistent, the synthetic data exhibits natural diversity that mitigates distribution shift problems common in hand-curated datasets. This creates a powerful flywheel: better environments produce richer data, which trains better agents, whose failures reveal gaps that inform the next generation of environment design.

It's like a driving school where every student's practice session automatically writes a new chapter in the driver's ed manual—complete with highlighted mistakes and gold-star moments—so the next class learns even faster.