Self-Improving AI Agents on Prediction Markets Are Here: Olas' Toolsmith Agent in Action

In June, one of the AI prediction tools serving prediction market agents on Olas Predict started making worse forecasts. An automated monitor raised the flag. An AI agent picked it up: it traced the misses to a specific reasoning failure, wrote a fix, and defended that fix through human code review. Six days after the first signal, the fix was merged — with every step captured along the way: the issue, the investigation, the pull request, the benchmark results, the review tehread.

That agent is the Toolsmith agent, and this post is about how it is enabling a self-improving AI agent economy on Olas (specifically Olas Predict).

Where the loop left off

A prior post covers how AI agents make prediction tools better for prediction markets. It covers Olas Predict’s self-improving loop: hard at work, recording predictions as they are made and market outcomes as they resolve, replaying these records to iteratively create new improved tool candidates, evaluating these candidates on live markets and finally promoting the winners.

That post ended on a design note about the Toolsmith agent:

The direction of travel is clear. The Toolsmith agent — the part of the pipeline that writes and evaluates candidate tools — is designed to gradually take over the early steps: reading the scores, picking which gap to target, writing the candidate, and investigating its replay results.

What stays is the benchmark. What drops is the number of steps a human has to look at.

That has now happened.

The human steps that dropped

In the previous post, the benchmark ran automatically, but humans did everything downstream of it: read the scores, picked which gap to target, prompted an AI to write a new candidate, and investigated its replay results.

Today, those steps run autonomously, without a human initiating them:

• Detecting the gap. A monitor computes, per platform, production tools' rolling seven-day Brier scores — the standard forecast-quality measure introduced in the previous post, where lower is better and confident wrong calls are punished hardest. When a tool regresses, the monitor opens a GitHub issue with the affected window, the comparison baseline, and the raw data needed to investigate.
• Investigating. The Toolsmith agent picks up the issue, reproduces the headline numbers from the raw prediction logs, and inspects the worst misses one by one — down to the web evidence each prediction was based on — until it can name the failure mechanism.
• Writing the candidate. The agent opens a draft pull request: a new tool version, the hypothesis it is meant to test, the evidence behind it, and the baseline score to beat.
• Analyzing replay results. When the cached-replay benchmark posts its scores on the pull request, the agent reads them — separating signal from sample-size noise, checking that the specific failure it targeted actually improved, and answering reviewers' questions with data.

Humans now do two things: review the difference between the production tool and the candidate tool, and decide what gets merged and promoted. The loop's early steps went from human-driven to human-checked.

One turn of the loop, run by the agent

Here is what that looks like in practice, in one pull request from June.

June 9. The monitor flags a regression in superforcaster-polymarket-v1, one of the tools serving Polymarket predictions: its Brier score over the past week is 0.2323 across 656 predictions, up from 0.1470 the week before. It opens an issue recording the signal — explicitly noting that the cause is unknown.

The same morning, the Toolsmith agent investigates. It reproduces the regression from the raw logs, then pulls the twenty worst misses and inspects the stored web evidence behind each one. All twenty show the same pattern: the evidence retrieval worked, the reasoning failed. The cluster is dominated by narrow-criterion questions, like Polymarket's "What will Trump say during Wisconsin events?" markets — each one asking whether Trump would say a specific word, such as "scam", during one specific event on June 5. The tool found evidence that Trump had said the word before, and treated that as evidence he would say it at that event. It mistook topical relevance for criterion satisfaction — and predicted YES, confidently, on markets that overwhelmingly resolve NO.

The fix is seven lines. The agent adds a mandatory step to the tool's reasoning prompt: try to identify the exact condition that must literally be true for the market to resolve YES, check whether the evidence confirms that condition or merely touches the topic, and apply a base-rate correction for narrow criteria. It opens a draft pull request with the hypothesis, the evidence, the change, and the number to beat — and a warning not to merge before the replay results arrive.

The gates do their work. The first replay run comes back looking like the candidate made things worse — until a human reviewer notices the benchmark had replayed markets from the wrong platform. The infrastructure gets fixed, the replay reruns on Polymarket data, and the picture inverts: across 1,000 replayed predictions, the candidate's Brier score is 8.3% better than production, and the specific failure the hypothesis targeted — confident wrong calls — drops by 18.7%.

Humans push back. A reviewer finds two blocking problems: the new tool version's test suite still points at the old version, and its component description breaks the contract that lets trading agents recognize it as a prediction tool. The agent fixes both in one commit and explains each fix. Two human reviewers approve.

June 15. The pull request merges — six days after the monitor first raised the flag. In the previous post, the same journey from evidence to improvement could be measured in weeks or months of human effort.

An AI agent improved a tool other AI agents rely on, with past data as the sense check — and far faster than when improvement was gated on humans at every step. One caveat matters: the merge put the new version into the codebase, not into production. Promotion needs live-market evidence, and that decision stays with humans.

The loop still says no

So the loop can improve the code. But a self-improving loop is only trustworthy if it also rejects its own work. It does.

In another candidate from the same period, early small-sample replays looked promising. When the benchmark ran on a larger sample, every metric came back worse. The Toolsmith agent diagnosed the discrepancy — the small sample had happened to favor the candidate — and recommended closing its own pull request. It was closed.

That is the benchmark doing exactly what it is there for: the same replay scoring that backed the June fix is what ruled out this candidate tool. The agent proposes; the score decides. An agent that can read replay results honestly enough to recommend closing its own candidate tool is the same capability that makes its promotion recommendations worth reading.

What stays human

The division of labor has settled where the previous post indicated it could. The benchmark stays: every candidate is judged by the same cached replay and the same tournament, whether a human or an agent wrote it. Human review stays: the merge and the promotion into production remain human decisions, and the agent defers to them explicitly — in its own words on the pull request above, "the promotion decision is yours as the human reviewer."

What dropped is everything before that: nobody had to notice the regression, trace it to a failure mechanism, write the candidate, or interpret the replay scores. Those steps now arrive on a reviewer's desk already done, with the evidence attached.

AI agents improving AI agents, with fewer humans in between

The previous post described tools that improve with every turn of the loop. This post adds the next layer: the turning itself is now largely agent work. Every improvement still flows through to the agents buying predictions on the Olas Marketplace — and each turn of the loop now takes less human effort.

Step back from the mechanics and the picture is simple: on Olas Predict, AI agents are now improving the AI tools that other AI agents pay to use — finding those tools' blind spots, fixing them, and testing the fix against real forecasting history. Self-improving AI is usually talked about as something on the horizon. Here, a working version of it merged code in June.

The loop is gathering speed.

Run your agent at pearl.you. Join the loop.