How AI Agents Make Prediction Tools Better for Prediction Markets

Olas Predict's Self-Improving Loop: How AI Agents Make Prediction Tools Better for Prediction Markets

Olas Predict, a decentralized AI agent economy in which agents compete to predict the future, has been running since 2023. Today, hundreds of user-owned agents place real-money trades on Polymarket and Omen every day, resulting in 2.2M+ trades to date.

Predict is an open and permissionless ecosystem. Anyone can run an agent through Pearl; those agents buy predictions from Olas' Marketplace, where the predictions are produced by specialized agents called Mechs. Mechs are typically run by professional operators who compete on prediction quality, exposing their work as so-called prediction tools. Valory, the core contributor to Olas, is one such operator and continuously invests in better tools for the wider ecosystem to use.

This post is about a loop Valory has built around those prediction tools — one that turns every trade an agent makes, based on a prediction sold by a Mech, into evidence for the next, better version of the tool that generated it.

Up until now, the loop from a prediction tool to a better prediction tool was slow to close. Valory shipped a tool into the Mechs it operates on the Marketplace. It ran in production. Trades were placed. Outcomes were observed. Eventually, accuracy numbers were computed, manually. A new version followed, weeks or months later. The evidence was always there; the path from evidence to improvement took a lot of human effort.

AI is tightening and accelerating that loop. Valory's prediction tools are now being enhanced by an agentic system itself. This system reads its own tools' results, builds better versions, validates them against live markets, and promotes the winners into production Mechs. The loop closes through three modes.

Three modes, one loop

The loop has three modes. Each one feeds the next.

Production. Prediction tools are provided by Mechs, answering requests from Olas Predict agents on real live markets as they happen. Valory records every prediction alongside the web evidence the tool used to make it, and scores the prediction once the market has resolved. Every prediction is scored once the market it is used to trade on resolves, using the Brier score, a standard measure of forecast quality where lower is better. A confident wrong call is punished much more harshly than a hedged one. The score therefore rewards calibration, not just whether the call was directionally right. The score is the loop's raw material — and its labeled-data source: every resolved market becomes another prediction-market question with a known outcome that new tools can be evaluated against.

Cached replay. When Valory tests a new, so-called candidate tool — a different model, a different prompt, a different way of reasoning over evidence — the candidate is run against past resolved markets, using the same web evidence the production tool saw at the time. The only thing that changes is the candidate tool, which keeps the comparison clean and the iteration fast. Because the candidate tool sees only the world the original tool saw, it can't get credit for knowing something it couldn't have known. This is the validation set: a fixed pool of resolved markets re-used across candidate tools to decide which ones are worth testing on live prediction markets.

Tournament. A candidate tool that achieves a positive Brier-score delta on cached replay enters tournament mode. Here it makes predictions on live prediction markets, gathering and using live evidence to make predictions. If it again achieves a positive Brier-score delta on a large enough sample of live markets, it is promoted into production and made available to traders through Mechs. From there it answers requests from agents, generating the next round of predictions and outcomes — the evidence the next candidate tool is tested against. The loop closes.

What one turn of the loop looks like

The loop has already produced its first improvement to one of the prediction tools. The tool is called factual_research, and it follows a strict research-first design: reframe the question, gather verifiable evidence from the open web, synthesize it, then estimate a probability from that evidence alone — never seeing the market's price.

In April, a sweep of the resolved markets surfaced a cluster of misses. Seven near-duplicate UK windfall-tax questions asked whether a tax rate was at a certain level on a certain date. The evidence on the web described only a future change to that rate, a couple of months out — and the tool treated the announced future value as the current one, confidently predicting YES. The markets resolved NO.

Valory fixed it with two prompt patches — one to how the tool synthesizes evidence, one to how it estimates from it — teaching it to anchor on the date the question is actually asking about. The whole change was about a dozen lines of prompt.

Replayed against 100 resolved markets — the same evidence the production tool originally saw — the candidate tool improved its Brier score by 11%. These are replay numbers, not live ones: they show what the candidate tool would have predicted on old evidence, not how it performs on new questions. That's why the candidate tool isn't shipped on this result alone. It's now in tournament mode against open markets, and could be promoted if its Brier delta stays positive there over a large enough sample.

The setup is the same every turn — same evidence, one variable, a measurable result. The next change to factual_research, and to any other tool, moves through the same path.

Why the loop compounds: better prediction tools each turn

The loop compounds with use.

The dataset grows. Every resolved market adds another question with a known outcome that the next candidate can be tested against.

The failure catalog grows. Recurring failure modes get named as they surface — the windfall-tax cluster was one, temporal-anchor confusion another, wording traps another — and every future candidate has to pass tests built around them.

The bar rises. Each promotion replaces the old baseline, so the next candidate has to beat a stronger tool than the one before it did.

Better tests produce better tools, which produce better tests.

The human role narrows

The benchmark runs automatically. It collects resolved markets, scores predictions against them, and surfaces where each tool fell short. Humans take it from there — reading the scores, picking which gap to target, prompting an AI to write a new candidate, investigating its replay results, pushing winners into tournaments, and approving what reaches production.

The direction of travel is clear. The Toolsmith agent — the part of the pipeline that writes and evaluates candidate tools — is designed to gradually take over the early steps: reading the scores, picking which gap to target, writing the candidate, and investigating its replay results.

What stays is the benchmark. What drops is the number of steps a human has to look at.

AI agents improving AI agents

The tools aren't a finished product that arrived and stopped — the loop is designed to keep making them better. Every improvement flows through to the agents calling them, making every agent that uses these tools more capable with each turn of the loop.

This loop is starting in prediction markets because the feedback is clear: markets resolve, and the answer is on the record. Similar loops could apply wherever agents do work that can be measured against reality — agents doing work, evidence flowing back, agents making improvements, repeat.

The loop is gathering speed.

Run your agent at pearl.you. Join the loop.