Valory turns Olas agent data into a stronger AI forecaster: meet Olas-Predict-R1-14B

A specialised open-weight model, Olas-Predict-R1-14B, cut forecast error by roughly 20% against base model DeepSeek-R1-Distill-Qwen-14B, with 14 billion total parameters, and performed on par with GPT-4.1, rumored to have on the order of 300 billion total parameters, in a six-week evaluation.

Whenever Olas agents trading in prediction markets ask for a forecast on a given future event, they create something useful: a prediction that can later be compared with what really happened. Valory, a core contributor to Olas, has now used that prediction history to improve an open-weight model’s forecasting capabilities.

Across 2,628 prediction markets on Omen and Polymarket, Valory’s strongest fine-tuned model lifted accuracy from 71.4% to 75.8% and reduced Brier score, the study’s primary measure of forecast quality, from 0.2249 to 0.1792. It achieved forecasting quality on par with GPT-4.1 under the study’s comparison conditions.

An agent economy that creates its own training material

In Olas Predict, user-owned AI agents buy forecasts from specialist agents called Mechs and trade on prediction markets. Olas Predict has operated since 2023. Its activity produces records of market questions, time-stamped evidence, predicted probabilities and eventual outcomes.

The model is one part of this forecasting process; it does not produce a market forecast on its own. A Mech answers each request with a forecasting tool. The tool gathers evidence about the question, builds a prompt from it, and asks a language model for a probability. The model brings the reasoning, and the tool brings the evidence.

For this experiment, Valory assembled a development dataset of 214,529 samples across 5,116 resolved markets. The data was split by market into training, evaluation and test sets; separate markets were reserved for calibration and reasoning generation.

Valory started with DeepSeek-R1-Distill-Qwen-14B, a 14-billion-parameter open-weight reasoning model. Reinforcement learning rewarded forecasts according to how closely their probabilities matched actual outcomes. The strongest version added a supervised fine-tuning (SFT) warm start before that training.

The training setup fits on a single A100 GPU by compressing the base weights and training small adapter layers. Valory automated the process from provisioning the GPU through training, evaluation and shutdown. A fresh experiment can be completed within a couple of days, making quick iteration practical.

Tested on later markets, using the same evidence

The critical test was whether the gains held beyond the training period. Valory ran a separate evaluation, designed and operated independently of its training pipeline, on markets that closed between Jul 16, 2026 and Aug 27, 2026, after the training data ended.

The study covered a total of 2,628 markets: 1,247 Omen-Presagio markets and 1,381 Polymarket markets. It focused on researchable binary questions, such as elections, court rulings and product launches. Before running the models, we set rules to exclude questions, e.g., about specific sports results, short-term asset prices and exact wording, where no amount of research could bring useful evidence for the forecaster.

Every model received the same information collected for each market question before the outcome was known. The models had no web access, in particular they had no information about later news. There were 4,347 information contexts in total, with each market weighted equally. Unusable answers received the worst possible Brier score. The open models had a shorter output allowance than GPT-4.1: 1,024 tokens versus 4,096.

This setup tested the model on its own, not the search stage of the forecasting tool. Every model saw the same evidence, so the comparison measured how well each model reasoned over that evidence, not how well it could search the web.

A clear improvement from training

Brier score measures the gap between a predicted probability and the outcome; lower is better. It matters because agents need useful probabilities to assess a trade, not just a correct YES or NO call.

For detailed confidence intervals, see the table in the source research linked at the end of this post.

Both trained models improved on the base model on both prediction platforms, with confidence intervals supporting the gains. The model trained with reinforcement learning alone also remained ahead when the base model’s unusable answers were excluded.

The next forecasting advantage starts with the next forecast

Valory has demonstrated that historical data generated by Olas Predict can be used to improve the forecasting ability of a model its agents later can use via a forecasting tool. The next step is to repeat that process as more records accumulate: train candidates, test them on new markets, and deploy those that earn their place.

This is the strategic opportunity for Valory and Olas. Better forecasting models help Olas Mechs provide more accurate probabilities to trading agents, helping those agents make better trading decisions. As agents request more forecasts and markets reveal their outcomes, the resulting records give Valory more data to improve the models.

A specialised open-weight model also gradually reduces Mechs’ serving costs. The forecasting tool can call a self-hosted model instead of a paid model API.

This experiment demonstrates a continuation and further enhancement of the self-improving loop powered by Olas Predict and Olas Mech Marketplace. Olas provides the activity and prediction history; Valory has shown how to turn them into measurable and actual forecasting progress.

Get started

To experience autonomous prediction trading, download Pearl, select, fund and launch an agent by visiting pearl.you/connect.

The new model Olas-Predict-R1-14B can be downloaded at https://huggingface.co/valory/Olas-Predict-R1-14B.

For more about the model, visit https://olas.network/models/olas-predict-r1-14b.

Disclaimer

This post is for information only. It is not financial, investment, legal, or tax advice, and nothing in it is a recommendation to use a model or to trade on prediction markets or to run any agent or application.
The figures described are observations of past activity over a limited period. Past observations do not indicate future results, and outcomes vary between users, operators, agents, markets, and time periods. Prediction market trading carries risks, including the risk of losing the full amount committed. As always, DYOR.

Valory's Annex

How these results were obtained

The data. Every sample is one resolved market prediction from our live agents: the question, the web-search context the agent gathered, the probability it produced, and the real outcome. The dataset holds 214,529 samples across 5,116 resolved markets. It is first split by market into train, eval and test. Two further slices are then carved out of train, one that was used for fitting the calibration step and one for generating reasoning examples, and both are removed from train, so the reinforcement-learning stage never trains on those markets.

Split	Samples	Markets	Purpose
train	115,824	2,769	reinforcement-learning training, after the two carve-outs
eval	64,559	1,534	picking the best checkpoint during training
test	21,583	513	final scoring, never touched during training
calib	5,595	154	fitting the calibration step, carved out of train
sft	6,968	146	generating reasoning examples, carved out of train

Splitting by market matters. One market produces many predictions, so splitting by row would let the model see a market during training and then be scored on it. Of the 21,583 rows in the test split, 7,239 exceed the 1,900-token prompt limit and are dropped, leaving the 14,344 cases scored above. The same limit applies during training, so the model is scored on the same kind of prompts it was trained on.

The setup. We start from DeepSeek-R1-Distill-Qwen-14B. The base model is compressed to a quarter of its normal size and only small adapter layers are trained, which is what lets a 14B model train on a single A100. The reward is the Brier score measured against the market base rate. Four answers are sampled per question, and we train for one epoch over a fifth of the training split, checking progress every 100 steps and keeping the checkpoint with the best balanced accuracy.

The experiment path.

1. Reinforcement learning from the untrained base model. Balanced accuracy 0.564, with poor calibration.
2. Reasoning generation. Using that checkpoint, we generated step-by-step reasoning for the markets carved out of train for this purpose, sampling four candidate answers per market. We discarded any answer without a readable probability, and kept only those that beat the trivial "always guess the base rate" prediction for that market. We kept every answer that passed rather than only the most confident one, since training on only the most confident answers teaches over-confidence. The real outcome was used only to select traces, never shown to the model producing them.
3. Warm start. We trained an adapter to imitate those traces.
4. Reinforcement learning again, this time starting from the warm-started adapter. Every setting is identical to step 1. The only change is where training begins. Balanced accuracy 0.614.

Calibration. We also tried Platt scaling on top of the checkpoints from steps 1 and 4, fitted on the calib split. On our test split, it cut Brier error to 0.175 and calibration error to about 0.05 while leaving balanced accuracy unchanged. Before adopting it, we ran a second evaluation, built separately from the training pipeline, and the improvement did not carry over. We dropped the step and report the uncalibrated model above.

Independent out-of-time evaluation

After training the models, we ran a separate evaluation operated independently from the model training pipeline. As usual, training results are reported on held-out test sets: markets the model never trained on. However, these markets were drawn from the same period as its training data. Therefore, there is a stricter question we would like to explore: does the model still perform well on markets opened after its training data ends, and how does it compare with the proprietary commercial model behind our production tool when both read the same evidence?

The evaluation treats each model as a black box. Every prediction was made from a prediction context frozen before the market resolved. Each model received that evidence — the market snapshot and the news and snippets collected at the capture date — plus the capture date itself, and nothing more: no web access and no later news. The evaluation experiment considered markets closed in a six-week period after the training data ended, between July 16 and August 27, 2026.

We started from the markets closing inside the window that the evaluation gathered: some were already covered by the agents' own request history, and the rest were collected directly from the two platforms for this experiment. From there we filtered. Markets that resolved without a YES or NO — an Omen question settled as invalid, for instance — cannot be scored and were dropped, and what remained was filtered to researchable questions. That left 2,628 markets: 1,247 on Omen and 1,381 on Polymarket. Researchable means that research can be useful for the forecast: elections, court rulings, product launches, scheduled announcements, etc. Every market question was labeled before any model ran, by a language-model classifier with a fixed rubric (ambiguous cases were resolved by a human). Questions that rely on a specific sports result, a short-horizon asset price or numeric band, what someone will say on an X post, or a headline's exact wording were excluded from the market population, because no amount of research helps in forecasting the outcome.

Each market carries up to two frozen evidence contexts, one built from search-result snippets and one from retrieved page content: 4,347 contexts in all. Every model answered the same set of contexts under the same protocol, so the only thing that differs between prediction executions is the model. Also, one asymmetry is that the open models had a 1,024-token output budget against 4,096 for GPT-4.1, a limit that, if anything, penalizes the open models. A market's score is the mean over its contexts, so each market counts once, and an answer with no usable probability scores the worst possible error of 1. Confidence intervals come from 10,000 bootstrap resamples, with markets linked to the same real-world event resampled together.

Results

The table below summarizes the results for the collection of markets where all models ran using the same evidence.

Model	Scope	Markets	Brier error	Δ vs. base [95% CI]	Δ vs. GPT-4.1 [95% CI]	Accuracy	AUC
Market price at capture (reference)	Omen	1,247	0.1619	−0.0999 [−0.1148, −0.0850]	−0.0178 [−0.0313, −0.0047]	77.7%	0.855
	Polymarket	1,381	0.0944	−0.0972 [−0.1113, −0.0828]	−0.1076 [−0.1274, −0.0881]	87.7%	0.942
	Total	2,628	0.1264	−0.0985 [−0.1088, −0.0881]	−0.0650 [−0.0774, −0.0525]	83.0%	0.905
GPT-4.1, same evidence	Omen	1,247	0.1797	−0.0821 [−0.1006, −0.0642]	—	77.5%	0.844
	Polymarket	1,381	0.2020	+0.0104 [−0.0043, +0.0252]	—	73.5%	0.810
	Total	2,628	0.1914	−0.0335 [−0.0455, −0.0212]	—	75.4%	0.824
Base model (qwen-14b-base)	Omen	1,247	0.2618	—	+0.0821 [+0.0642, +0.1006]	68.5%	0.747
	Polymarket	1,381	0.1916	—	−0.0104 [−0.0252, +0.0043]	74.0%	0.800
	Total	2,628	0.2249	—	+0.0335 [+0.0212, +0.0455]	71.4%	0.774
Fine-tuned, no warm start (qwen-14b-fine-tuned) — trained in this project	Omen	1,247	0.2113	−0.0505 [−0.0637, −0.0376]	+0.0316 [+0.0167, +0.0471]	69.8%	0.763
	Polymarket	1,381	0.1798	−0.0119 [−0.0182, −0.0054]	−0.0223 [−0.0389, −0.0066]	75.7%	0.811
	Total	2,628	0.1947	−0.0302 [−0.0372, −0.0231]	+0.0033 [−0.0085, +0.0147]	72.9%	0.787
Fine-tuned, with supervised fine-tuning (SFT) warm start (qwen-14b-sft) — trained in this project	Omen	1,247	0.1915	−0.0703 [−0.0856, −0.0554]	+0.0118 [−0.0039, +0.0276]	73.5%	0.790
	Polymarket	1,381	0.1681	−0.0235 [−0.0313, −0.0156]	−0.0339 [−0.0507, −0.0171]	77.9%	0.835
	Total	2,628	0.1792	−0.0457 [−0.0539, −0.0375]	−0.0122 [−0.0241, −0.0005]	75.8%	0.810

Lower Brier error is better, and a negative Δ means lower error than the reference; a Δ whose interval does not cross zero is a statistically clear difference. AUC (area under the ROC curve) measures how well a model sorts the markets that resolve YES above those that resolve NO, where 0.5 is a coin flip. Δ is simply the difference between the two Brier values in the table, and every row is scored on the same 2,628 markets. The market price is the price at the moment each context was captured, read from the platform's own record.

Left: final scores over all 2,628 markets. Right: the same scores as markets resolve, with 95% intervals. The SFT model leads on both measures for almost the whole window. The last two weeks, taken alone, favour GPT-4.1. The price is ahead of every model on both measures for the whole window.

From the above results, we can draw the following conclusions:

• Training beats the base model clearly, on both platforms (Omen and Polymarket). The fine-tuned model cut Brier error from 0.2249 to 0.1947, a difference of −0.0302. The SFT model reached 0.1792, a difference of −0.0457. The gain is not an artifact of the failure penalty: only the base model ever failed to return a usable probability, on 72 of the 2,628 markets, and without those markets the fine-tuned model is still ahead (−0.0128 [−0.0185, −0.0071]).
• On identical evidence, the SFT model is on par with GPT-4.1. Its pooled Brier error is slightly lower, but the margin is small and only just statistically clear, and the two models win in different ways: GPT-4.1 is the closer forecaster on about seven markets in ten, but when it is wrong it tends to be wrong with more confidence, and those misses cost it more than its many narrow wins gain. By platform, the SFT model is clearly better on Polymarket and level with GPT-4.1 on Omen, and GPT-4.1 still separates likely from unlikely outcomes slightly better overall (the AUC column). The fine-tuned model without the warm start is level with GPT-4.1.
• The market price is a better forecaster than every model, GPT-4.1 included. Simply repeating the market price at capture time gives a Brier error of 0.1264, against 0.1792 for the SFT model and 0.1914 for GPT-4.1 on the same evidence. This is of no use to a trading agent, though: a forecast that copies the price never disagrees with the market, so it implies no edge over it. Only a model that departs from the price, and is right when it does, can do better than the market.
• Fresher evidence explains half of the production tools' better performance. Those tools are what a trained model would have to replace, so it matters why we are behind, not just by how much. They run GPT-4.1 with a live web search, and answer each market again as it moves, so they see fresher evidence than anything here. On Omen the production tools scored 0.1682, against 0.1915 for the SFT model. GPT-4.1 on our frozen evidence scored 0.1797, right between the two. So half the gap is the evidence and half is the model. Giving the SFT model the same live evidence could close the evidence half. The 0.1797 score is what splits the gap in two, so we checked it. GPT-4.1 answered real production requests again under our rules, and came within 0.05 of the original answer 82% of the time.
• Better forecasts also gave better simulated trading results. We fed each model's predictions into the trading rule the agents use, and replayed the positions against the recorded prices and outcomes. These are simulated ROI figures, not live results. On Omen the ROI rose from +6.1% for the base model to +12.7% for the fine-tuned model and +18.3% for the SFT model. On Polymarket every model is negative, but only because we equalised the thresholds across both platforms to keep the comparison fair. Under the thresholds the agents actually use, the SFT model's Polymarket figure rises from −16.6% to −2.9%, and its Omen figure to +24.0%. The ranking is the same either way. This shows that training improved the forecaster, not that any model forecasts better than the market price.

What this evaluation establishes

A 14B open model, trained on our agents' own records, matched GPT-4.1 on the same evidence. The test was a hard one: the markets closed after its training data ended, the evidence was frozen before they were resolved, and it ran on a quarter of GPT-4.1's output budget.

The gains were consistent. Training cut Brier error by about a fifth from the base model, on both platforms, with intervals that exclude zero. The same ordering was held in simulated trading, where each training stage improved on the one before it.

Several directions follow from here. The evaluation deliberately froze the evidence, so the obvious next experiment is to let the model gather its own. Beyond that: larger open-weight models, several answers per context instead of one, and better calibration. The market price remains the forecaster to beat, and the protocol is now fixed, so every candidate takes the same test and its result can be compared directly with this one.