Valory Unveils Olas-Predict-Qwen3.8-27B: A Lean, Open-Weights Model That Approaches GPT-5.6’s Forecasting Abilities

Valory’s latest open-weight forecasting model in the Olas Predict model series comes within 1.2 percentage points of GPT-5.6 in accuracy at predicting real-world events. With just 27 billion parameters, small enough to run on a single high-end consumer GPU when quantized, it reached 67.3% balanced accuracy in Valory’s evaluation, compared with 68.5% for GPT-5.6.

The results suggest that a model doesn't necessarily need the broad factual knowledge of a frontier AI system to perform competitively at a specialized task, particularly when trained on relevant data. If smaller labs keep training specialized models on niche real-world data, this could matter for AI development as much as progress in large general-purpose models.

In an internal comparison across 6,824 researchable prediction markets, this new model, built on a stronger open-weight foundation, Qwen3.8-27B, cuts forecasting error by a further 22% compared with Valory's previous model. It also reduced Brier error by 32% against Valory’s historical GPT-4.1 baseline in the same reported comparison.

The new specialist reached 67.3% balanced accuracy, compared with 68.5% for the GPT-5.6 comparator. While GPT-5.6 still performs slightly better in this test, Valory now has a stronger specialist to build on, and a quantified gap to target in future training cycles.

Strong forecasting despite a large factual knowledge gap

What’s particularly encouraging is how close the two models perform on forecasting, despite a significant gap in their underlying factual knowledge. On Artificial Analysis’s AA-Omniscience v1, GPT-5.6 achieves approximately 59% accuracy, compared with 15.6% for Qwen3.8-27B, the starting point used to build Valory’s specialist forecaster.

Yet in Valory’s separate forecasting evaluation, when both models were given the same market evidence to work with, Valory's 27B model reached 67.3% balanced accuracy against GPT-5.6’s 68.5%, a gap of just 1.2 percentage points. Its Brier score was also 0.2124, compared with 0.2029 for GPT-5.6. That is strong specialist performance from an open foundation model with substantially lower factual recall, and a model that developers could adapt, self-host, and build on, under its license.

Building on the first Olas Predict model

This work builds on Olas-Predict-R1-14B, which was introduced by Valory in September 2026. Olas-Predict-R1-14B was trained on more than 200,000 predictions generated by Olas agents across live prediction markets. The first model reduced Brier error by roughly 20% against its starting checkpoint, and achieved forecasting performance comparable to GPT-4.1. The new experiment extends this program further with a stronger open-weight foundation that gets within 1.2 percentage points of GPT-5.6.

A strong open-weight foundation and domain training both contribute

A stronger starting model accounts for most of the improvement, but frontier-scale factual breadth does not appear necessary for competitive performance on this specialized task. Valory’s domain training provides additional gains over Qwen3.8-27B, reducing Brier error by 5% and expected calibration error by 12%. These results suggest that specialized training complements a strong foundation model.

Valory achieves this by training the model on reviewed forecasting examples and using reinforcement learning to improve how accurately it estimates the likelihood of different outcomes. A separate forecasting tool supplies market intelligence and gathers evidence, which the model then uses to produce its predictions.

Confidence for the next training cycle

Valory continues to close the gap with frontier models on forecasting via domain-specific training. Olas Predict offers a scalable data pipeline that is enabling these gains.

Olas Predict also creates a repeatable training loop. Olas agents generate forecasts, resolved markets provide verified outcomes, and those results can inform later training cycles. This growing set of evaluated predictions can support continued improvements in future forecasting models.

You can try the new model on Hugging Face, at valory/olas-predict-qwen3.8-27b.

APPENDIX

Results behind the announcement

This comparison is separate from the published 14B study, because Valory used markets created after GPT-5.6’s training cutoff.

Brier and ECE are better when lower; AUC and balanced accuracy are better when higher. Model names are shortened for readability.

The gains in concrete terms

Against the previous Olas-Predict-R1-14B model, the new model records 22.0% lower Brier error, 31.3% lower ECE and an 8.4-percentage-point increase in balanced accuracy. Against the historical GPT-4.1 baseline, Brier error is 31.7% lower and ECE is 49.0% lower. These are calculations from the table above.

Valory’s incremental training gains over Qwen3.8-27B are 5.0% lower Brier error, 12.3% lower ECE and 72.5% fewer malformed responses. The new model is 1.2 percentage points behind GPT-5.6 on balanced accuracy.

What these measures tell us

Brier error measures the quality of probability forecasts, including how strongly a model commits to an outcome. ECE (expected calibration error) summarizes how well stated confidence aligns with observed outcomes. AUC (area under the ROC curve) measures how well the forecasts separate YES and NO cases. Balanced accuracy gives equal weight to performance on the two outcome classes.

Scope of the comparison

The report describes common prompts and evidence at temperature 0 and a date filter tied to GPT-5.6’s stated February 16, 2026 cutoff. This supports a focused comparison of forecasting over supplied evidence.