Bigger Wasn’t Better This Time: Fine-Tuning Nemotron 120B for Forecasting

A bigger model that did not forecast better

Olas core contributor Valory fine-tuned a 120 billion parameter open model to forecast prediction-market outcomes, using the same training recipe that works on its 14 billion parameter model, Olas-Predict-R1-14B. The large model is a mixture-of-experts, a design that stores a lot of knowledge but uses only a small part of the network for each token. The result was clear: it forecasts worse than the smaller dense model Olas-Predict-14B, and training did not improve it at all over its untrained state. This is not about how many parameters a model has, but about how it spends them. This post explains what Valory tried, why Valory thinks it did not work, and what it changed in how Valory picks models.

What Valory tried

Valory's forecasting model is trained with reinforcement learning. The model reads a market question and the web context an agent gathered, writes out its reasoning, and ends with a probability. Valory scores that probability against the real outcome and pushes the model toward the answers that scored better. Valory also warms the model up first by teaching it to imitate good reasoning examples. On the 14B dense model, this recipe raises balanced accuracy well above the untrained base and matches a far larger proprietary model.

The natural question was whether a bigger model would go further. Valory picked NVIDIA's Nemotron-3-Super-120B, an open mixture-of-experts model. On Artificial Analysis’s Intelligence Index, Nemotron 3 Super scored 13, compared with an estimated 8 for DeepSeek-R1-Distill-Qwen-14B, the base model used in the Olas-Predict-14B fine-tuning. A mixture-of-experts model holds many specialist sub-networks and routes each token through only a few of them. Nemotron has a total of 120 billion parameters but uses about 12 billion for any given token. That design makes it cheap to run for its size, and it was the cheapest large model Valory could train through a hosted training API, Tinker by Thinking Machines Lab, so this time no direct GPU access was needed on Valory's side.

Valory ran the full recipe: reasoning examples generated by the model itself, a warm-up pass on the ones that scored well, then reinforcement learning over the same training markets and with the same settings as the 14B run. The only change was the base model.

What happened

Scored on 11,496 held-out market predictions that neither version had seen, every metric is unchanged within noise. The trained model is not better calibrated, not better at telling YES markets from NO markets, and it produced slightly more answers without a readable probability. Training did nothing useful.

Valory scored the fine-tuned 14B model on exactly these cases too. On the common test set, the trained 14B model has lower calibration error, higher accuracy and balanced accuracy, and fewer unreadable answers than the trained Nemotron. Nemotron has slightly lower Brier error: 0.267 versus 0.270.. The 14B model figures on this subset differ a little from the ones in the earlier post, because that post scored a different slice of the test set. The ranking is the same either way: the small dense model wins.

Why Valory thinks this happened

The same data, reward and settings improved the 14B model but did not improve Nemotron. This shows that the training recipe did not transfer successfully. It does not show which part needs to change.

What differed was the model, and a possible explanation is how the model uses its experts. In a dense model every parameter takes part in every answer, so each training step adjusts the same network and the skill builds up in one place. In a mixture-of-experts model a router picks a few experts for each token, and that choice changes from token to token and from question to question. The path through the network is different almost every time. A training signal that is already weak and noisy, a single yes or no per market, then gets split across many experts, each of which sees only a fraction of the updates and only some of the reasoning. This provides a potential explanation on why the model could not build a consistent forecasting skill.

The conclusion is therefore specific to this experiment: Nemotron did not improve with the existing recipe and training budget. Whether more data or a better-suited recipe would change that remains open.

What Valory learned

• The architecture matters as much as the size. For a task where the model has to learn one consistent skill from a sparse reward, a network whose active path changes with every token may be harder to train than a dense one. The next model selection will favor dense architectures, and treats a mixture-of-experts as a candidate only when its training story is already proven on a similar task.
• Check for movement early. A full run is only worth its cost if the first evaluations show the metrics moving. Valory now treats a flat early curve as a stop signal rather than something to wait out.
• Fix the output format before training on it. A base model that often fails to produce a readable probability needs that solved first, or the training signal is spent on noise.
• A negative result is still a result. One run ruled out a whole class of models for this task, at a fraction of what a wrong full-scale run would have cost, and showed where not to spend the next budget.

The next experiment applies these lessons: a larger dense model, with every parameter in use on every token, and an early go/no-go check before the full run.

Appendix: how these results were obtained

The data. Every sample is one resolved market prediction from Olas live agents on Polymarket and Omen: the question, the web context the agent gathered, and the real outcome. The data is split by market, so no market appears in both training and test. The 11,496 cases above are the held-out test predictions for which every compared system produced an answer.

The setup. Training ran through a hosted training API on the base model's own weights. Valory generated four reasoning traces per market for 5,000 training markets, kept the traces whose final probability beat the market's base rate, and trained a small adapter to imitate them. Reinforcement learning then started from that adapter, sampling four answers per question over a fifth of the training split, with the Brier score against the base rate as the reward. The checkpoint with the best balanced accuracy on the evaluation split was kept. Every setting matched the 14B run except the base model and a longer answer limit, since Nemotron writes longer reasoning.

The Metrics. Brier error is the squared gap between the predicted probability and the 0 or 1 outcome. Calibration error measures whether the model's confidence is justified. Balanced accuracy measures whether it gets the direction right, with 0.5 being a coin flip. Answers without a readable probability are scored as a neutral 0.5.

Notes

• DeepSeek-R1-Distill-Qwen-14B intelligence index. https://artificialanalysis.ai/models/comparisons/deepseek-r1-0120-vs-deepseek-r1-distill-qwen-14b
• NVIDIA's Nemotron-3-Super-120B intelligence index. https://artificialanalysis.ai/models/comparisons/grok-4-3-vs-nvidia-nemotron-3-super-120b-a12b