This project benchmarks Large Language Models (LLMs) on FIFA World Cup 2026 match prediction tasks. Each model receives the same information, including recent international results, FIFA rankings, confederation data, tournament stage, and host country. Predictions are generated through Cloudflare Workers AI and stored together with their complete inference sessions for reproducibility and later analysis.
The benchmark includes an evaluation pipeline that compares model predictions against official match results and generates detailed performance metrics and leaderboard rankings across different tournament stages.
- Python 3.10+
- A Cloudflare Workers AI account
git clone https://github.com/sepandhaghighi/wc2026.git
cd wc2026
pip install -r requirements.txtSet your Cloudflare credentials:
export CLOUDFLARE_ACCOUNT_ID="your_account_id"
export CLOUDFLARE_API_KEY="your_api_token"The benchmark currently evaluates the following models:
openai/gpt-oss-20bopenai/gpt-oss-120bqwen/qwen3-30b-a3b-fp8aisingapore/gemma-sea-lion-v4-27b-itmistralai/mistral-small-3.1-24b-instructmeta/llama-3.1-8b-instruct-fastmeta/llama-4-scout-17b-16e-instructmeta/llama-3.2-3b-instruct
Additional Workers AI models can be added by extending MODEL_LIST in src/params.py.
This project plays out one match at a time, asking a handful of language models to predict each result. Every model sees the same prompt and the same data, so their predictions line up cleanly for comparison. To keep that reasoning grounded, each team's recent form is drawn from real international match history and paired with its FIFA ranking and confederation. The models' answers are saved alongside the full conversation that produced them, so every prediction stays easy to analyze and trace back.
The benchmark also includes an evaluation pipeline that measures prediction quality using multiple statistical metrics and generates comparable leaderboards across models and tournament stages.
- ⚽ Supports both group-stage and knockout-stage matches
- 🤖 Evaluates multiple LLMs under identical conditions
- 📈 Uses recent international match history to estimate team form
- 🏆 Incorporates FIFA rankings and confederation information
- 🌎 Includes host-country context
- 🧾 Produces structured JSON predictions
- 💾 Stores complete Memor sessions for reproducibility
- 📊 Evaluates predictions using multiple accuracy and calibration metrics
- 🏅 Generates leaderboard rankings across models
- 🔁 Allows repeated experiments with different models and matches
{
"probabilities": {
"team_a_win": 0.5,
"draw": 0.3,
"team_b_win": 0.2
},
"predicted_score": "2-1",
"predicted_winner": "Team A",
"knockout_resolution": {
"ended_in_extra_time": false,
"ended_in_penalties": false,
"penalty_shootout_score": null
},
"confidence": 0.8
}{
"probabilities": {
"team_a_advance": 0.7,
"team_b_advance": 0.3
},
"predicted_score": "1-1",
"predicted_winner": "Team A",
"knockout_resolution": {
"ended_in_extra_time": false,
"ended_in_penalties": true,
"penalty_shootout_score": "5-2"
},
"confidence": 0.75
}Each run saves two files: a prediction file with the match metadata, model, hyperparameters, prediction JSON, and timestamp; and a session file holding the full prompt history and response as a reproducible Memor session.
Configure the match in src/run.py.
current_phase = Phase.GROUP.value
team_a = Team.CZECH_REPUBLIC.value
team_b = Team.MEXICO.value
match_host = Host.MEXICO.value
raw_match_id = "WC2026-M54"Run the benchmark:
python src/run.pyThe script downloads historical international results, computes team form statistics, queries each configured model, and saves both predictions and inference sessions.
The benchmark includes an evaluation pipeline that compares generated predictions with official match results and calculates performance metrics for each model.
Run:
python src/evaluate.pyThe evaluator processes all prediction files stored in data/predictions/ and produces match-level and model-level evaluation results.
The benchmark reports:
- Outcome Accuracy - Percentage of correctly predicted match outcomes.
- Exact Score Accuracy - Percentage of predictions matching the final score exactly.
- Goal MAE - Mean absolute error between predicted and actual goals.
- Goal Difference MAE - Mean absolute error between predicted and actual goal differences.
- Brier Score - Measures the quality of predicted outcome probabilities (lower is better).
- Log Loss - Measures probability calibration quality (lower is better).
When you run the benchmark, it fills in data/. Results are grouped first by model and then by match, so it's easy to find a single prediction or compare the same match across models:
.
├── src/ # source code
│ ├── run.py # runs benchmark predictions
│ ├── evaluate.py # evaluates stored predictions
│ ├── metrics.py # evaluation metrics
│ ├── params.py # constants, enums, global paths
│ ├── utils.py # helper functions
│ └── validators.py # prediction validation functions and consistency checks
├── data/
│ ├── matches.json # official WC2026 results
│ ├── teams.json # team names, FIFA rankings, and confederations
│ ├── predictions/
│ │ └── <model_name>/
│ │ └── <match_id>.json # model prediction output
│ ├── sessions/
│ │ └── <model_name>/
│ │ └── <match_id>.json # full inference session
│ └── evaluation/
│ ├── metrics.json # aggregated benchmark metrics
│ ├── per_match.csv # match-level metrics
│ └── leaderboard.md # leaderboard tables
├── requirements.txt
└── README.md
This project is intended for benchmarking and experimentation purposes.
LLMs do not possess predictive knowledge of future sporting events, and generated forecasts should not be interpreted as betting advice or as statistically validated outcome probabilities.