Dataset
Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce MathNet, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems.
MathNet spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts.
MathNet supports three tasks: (i) Problem Solving, (ii) Math-Aware Retrieval, and (iii) Retrieval-Augmented Problem Solving. Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that RAG performance is highly sensitive to retrieval quality; for example, DeepSeek-V3.2-Speciale achieves gains of up to 12%, obtaining the highest scores on the benchmark. MathNet provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark.
The examples section makes the benchmark concrete. MathNet contains standard text-only problems, diagram-heavy geometry, and retrieval pairs where the challenge is not exact lexical matching but mathematical equivalence or structural resonance.
MathNet-RAG pairs problems by three levels of mathematical similarity — from strict equivalence to loose thematic clustering.
| Mode | Problem A | Problem B |
|---|---|---|
| Invariance | ||
| Syntactic Equivalence | Find $f:\mathbb{R}\to\mathbb{R}$ such that $f(x^2-y^2)=(x-y)(f(x)+f(y))$. | Find $g:\mathbb{R}\to\mathbb{R}$ such that $(g(a)+g(b))(a-b)=g(a^2-b^2)$. |
| Reformulation | Let $a_i>0$. Prove $\sum_{i=1}^n \frac{a_i}{a_i^2+a_{i+1}a_{i+2}} \le \sum_{i=1}^n \frac{1}{a_i+a_{i+1}}$. | Let $a_i>0$. Prove $\sum_{i=1}^n \frac{a_i^2}{a_i^2+a_{i+1}a_{i+2}} \ge \tfrac{1}{2}$. |
| Transformational | Find all $x\in\mathbb{R}$ such that $4^x+6^x=9^x$. | Find all $x\in\mathbb{R}$ such that $(2/3)^x+(3/2)^x=5/2$. |
| Structural Resonance | ||
| Generalization | For $k\ge1$, prove that $k$ divides $\binom{n}{k}$ for all $n\ge k$. | Show $\binom{n}{m}\equiv\prod\binom{n_i}{m_i}\pmod{p}$, where $n=\sum n_i p^i$, $m=\sum m_i p^i$. |
| Common Lemma | If $ab+1\mid a^2+b^2$, show $\frac{a^2+b^2}{ab+1}$ is a perfect square. | If $a^2+b^2+c^2=k(ab+bc+ca)$, show $k\in\{1,2,3\}$. |
| Structural Reduction | Prove that $4^n+2^n+1$ is never prime. | Prove $2^{2n}+2^n+1$ is divisible by $3$ for all $n$. |
| Affinity (Thematic) | ||
| Affinity | Show the largest prime factor of $\binom{2n}{n}$ is greater than $n^{2/3}$. | For every $n>1$, there exists a prime $p$ with $n<p<2n$. |
Table: Taxonomy of mathematical similarity with Olympiad examples. Invariance captures strict equivalence under reformulation; Structural Resonance reflects shared lemmas or reductions; Affinity denotes looser thematic clustering.
Rather than treating evaluation as a single benchmark score, MathNet separates the problem into three linked tasks. This helps distinguish failures of reasoning from failures of retrieval, and makes it easier to study when external mathematical context is genuinely useful.
The statistical view is useful for understanding why the benchmark is hard. MathNet is not only large; it is heterogeneous in language, topic, contest type, and solution length, which makes naive transfer from smaller math benchmarks unreliable.
MathNet dataset statistics. (a) Contest type distribution. (b) Solution length compared to existing benchmarks — MathNet solutions are substantially longer. (c) Problems per year. (d) Topic and sub-topic distribution. (e) Language distribution: 74% English, 26% non-English across 17 languages.
The dataset is not just scraped from PDFs. The pipeline combines OCR, segmentation, normalization, and human verification so that the final benchmark preserves both the original problem statements and the long-form solutions needed for evaluation.
Data extraction and curation pipeline. PDF booklets are processed via OCR to extract markdown text, segmented into problem–solution blocks, normalized with GPT-4.1, and verified by human experts to produce the curated dataset.
We close with the main quantitative results. These charts summarize the benchmark at a glance: reasoning models still have meaningful headroom on Olympiad solving, retrieval remains difficult at low recall depths, and expert retrieval gives the most reliable gains in RAG.
MathNet-Solve-Test
Problem Solving on MathNet-Solve-Test (6,400 problems). The chart uses the paper's overall micro-average accuracy values to make the ranking legible on the page.
Takeaway: LMMs with reasoning are clearly strongest overall, but even the top model remains well below perfect performance.
MathNet-Retrieve
Math-Aware Retrieval on MathNet-Retrieve (10,000 anchor problems). This chart uses the paper's aggregate “All” Recall@1 and Recall@5 values.
Takeaway: Recall@1 stays very low even for the best models, while Recall@5 is much stronger, showing that mathematically equivalent retrieval is still unreliable at shallow depths.
MathNet-RAG
Retrieval-Augmented Problem Solving on MathNet-RAG (35 problems). These bars highlight the most legible comparisons from the paper across zero-shot, Embed-RAG, and Expert-RAG settings.
Takeaway: expert retrieval most often gives the strongest gains, but improvements remain model-dependent and grading-dependent.
For implementation details, benchmark construction choices, and the full experimental setup, the paper remains the primary reference. The citation is included here at the end for convenience.
@inproceedings{alshammari2026mathnet,
title = {{MathNet}: A Global Multimodal Benchmark for Mathematical
Reasoning and Retrieval},
author = {Alshammari, Shaden and Wen, Kevin and Zainal, Abrar and
Hamilton, Mark and Safaei, Navid and Albarakati, Sultan and
Freeman, William T. and Torralba, Antonio},
booktitle = {International Conference on Learning Representations},
year = {2026},
url = {https://mathnet.csail.mit.edu}
}
For questions about the dataset, benchmark, or paper, reach out to shaden@mit.edu.