Reviewing Epoch AI’s newsletter on FrontierMath, Aletheia, and First Proof — covered in IEEE Spectrum.
What’s New
Epoch AI has expanded their FrontierMath benchmark to include genuinely open problems from research mathematics — problems that professional mathematicians have attempted and failed to solve. This goes beyond their original benchmark (which used extremely difficult but solvable problems) into territory where the correct answers aren’t known by humans either.
The newsletter also highlights two related initiatives — Aletheia and First Proof — suggesting a broader push toward proof verification and novel proof generation benchmarks. Epoch researcher Greg Burnham frames the proliferation of math benchmarks as “a more-the-merrier situation.”
Why This Matters
The arms race between AI capabilities and benchmarks continues to accelerate. The original FrontierMath was already considered brutally hard — most frontier models scored in the single digits when it launched. We covered Gemini 3.1 Pro’s performance on it recently. Now Epoch is preemptively moving to open problems, anticipating that current benchmarks will saturate.
This is a smart defensive move against two threats:
- Capability saturation — models are improving at math faster than new benchmarks can be written
- Data contamination — if your benchmark uses solved problems, training data leakage can inflate scores. Open problems are contamination-proof by definition.
Our Take
If models start making progress on genuinely open problems, that’s qualitatively different from solving known problems faster. It would represent actual mathematical discovery — not pattern matching, not memorization, but novel reasoning.
We’re not there yet. But the fact that Epoch is building the infrastructure to detect it when it happens is exactly the kind of forward-looking measurement work the field needs. Worth watching for anyone tracking AI capability timelines.