jlebar · 46 replies
sigmar ·
>The speed with which we were able to produce this proof demonstrates that it is now possible to formalize large swaths of mathematics, which may both catch errors in the common body of mathematical proofs and reduce the burden of refereeing new work.

^ this section should have been in the first few paragraphs imho. Explaining why this is relevant shouldn't be so far down.

imranq ·
Note that this proof while impressive does not add any value to mathematics as a human pursuit. But it does show we can throw these LLM beasts at much gnarlier problems than we could have imagined previously. Maybe even formally verify papers the day they are posted?

I'd love to see an e2e compiler or OS kernel verification or Full-stack chip design with formal equivalence checking at each stage that would be pretty cool.

What else is interesting is how they staged this problem : (a) maintain an explicit DAG/roadmap of sub-goals rather than one flat prompt, (b) separate statements from proofs so many agents can work on different nodes without stepping on each other, (c) keep a natural-language index alongside the formal one so search/reuse works... I feel like this is the future of long horizon agents and how you can do work that's making the most of every agent. This approach will likely be baked into the next versions of coding harnesses

herbcso ·
So I don't know Lean or Mathematics to any degree to really be able to say this with any level of confidence, but speaking from a pure software engineering backgrouand, how do we know that 13 MILLION lines of Lean code are bug-free? It seems to me that for a mathematical proof, bug-free would be an absolute requirement. Maybe the structure of Lean imposes that, I don't know, but that seems highly unlikely to me. That just feels like a LOT of code to be comletely error-free... What am I missing here?
glimshe ·
"The proof is not the modern proof which I have been formalizing myself following ideas of Khare, Taylor etc, but the Darmon–Diamond–Taylor exposition from 1995 of the Wiles–Taylor–Wiles argument, via the Langlands–Tunnell theorem and Ribet’s level-lowering theorem. Anthropic’s repository develops Fontaine theory (to study flat deformations of Galois representations) and develops enough of Mazur’s work on the Eisenstein ideal to conclude that no Frey curve can have a point of order p>=17. This means that their FLT proof only works for p>=17, however FLT was already formalized for odd regular primes by Best-Birkbeck-Brasca-Rodriguez, and the smallest irregular prime is 37, so it’s all good."

My question to any mathematician reading this: does the above make ANY sense to you?

I ask that because I can read most technical material related to computer engineering, programming, hardware specifications etc. Even if I don't fully understand all details, I can follow them pretty well. So I wonder if professional mathematicians can look at the above and still make sense of it like experienced software engineers do for computer stuff.

m_w_ ·
> Along the way, it wrote 13 million lines of Lean and proved 29,500 intermediate theorems.

Pretty insane. I suppose it lends further credence to the idea that anything that can be shown to be correct can be done by a model.

davmre ·
> a team of agents completed the proof in a little under two weeks, consuming about six billion output tokens from a general-purpose internal research model roughly comparable to Claude Fable 5.1.

At $50/M output tokens, this would have cost on the order of $300k (plus a bit for input/prefill tokens) at API rates.

mettamage ·
First of all, this is an amazing result. Second, I'm not too surprised, given all what has happened before.

The thing is: LLMs are not grounded in reality enough as much as we are. Using Lean is exactly what that is: grounding LLMs in reality.

We have (at least) 30 FPS vision, and can detect 5 ms audio delays, we do that in real-time. LLMs have access to some images and large amounts of text. Their propensity is to predict the next token. So the propensity to be additive and just say something (aka predict the next token) is higher than predicting something to stop.

If LLMs would have: - 30 FPS vision - similar hearing ability - an ability to feel their lived experience - consequences to their "life"

They'd be making more intelligent decisions than they are doing now. Simply because they have more context.

Because in this sense, we have a lot more context than LLMs. Yet, I see people sometimes treating them as if they are at the same level as humans because their intelligence is similar. And that might be true, but where they get their data from is vastly different. Given our tasks, they are at a disadvantage. They need to sense more of reality.

Have fun sharing the room with these digital intelligences. Given the topics they can consume, they are already better generalists than any individual. I might be wrong of course, I'd love to meet any individual that's a better generalist than an LLM.

cyode ·
I saw the 1996 FLT documentary in high school calculus class. For me, it forever cemented that archetype of modern math researcher at the top of my mental “smart” totem pole.

It also convinced me I had no interest in that path. Setting aside the grinding work of producing a proof that can only be reached by existing years in the abstract and hyper niche isolation of the problem space (not to mention that you might never discover it or that it DNE), the anguish of the output being a paper or presentation or some other artifact of human symbology (_words_, really) that could at any moment be refuted by a single observation of a single mistake—-that sounded like hell to me.

An equivalent high schooler today probably sees things differently, in light of this news and the undeniable implications of LLMs on mathematics. Sturdy autoformalization tooling should with time completely dispel the aforementioned anguish, once our confidence in converting a human proof to Lean/etc. reaches that of a compiler translating Java application language to bytecode. Errata may always exist, but in practice these new methods will do wonders for rigor and peace of mind.

(I’m far less confident re novel discoveries. There’s too much chance of derivative findings based on something part of the training looking like genius but really just tiptoeing on the shoulders of humans, whereas autoformalization is absolutely convincing to me as transformative, particularly to check correctness of AI outputted proofs as mentioned in the post.)

Vakaiser ·
We'll increasingly observe announcements of this kind as AI tooling scales. As impressive as agentic coding is, it pales in comparison to the value proposition of medical, mathematical, and physics research.

I optimistically expect to witness the advent of a global 'panacea' in my lifetime thanks to AI's efforts. Cost effective large scale genetic engineering, a cure for every disease, potentially even a cure for aging.

The future is both beautiful and terrifying.

henryrobbins00 ·
Back in February, I was talking with my PhD advisor about using Lean to formally verify automated optimization modeling outputs. It eventually turned into this paper [1]. It’s been truly incredible to see how much the frontier models have progressed in both autoformalization and automated theorem proving in the last six months. Back in February, it was cool to see them prove the validity of some simple cutting planes. Now it can churn out a min-cut max-flow duality formalization (not to mention FLT). Very exciting times!

I’ll also share a Python package I wrote for automated theorem proving that has been super useful in my own research [2].

[1] https://arxiv.org/abs/2608.25220

[2] https://github.com/henryrobbins/open-atp

KaiserPister ·
13M LoC, are we sure it didn't exploit any latent issues in the lean proof system?
satnhak ·
I used to attend Kevin's Number Theory seminars at Imperial College many years ago and he's both a first rate mathematician and a very nice person. His blog has got me interested in maths again. Considering how pro AI he is and that he's been working on this problem for such a long time I'm a bit disappointed that Anthropic didn't involve him directly in this work. However, I think it's important to remember that without all of the work Kevin and people like him have done, the machines wouldn't be able to do this.
chvid ·
Looking forward to the 5 billion LoC proof of the Riemann hypothesis.
jeanmichelselli ·
I'm a mathematician and I'm not sure one should believe those results right now.. An automatic formalization requires a system of logic rules to be applied, which is not something LLMs are great at (remember the Apple paper a while ago?). I'm very curious to see how the community will react after the initial hype.. so far, it's being quite disappointing..
mikmoila ·
"The effort succeeded when we switched to using Prove2Me, an open collaborative platform for formalizing mathematics designed by Tianyi Peng and his collaborators at Columbia University."

So in the end, it required tooling crafted by humans.

ojo-rojo ·
I'm really impressed by mathematicians. It's cool that Fermat had the intuition to conjecture that "aⁿ + bⁿ = cⁿ" could not be satisfied for n > 2, and that other mathematicians can create proofs, and that others still can understand AI's formulation of those proofs. Really cool.
margorczynski ·
With how capable and cheap automatic proof verification is becoming I wonder how many proofs assumed to be true by almost all of the math community will be proven false. And not by some marginal easy to fix error by some fundamental flaw in reasoning.
crawshaw ·
More (strong) evidence that agents make formal methods far more useful. The cost of creating that Lean proof has dropped dramatically.

Hopefully this helps mathematicians. It seems very clear to me that it will help software engineers apply formal methods to more of our software.

sva_ ·
Hmm kind of funny, some years ago someone claimed LLMs can do math, and I replied if it could prove fermants theorem:

https://news.ycombinator.com/item?id=33176996#33177939

> Now try to make a computer prove that there are no natural numbers a,b,c; so that a^n + b^n = c^n for any n > 2.

> > Shifting the goal posts a bit, aren't we?

I guess the goalposts did change a bit, and in a pretty short time.

kristjansson ·
Well, time to set down the glass beads and dive into a an alpine lake.
chi_features ·
There's a wonderful documentary by BBC Horizon with Andrew Wiles from 1996 – highly recommend! I saw it in the 90's and it's a documentary for everyone. It captures the effort, struggle, highs and lows of a 7 year effort working on Fermat's Last Theorem.
kzrdude ·
The part about prove2.me was interesting. That means that a co-working tool was instrumental in the project, and I think AI companies will take note of this. Is this proof specific or will we need to give agents access to JIRA or similar tools to solve large projects in the future?
fspeech ·
First I have to say this is sooner than expected, even though I never doubted that this could be done. I am grateful that they dedicated resources to accomplish this. It is clear that agents are very good at discerning and holding onto very weak signals from RL traing on long horizon tasks, so much so that in my own experience even very chaotic agent thinking can converge to meaningful solutions if there is a verifier. I have not dug through the proof yet so I don't know how readable it is to a human. But it has been a dream of mine to understand the FLT proof. I think LLMs will be a big part of making it truly accessible to humans.
vatsachak ·
This is quite useless actually. The whole point of formalizing FLT was to clean up modern number theory into reusable abstractions that prove it.

If its 13 million LoC, it might involve so much spaghetti that its unusable other than the result

atleastoptimal ·
It seems clear AI has the potential to perform any cognitive task at far greater speeds, reliability, and scale than any human. The question is whether it will be allowed to scale to that point, and what will happen to humans after this occurs.
estetlinus ·
I can recommend the book telling the full story behind Fermats Last Theorem (by Simon Singh). It’s quite fascinating, and paved with really, _really_ weird characters each chipping in on the final solution.
vagab0nd ·
> it wrote 13 million lines of Lean

Is this basically like opening up a black box and seeing 13 million gears all rotating seemingly randomly and still having no idea how the machine actually works?

deleted · [deleted]

[comment deleted]

throwaboat ·
I wrote a similar DAG-based verifier as a skill a few months ago: https://github.com/sethlei/Warrant . The thing mine has that I didn't see in their's is a verification of the composition rules.

Mine also does more than just math.

deleted · [deleted]

[comment deleted]

black_knight ·
I wonder if any piece of the lean code is in a shape which means it could be contributed to one of the Lean libraries.

My experience is that it takes a lot of human input to make Fable write code nice enough for a formalisation library others can work on. But since this is certainly a lot of prerequisites formalised as well, it would be nice if not all of the effort was wasted on one capstone proof!

deleted · [deleted]

[comment deleted]

throw567643u8 ·
13 million lines of code, a lot of which is new to Mathlib. So it hasn't built on what is already there but synthesised a bunch of new stuff.

LLM generated Lean code in the past has been known to exploit bugs in the Lean kernel, it would be foolish to rule this out happening again.

Goofy_Coyote ·
For math illiterate people like me, my understanding is that FLT was already proven, but the proof was beyond complex, certainly for mere mortals like me, and now Claude has codified it, correct?
richard_chase ·
Anyone know of a good Lean tutorial? I've played around with it a bit but never really learned it properly.
vmilner ·
Formalisation of the classification of finite simple groups must be on someone’s ‘moonshot’ list.
amelius ·
They should let AI work on it until the proof fits in the margin of a page.
vitriol83 ·
i find this and other efforts from anthropic somewhat antisocial. technically they have achieved their goal, but in a way which does not benefit mathematics or humanity. Kevin Buzzards headline goal was to formalise FLT, but i’m sure the real aim was to create a formalised library of mathematics which is comprehensible to humans. By solving these famous problems by brute force, they are disincentivising the important work of making it digestible for everyone else, and so in my view this work in particular has negative societal value.
prometheus1992 ·
Can someone with more knowledge help me with this silly question in my head?

>>Along the way, it wrote 13 million lines of Lean and proved 29,500 intermediate theorems

Did a human check the 13 million lines of code? How does QA'ing this type of work works?

dextrous ·
Ok, let’s get a rabid pack of agents cranking on P = NP? next!