A lovely & broad lay-person discussion of the OpenAI Navier-Stokes story and LLMs in mathematics.
We'd hopefully by now all know that hosted LLMs companies train upon their customers uploads and sessions (). It's clear this *"surveillance plagiarism" is a threat to cryptographers and security researchers who do really original work, as well as to any corporation using AI, but open weights models address this threat.
Vi Hart touches upon a broader array of topics: LLM psychosis. Modern mathematics was always a machine to be applied, ala the phrase "tenure machine", so AI doing so makes some sense. LLM proofs could retard the development of important abstractions, like how Aschbacher's announcement of the CFSG slowed research on the CFSG.
It stays solidly focused upon mathematics research however, without much direct discussion of applied topics like cryptography. Yet, there are fairly obvious impacts upon cryptography.
We correctly assume LLMs should overall raise our confidence in cryptographic software, because of auto-formalisation, eventually someone should figure out how to make LLMs write SE Linux profiles, etc and these are by far the lions share of the attack vectors.
Yet conversely, if LLM proofs cause mathematical subfields to be abandoned, then overall our confidence in problems being hard could decline, because organisations like IDA-CCR or China's version might not abandon some subtly relevant subfield. In other words, hybrid PQC+ECC protocols should become more important because of LLMs, not less.
Scott Aaronson also has an excellent post about LLM usage in mathematics too, somewhat more targeted towards mathematicians and computer scientists. It's slightly weakened towards the end by giving credence to AI doomer drama, but that's easily ignored.
Around (*), you should expect hosted LLM compajies would train upon data they promise not to train upon, either now or in future. In particular, there is a rich literature of non-cryptographic privacy preserving machine learning techniques, so similar work would buys them plausible deniability when training upon restricted data, but such techniques typically wind up being broken eventually.
Aside: Navier-Stokes lost in translation: Why Lean verification of AI autoformalisation does not guarantee correct natural language proofs