Efficiently using LLMs to find and fix vulnerabilities in live protocols_

I'm going to talk about using LLMs to efficiently find live exploits in live protocols — from what's essentially an energy angle: what would it take to audit everything that's deployed on-chain right now, and find all the bugs that are discoverable with LLMs as they are? A lot of this is going to be exploits you obviously want white-hat auditors to find and fix. So: what's the state of the art today?

The question decomposes into parts. First: are the LLMs actually good enough for this to be viable and practical? Second: where do we point them — because it's primarily not an intelligence question, it's a budget question. Then I can give you results from an audit I ran on a fixed budget, and finish with how to make the models better and this kind of discovery intelligence cheaper.

“It’s primarily not an intelligence question; it’s a budget question.”

Are LLMs actually good? They're good, but it's jagged.[2] The frontier ones are really good, and the open ones are catching up. The best benchmark we have right now comes out of Anthropic. Who here knows the DeFiHackLabs repo? It's a database of basically all past EVM exploits, replicated in FoundryThe standard Ethereum development toolkit. Its test runner can fork the real chain at any block, so an exploit can be replayed — or attempted fresh — against real state, locally.. There's a lot to learn from that data. So: run a benchmark of which LLMs reproduce which exploits — all the way from finding the bug to taking money out of the contract.

In that setting, frontier LLMs discover more than half of all vulnerabilities. Opus 4.5 discovers about two-thirds of the ones after its knowledge cutoff — so it has never seen those exploits.[3] And Mythos discovered twelve out of twelve recently exploited vulnerabilities after its cutoff.[4]

There are a couple of other benchmarks now — everyone is building these.[5] The pattern across them is the same: the benchmarks moved from spot-the-bug to write-the-exploit — the phrasing on the slide — and the newest frontier models clear bars that the previous generation failed badly.

Who here followed the Zcash vulnerability from a few months ago? The auditor — he's Zcash-affiliated — had a custom harness and was running every model through it to find bugs. The day Opus 4.8 came out, it was discovered.[6]

And funny enough, I ran the reproduction in vanilla Claude Code. You don't even need the harness. It's not about the harness and the tool calling — it's about the intelligence of the model. I told Opus to go find me an infinite money glitch by counterfeiting Zcash — and that was enough.[7]

“You don’t even need the harness. It’s not about the harness and tool calling; it’s about the intelligence of the model.”

The short version: the discovery skills are incredibly impressive. But they're also very expensive — these audits run to hundreds or thousands of dollars in API prices. So I want to figure out where to point the LLMs so we audit most efficiently, if the goal is to run through all of Ethereum and report all the bugs.

There are about 70 million smart contracts deployed. A lot of reduction needs to happen. Step one is deciding what we actually audit — which deployed protocols are high-value. And the line is genuinely fine: it's not Uniswap versus random junk; there's a long tail of obscure NFT protocols and the like in between. The rest of the pipeline is: run the LLM, and force its output through a deterministic grader on a fork.

The reframe that matters: the "harness" is a list of contracts, not a list of tools. The filter decides which contracts the expensive model ever sees — so the filter is where the leverage is.

My starting point was research from Anthropic's Frontier Red Team. They have a set of reduction heuristics to filter smart contracts, and one of them is quite inefficient — it only looks at ERC-20 contracts. The recallOf all the contracts known to have been exploited in the past, the fraction the filter keeps. High recall means the filter rarely throws away a contract that would have mattered. on past exploits was about 15% out of the box; a lot gets missed.

So I had an LLM experiment with different human-readable conditions — keep it readable, a handful of clauses, but lift recall on past exploits as much as possible while shrinking the candidate set. At the same number of candidates, that boosted recall to about 60%: almost all the past DeFiHackLabs hacks land inside a same-size set. Four times the exploit density, four times better use of budget. Then I trained a boosted decision treeA gradient-boosted decision tree (GBDT): an ensemble of small decision trees trained on features — here on-chain activity like value flowing through a contract and how many distinct callers it has — that learns to rank contracts by how likely they are to be exploitable. on on-chain activity features — value passed through, number of distinct callers, that kind of thing. That bought roughly another 5× density at the same set size. About 20× total, per unit of budget.[8]

I ran an early version of this on a thousand-dollar API budget — very, very tiny; after the filtering pipeline it comes to maybe ten bucks per contract. It found two keeper opportunities.[9]

The funny one: the DELTA Deep Farming Vault opportunity had been sitting there for four or five years. An abandoned protocol with a keeper reward — a public function that routes one percent of a target account's recycled tokens to whoever calls it, over a hundred decayed farmer accounts to loop through. My LLM discovered it on July 7th, and I was a little disappointed — I spent a thousand bucks and didn't get to report a billion-dollar ZK glitch — so I figured I'd get to it when I get to it. The next day, some other bots took the opportunity. They're roving. It could have been taken at any time in those four years; with LLMs, it's much more active these days.

More interesting: I had a very limited budget — would it have been better with a much higher one? The answer is unequivocally yes. There was an Aztec exploit a couple of months ago — more obscure. Aztec is a ZK rollup; there was an under-constrained ZK circuit in their older system, and you could take out all the money — about two million dollars.[10]

Check number one for me: is this contract in the shortlist my gradient-boosted classifier produces? Yes. But I didn't find the bug, because you can't find it at five dollars per contract. What if I put more inference budget into it? Then Opus not only discovered the hack — it discovered a novel, previously undisclosed way to steal the same two million dollars. I won't get into the details. The short version is that it's all about the cost of intelligence. What you can discover at five dollars versus fifty versus five hundred versus five thousand is materially different. In a program of "audit all of Ethereum, find all the bugs, fix all the bugs," the cost of intelligence is the critical factor.

“It’s all about the cost of intelligence. It’s all about your inference budget. What you can discover — five dollars versus fifty versus five hundred versus five thousand — it’s just very materially different.”

So there are a few approaches to RLVRReinforcement learning from verifiable rewards: instead of human ratings, the training signal is an objective check the environment can run — a test passing, a flag captured, a balance increasing. specifically for crypto and cyber vulnerability discovery. I'm not an expert, so I'll mostly skip it — but if anyone is looking into lifting model capabilities through RLVR, and specifically distilling frontier-model capability into cheaper intelligence, I would love to chat afterwards.[11]

One: LLMs are increasingly good. Two: it's all about where you point them, and the inference budget. Three: the cost frontier is very jagged — and that's a uniquely good situation. Your margin is my opportunity. You don't want to be where none of the LLMs can do anything, and you don't want to be where the benchmarks are saturated. You want to be where one model can discover things the others cannot — and then you can backport and diffuse that capability. Hopefully as defense: fixing the fuses ahead of all of it.

“The cost frontier is very jagged. Your margin is my opportunity. It’s a uniquely good situation.”

audience questions, lightly condensed

Askers are unnamed on tape; wording is lightly condensed from the transcript.[12]

Every dotted-underline term above, in full, with sources.