Learnings from Optimization Arena_

Dan opens with the question that motivated the whole project — one he and Benedict Brady started chewing on about six months before this talk. AI systems can be extraordinary at the numerical side of trading: train a model on quantitative data and it will find remarkable things. But ask the same models to explain what they're doing, or to design a strategy in advance — the "word cell" part of quantitative research — and they fall apart.

“If you ask the models to explain to you what they’re doing, or try to design a strategy for what they’re going to do… they’re very bad at it.”

His leading theory: unlike humans, agents lack an intuitive social sense. In adversarial games, humans naturally run the loop of "what is this person thinking that I'm thinking?" LLMs — otherwise brilliant — are bad at modeling other mindsReasoning about what another player believes and intends — "I think that you think that I think." The machinery behind strategic play, and a known weak spot for LLMs. and reacting to them. Strategies of the form "if you did X, it would cause someone else to do Y" — the bread and butter of game theory — reliably trip them up.

Which raised the research question behind Optimization Arena: agents can clearly compete inside mechanisms. Can we teach them to design mechanismsReverse game theory: instead of predicting play inside fixed rules, you design the rules so self-interested behavior produces the outcome you want. — to build a mechanism that behaves well under specified conditions, not just to break one?

That question became Optimization Arena — since moved to Paradigm and rebranded Paradigm Puzzles. The format: a set of open challenges where anyone can submit agents (or hand-crafted solutions) that optimize different components of a problem, scored on a public leaderboard. The numbers grew fast — thousands of players on the first couple of puzzles, more than twelve challenges live by the time of the talk, with new ones shipping weekly.

The current Paradigm Puzzles lineup includes the Simple AMM and Prop AMM challenges, a prediction-market market-making challenge, cryptography puzzles, and even an attempt on the unsolved fourth passage of Kryptos.

One immediate, slightly cynical discovery: an open leaderboard is an incredibly effective way to outsource your token spend. Put up a challenge and people will burn their own CodexOpenAI's agentic coding system, standing in here for the whole class of coding agents people pointed at the leaderboard. Running one for hours costs real money in model usage. budgets trying to climb it. If the thing you want is something an agent can find, you don't have to spend your own tokens — you just post a leaderboard and solutions arrive.

“Just put up a leaderboard and people will submit solutions to you.”

Which sounds amazing — and you genuinely do get a lot of compute thrown at your problem. The interesting lesson is what comes next: it turns out to be very hard to get effective research out of it.

The first puzzle: take a standard trading poolA pool of two assets where a formula sets the price. Uniswap V2's rule keeps the product of the reserves constant, so every trade slides the price along a curve. and decide what fee it charges. The fee doesn't have to be fixed — you can raise or lower it moment to moment, using only information visible on the blockchain. (In the jargon: a Uniswap V2–style dynamic fee strategy.) The design space is deliberately narrow. All you adjust is the fee, but there is a lot of subtlety in it.

Dan competed too — and held the top leaderboard spot on the first AMM challenge for about a day before getting, in his words, completely destroyed. His main trick is worth stealing: run an outer agent whose job is to figure out how to prompt other agents. The outer loop spawns inner agents in parallel, each trying different approaches, then evolves not just the best solutions but the best prompts.

The second benefit of that diversity: you're optimizing not only for incremental gains but for the big breakthroughs that only come from injecting entropy into the searchInstead of one agent hill-climbing, keep a population of attempts, select the best, mutate and recombine, repeat — genetic algorithms applied to prompts and harnesses.. (Benedict's writeup expands on this: without fresh entropy — multi-agent debate, literature-review steps, introspection, genetic-algorithm framings — a harness inevitably flatlines in a local optimum, grinding hyperparameters that barely matter.)

“You can basically try to evolve those, and end up finding not just better solutions, but better prompts.”

Thousands of submissions came in, and the codebases looked genuinely different. One person trained a neural net (implemented in Solidity, no less). Others wrote compact, elegant mathematical models by hand. Different agents, different harnesses, different philosophies.

And yet: they all plateaued at basically the same score — within about one hundredth of a percent of each other. The problem seemed to have a ceiling on how much could be extracted. Stranger still, when Dan and Benedict looked at out-of-sampleEvaluating on data the strategy never saw while being built. It guards against fitting the data — but not against fitting the rules of the game itself. runs, the top strategies weren't just equally profitable — their trades were synchronized.

“They basically did the exact same thing at the exact same moment, in response to the same signal, which I thought was very counterintuitive to me.”

Dan expected a diversity of strategies bumping into a shared ceiling. Instead, the problem apparently had a single optimum — and everyone found it.

It would be a good story if this were the discovery of an ideal AMM strategy. It isn't. What everyone actually converged on was the optimum of the simulation and the evaluation rules — not of markets. The testing was out-of-sample, so competitors weren't fitting a particular price path. They were fitting something deeper: the exact conditions of the world they were dropped into.

And that world was gentle in ways real chains are not. Inside the simulation there is no MEV, prices move by a known process, arbitrageurs behave exactly as specified, and the scoring function never changes. On a real chain, searchers are adversarial, prices jump on news, arbitrageurs adapt to you, and there are no fixed rules at all. Change any one of those and, as Dan puts it, all of this goes out the window.

“They were really hyper-optimizing to the rules of how we were evaluating them and exactly the conditions that they were under, which I think would break completely if you were, for example, on-chain in a real environment.”

Not all of it was simulator artifact. The most common and most powerful strategy is simple to state: keep your own running estimate of what the asset is really worth, instead of trusting the price your pool's formula reports. A pool's own price only updates when someone trades against it, so between trades it drifts out of date — and that staleness is exactly what arbitrageursA trader who profits when your pool’s price drifts from the real market price. Useful (they correct your price) and costly (every correction is money out of the pool). profit from. Trading signals leaked enough information to reconstruct where the market really was — and pricing around that estimate beat pricing around the midpoint by a wide margin.

Dan thinks this family of ideas could survive contact with reality — whether the estimate comes from an oracleAn external feed that brings off-chain data, like a market price, onto the chain — so a pool can quote around reality instead of its own lagging reserves., or from cleverer mechanisms that elicit the true price even in an adversarial MEV environment. A few other robust motifs showed up too, like setting fees proportional to volatility. But the bulk of the hyper-optimization effort went into solving "our version of the puzzle rather than the real puzzle."

So, lesson one of running an optimization arena: it is incredibly hard to pose a puzzle that yields interesting research — one where competitors end up happy in reality rather than in your simulation. There's a sim-to-real gap between auto-research challenges and research proper.

And it cuts both ways, which is the cruel part:

If the puzzle is too simple, it's an unrealistic simulation and a trivial rule will winExploiting quirks of the test rather than learning anything general. Goodhart's Law in one line: when a measure becomes a target, it stops being a good measure.. But if it's too complex, that actually makes it easier to overfit — and harder to extract the core insight you were fishing for. The solutions become less legible, which is the exact opposite of the point. And an overfit solution probably won't transfer to the real world even if it aces every out-of-sample test you run.

Which leads to the uncomfortable bottom line on auto-research challenges:

“Once you’ve actually really posed the challenge very well, you actually might as well just put your own agent on it.”

Unless you're severely budget-constrained and running the leaderboard as a nerd-sniping hack to harvest other people's tokens, the crowd doesn't buy you much: everyone converges on the same agent-orchestration tricks, and it doesn't take much to hit the ceiling of agent performance on black-box optimization problems. You mostly get back what Codex would have given you anyway — unless your challenge can genuinely elicit real human insight, which most of these don't.

Benedict's writeup adds a counterpoint: across ~20k submissions, skill on these challenges transferred across domains, and generalists with big token budgets beat domain experts. Harness-building may be a real skill even when the research output is thin.

audience questions reconstructed — see note below

Questions below are reconstructed; Dan's answers are near-verbatim.[2]

slides shown but not discussed

Two slides from Benedict's deck that Dan never discusses.[3]

The second: a critic agent that watches the loop from a distance and periodically improves the process itself — caching repeated operations, collecting extra data during verification.

Every term marked with a dotted underline above, in full — with the papers and primary sources behind each one.