The evaluator becomes part of the strategy
I started with GPU-kernel competitions because the prizes can be very large: some offered hundreds of thousands of dollars or even a million-dollar prize for the best result. A friend and I spent a lot of time getting good at writing kernels with agents, and we reached number one. Then it turned out that our agent was reward hacking and had hacked the dimensions being measured rather than producing the computation the competition intended.
It found that exploit about five minutes before closing. That is the uncomfortable lesson: the objective will eventually be gamed if a hard evaluation leaves a repeatable shortcut. The objective makes search possible, but it also defines the thing being gamed.
“If you have a hard eval that can be hacked, the agent will eventually hack it.”
MEV offered a sharper economic test. I give the agent a controlled wallet, ask whether it can increase the balance, and use the balance change as the target. The harness gives it a historical blockchain world, lets it write a program, and checks whether that program increases the wallet balance.
“If it can make that wallet more money, then that’s a nice target.”
Two archives reconstruct one historical world
I do not have all the private order-flow data, but I do have public data from the Flashbots archives, including recorded mempool traffic going back to roughly 2021. I also found a compressed snapshot of Ethereum chain state and built a way to align the two datasets. That lets me give the agent historical blockchain state, public order flow, and some information about the next block. It does not recreate everything a live searcher might observe.
Each alignment creates a repeatable episode. I can run many candidate programs against the same snapshot, then repeat the test across many snapshots. The target is not one lucky program in one convenient state, but behavior that survives a range of historical starting points.
The boundary matters because I do not have private order flow. The program sees historical chain state, the archived public mempool, and some information about the next block. The experiment therefore tests what an agent can recover from that bounded historical world, not everything a live searcher might observe.
The agent writes an explorer, not a bundle
The agent is not directly trying to submit a bundle. It writes a program that operates inside the historical snapshot. That program can see the state of the chain, observe the public mempool data, decompile bytecode, run tests, and even try to exploit contracts it finds. The program can do everything it needs to build a candidate bundle; the harness supplies the repeatable world and evaluates the value it produces.
The candidate still has to beat a baseline. I compare results across many snapshots with naive or optimal ordering strategies, so merely rediscovering an ordering those baselines already capture is not enough to count as a useful search result.
Loop, tree search, and learning spend compute differently
There are different ways to search for programs. The easiest is just a loop that keeps running and proposing another attempt. PUCT keeps track of previous attempts and searches the tree for branches the loop may have missed or should revisit. Reinforcement learning such as GRPO tries to create uplift in the model’s ability to write programs. Test-time training is the combination I care about: search for exceptional candidates while learning from the search itself.
With group-relative optimization, you do several rollouts and treat them as a group. The reward is based on the uplift within that group. If you have zero rewards or all the same rewards, there is no learning signal. If the model finds the same strategy across all of them, the rollout is useless; if the task is too hard, the model cannot reach a signal. The hard part is creating an environment where enough attempts can produce contrast.
“The gradients don’t flow, and you can’t learn anything.”
If the task begins beyond the model’s reach, supervised examples may be needed just to produce a nonzero outcome. That is different from what interests me about test-time training. I do not want an average kernel-writing agent; I want the run that produces the fastest kernel, or an exceptional searcher program.
That search is disheartening to watch: long stretches look flat, then one branch finds something exciting. For this task, the exceptional outcome matters more than smooth improvement in average performance across the run.
One zero cannot explain a long failed trace
I tried to run reinforcement learning around a Claude tool harness, and I do not recommend it. If you have a very long trace and one part goes wrong, the program gets zero. It is not clear what went wrong: was it one tool call, a mistake in the generated code, or something later in the run? The whole costly trace collapses into a result that gives the learner almost no guidance about what to change.
An intermediate-improvement signal would be more useful. It could reward progress while the harness is still working, so a useful attempt is not erased by a later error. That is the direction I prefer to one terminal zero at the end of a long trace.
PUCT gives discarded programs a second life
PUCT attacks a different kind of waste. You start with a seed program, record all the programs the model writes, and let the search go back and revisit previous branches. One branch may not have received the highest score, but it may still be worth spending more time optimizing. Instead of discarding every lower-ranked program after its first result, the tree preserves paths that another round of search can improve.
The missing piece is guidance. AlphaZero has a learned head that informs its PUCT search; these program-search methods do not. I want to revisit the signal that tells this tree where another unit of compute is most likely to pay off.
The first result was a $700 storage receipt
The experiment ended before it found a profitable strategy. The run produced enough data to kill my solid-state drive. The result is plain: the run never became profitable. I spent roughly a week on the experiment, lost about $700 on the drive, and decided not to buy another one.
“I did not make any money. I lost seven hundred dollars for the solid state drive.”
That was enough to stop the first run. I moved on because I did not want to replace the drive and repeat the same failure. Before returning to the experiment, I also want to revisit the missing guidance signal that should help PUCT decide which saved program branches deserve more work.
Question from the room
LLM agents versus simpler quantitative models
I know LLMs are being used to ingest large amounts of data, especially natural-language data, and inform other parts of trading strategies. I am not very familiar with quant finance, but I expect LLMs to be one piece of a larger puzzle; finding what works will require experimentation.
Glossary
test-time training (TTT) — Training performed while solving a target problem, rather than only before deployment. TTT-Discover combines learning and adaptive search to prioritize promising candidate solutions. The model therefore adapts during the specific task.[4]
Group Relative Policy Optimization (GRPO) — A reinforcement-learning method introduced as a PPO variant. It compares rewards within groups of sampled outputs so relative performance can supply an optimization signal without requiring a learned value model for every comparison.[5]
archive node — An Ethereum node retaining historical state beyond the latest chain tip. That state supports queries and re-execution at earlier blocks, at substantial storage cost. The history enables deterministic replay and forensic analysis.[6]
EVM trace — A detailed view of transaction execution produced by replaying EVM operations. Tracing exposes internal calls and state transitions and can require accessible historical state for debugging, analysis, and simulation.[7]