Latency-optimized LLMs for trading_

I came across an article about an academic study showing that you could have used GPT-3 to build a long-short strategy that stayed profitable for several years. So yes, alpha from LLMs has definitely been done. But if you use it naively now, the edge is squeezed out. I think one reason you cannot naively use GPT to get alpha is the latency that it takes. Once traders share the same signal, getting the classification first becomes part of the strategy.

The strategy looked at whether sentiment was positive, neutral, or negative. Positive meant long, neutral meant no change, and negative meant short. GPT-3's cutoff preceded the backtest news, and the reported Sharpe ratio was three.

Backtesting always has a problem with fees and cross-impact, so I do not treat that result as a ready-made trading system. More important, later researchers duplicated the study with GPT-4. In Q1 2024, a long-short sentiment model was right sixty-two percent of the time. By Q4 2025, all that alpha had been squeezed out—the edge decayed over time—with the implication that large traders were already using the technique in production. Unless your system is highly optimized, you will not be able to capture it.

“By Q4 2025, all that alpha had been squeezed out. So the strategy stopped working over time.”

This got me thinking about my PhD work on efficient models and distillation. In those experiments I took large, medium, and small convolutional policy networks and trained them from scratch on Atari game environments. The larger the model, the faster it would learn and the better it would do in general. The red large-policy network almost always learned faster and reached a higher score than the medium and small networks. The chart's summary is direct: model size accelerated learning.

“Large models learn faster. They reach a higher plateau than smaller models.”

By default, a small model learned more slowly and reached a lower plateau. It was cheaper to run, but weaker when learning the policy from scratch.

The technique separates learning from execution. Start with a large policy, let it interact with the environment, and get it to high performance. Then have it interact again while you record the states and actions—or the logits from the teacher network—to a database. That database of observations lets smaller networks learn without doing the original exploration. The large model does the hard learning; the compact model inherits its behavior for faster deployment.

After the large policy becomes high performance, I let it interact with the environment and record the states and actions, or the logits that come out of the teacher network. Those observations go into a big database. You can then treat it like a supervised training problem and use that information with smaller networks. The teacher has already learned from the environment; its recorded outputs become the examples the student needs.

The student learns from the teacher, not through its own exploration. It treats the recorded observations and outputs as a supervised training problem.

It works really well. The table compares the final teacher scores with the student scores. Those students never actually got to experience the environment during training; they just learned from the teacher, with no fine-tuning after distillation. When they later interacted with the environment, their scores were about the same or better. That is evidence for this particular experiment, not a claim that every teacher and student pair will transfer equally well.

To apply this technique with LLMs, get tons of news and start building a corpus of news for the assets that you are interested in trading.

Then use an open model to do sentiment analysis on all of it. You can create embedding vectors for the model's output, pair them with the news as input, and create a supervised training corpus. With that corpus, start experimenting with smaller models. The teacher's output supplies the labels while the expensive model is off the live execution path. This is the same pattern as recording the states and actions of a reinforcement-learning teacher, only the observations are news stories and the targets are language-model judgments.

You want smaller models because you want hardware-optimized models for latency. Distillation supplies the behavior to imitate; it does not choose which operations run fastest. The next question is how to design an architecture with hardware constraints, including response time, built into the search.

The other technique asks how to do architecture search with hardware constraints in mind. Latency is a hardware constraint. In the toy example, softmax weights determine how much each candidate operation counts, and you assign a cost to each hardware operation. In a transformer, that could include how many layers you have; on the target machine, it could be measured inference time. Now the search can evaluate things other than error: it can see the cost of the path used to produce an answer.

Take that cost and combine it with the loss function. The penalty becomes a controllable knob: turn down the hardware cost and you tend to get larger neural networks; turn it up and you get smaller networks.

The dashed lines are the performance cutoff—the minimum acceptable performance. They keep the search from shrinking a network until it becomes useless. For an LLM used in trading, you could track how long it took to get the output, then find the sweet spot between that time and model quality. Distillation gives you the teacher's behavior; architecture search gives you a hardware-aware student. The practical target is not the smallest possible network, but the one that stays above the cutoff while answering fast enough to compete for the signal.

The cited sentiment paper identifies its classifier more precisely as OPT, described by the authors as GPT-3-based, and reports 74.4% classification accuracy plus a long-short Sharpe ratio of 3.05. The talk rounds this to “GPT-3” and a Sharpe ratio of three. This correction identifies the cited experiment; it does not validate the separate Q1 2024-to-Q4 2025 alpha-decay series or turn the backtest into a production recipe.[4]

Slide 12 contains GBNAS pseudocode that alternates model-weight updates with architecture updates under a resource-aware objective. Sam explains operation costs and the penalty knob in the surrounding talk but does not verbally walk through this algorithm page. The slide names that alternating schedule explicitly. It is retained as slide-sourced implementation context, not as a separately explained method or a claimed production system.