dstack Private Inference Gateway and Attestation Interoperability_

Private inference asks two questions at once: can a service keep prompts and outputs private, and can it prove that it did?

The proof can also answer which model served the request: someone paying for one model should not silently receive another. Without evidence, a provider may promise that it will never inspect your data, but that remains a promise. A backend engineer with access to the data and hardware can technically inspect an ordinary deployment. Organizational policy matters; it is different from isolating the data from the operator.

Running a model on-premise removes the cloud exposure, but replaces it with the cost and delay of owning scarce GPUs.

A conventional cloud provider restores capacity and convenience, but then you essentially hand it the data. Confidential computing tries to combine protected hardware with rented cloud compute instead. The user encrypts the request and sends ciphertext to a trusted execution environment; the TEE decrypts it, runs the model inside the protected boundary, and returns the result through the protected path. The user gets remote compute without owning the machine.

I describe the TEE as a black box that keeps the working data from the hardware administrator through encryption and isolation. That description has assumptions: I explicitly set aside extreme cases where an attacker compromises the hardware itself. The claim is protected execution under a threat model, not invulnerability.

The appeal is obvious: rent confidential compute by the token without owning the GPU. The biggest obstacle is not the idea but capacity.

Confidential-computing providers may have a few protected GPUs, not thousands or tens of thousands. An AI agent makes that shortage visible immediately: it can spin up several sub-agents and send parallel requests, saturating a small pool all at once. Capacity that feels adequate for one human conversation can fail as soon as several branches arrive together, even when the privacy mechanism itself works as intended.

When that pool fills, users retry. A capacity error feels like a slow or unreliable TEE service even when the protected hardware is not the bottleneck.

That misdiagnosis drives the loop backward. Failed requests create a bad experience; a bad experience loses users; fewer users mean less revenue to buy more GPUs. No single provider wants to finance a huge confidential fleet before demand exists, yet demand cannot mature while requests regularly fail. The result is a supply problem that looks, from the outside, like a performance problem.

The desired flywheel reverses every arrow: more capacity improves the experience, which attracts users and revenue, which funds more GPUs. The bootstrap question is how to assemble enough supply without asking one operator to own the entire fleet first.

That is the chicken-and-egg problem: how do we assemble capacity before there are enough users to pay for it? My answer begins with aggregation.

Ordinary model routers already combine providers. OpenRouter is the familiar pattern: it finds an upstream with capacity and an acceptable result, then routes the request there. The user benefits from the group's inventory instead of waiting on one operator's machines. Confidential inference needs the same pooling effect because several small providers can collectively absorb traffic that would overwhelm each of them alone.

But the aggregator must be confidential-computing native. Pooling supply cannot solve fragmentation by terminating privacy at an ordinary load balancer.

The user should not have to manage which provider happens to serve each request, yet the request still has to arrive on the selected GPU privately and securely. Free capacity is not enough; the upstream must preserve the protected path. If it does, bursty traffic can move toward the GPUs that are ready while several small providers behave like one larger confidential pool.

Aggregation breaks the supply deadlock only if it does not recreate the trust problem in the middle. The gateway is part of the security model, not plumbing outside it.

A load balancer cannot find an arbitrary provider and send your request out, because in confidential computing, serving must come with proof.

“We don’t just serve, we also prove.”

The problem is that we need to make sure from the user to the aggregator to the GPU, this whole process is secure and private. The GPU can prove it is private, but it is not enough. We still want to make sure the router itself can prove the entire link is private. Proofs from Phala, NEAR, or another provider combine to show that your prompt will never get out of trusted execution environments.

The Private AI Gateway is the project proposed for that job. Its backend tracks the available private-inference providers, whether an upstream is a GPU or another service.

The gateway routes only when that upstream can prove it is secure and private enough. On the client side, the user still talks to one endpoint but wants evidence for the whole system. We combine the upstream proofs with proof for the gateway itself, so provider selection is not an invisible capacity check. The selected route and the evidence returned to the client have to agree.

The deck marks those protected links in purple. Each is end-to-end encrypted, and the client can verify what is running on the server side of the channel carrying the prompt and response.

The architecture need not stop at one gateway. A narrow load balancer might route among several GPU TEEs, while another gateway treats it as one upstream beside direct providers.

The same verify-before-routing relationship can repeat at each layer. This is where each gateway becomes a composable building block: every gateway verifies the next layer, preserves the protected connection, and contributes proof back toward the client. Composition is useful because it avoids asking the client to inspect every machine directly, without letting an intermediate router disappear from the evidence chain.

“From the user side, you don’t need to verify everything. You just trust your counterparty.”

The user connects to one gateway, verifies it, and relies on that gateway to verify every upstream in the chain.

That recursion combines a network of TEEs and, in my description, lets the network extend almost without limit. It also keeps the edge interface small: a client connects in one place while each gateway verifies its direct upstream and passes the combined proof back toward the client.

That common edge matters because NEAR AI, Tinfoil, Phala, and other providers each prove their systems differently. The gateway offers one API while still understanding and checking the evidence underneath it.

At the event, I described this as already operating through RedPill: with its TEE option enabled, protected models could come from our infrastructure or another provider.

That is an event-time deployment statement, not a timeless claim about every later release or route. The demonstration then opened the Private AI Gateway implementation in the dstack organization, making the architecture inspectable instead of leaving the gateway, route selection, and proof chain as boxes on a slide.

“It’s fully open source, and you can run that in any supported TEE environment.”

The accompanying specification invites feedback on a common interface: trust model, workload identity, presentation of attestation evidence, and protocol-level end-to-end encryption.

For every provider, we have a table to make sure its implementation actually meets our standard before it can be included. The checklist verifies what is running in the confidential computing environment, how we make sure there is no backdoor, how we make sure it is end-to-end encrypted, and whether TLS terminates within the TEE. A new provider can simply make a pull request, and then we can include it in the gateway.

Once included, an upstream can serve RedPill or another gateway user. The demo showed multiple providers for one model and a monitor around those routes—closing the loop from reviewed proof to pooled capacity.