Memory is a privacy boundary
Private context matters because an unencrypted database exposes far more than one search term. It can reveal the query, stored memory, preferences, decisions, and all the institutional context you have. Agent trajectories add a record of how work was attempted. Together, they show what the agent has learned and how its user works. My philosophy is that durable context belongs to the user. The durable context should remain under the user's control even when the model is remote. Renting a model or service can make sense; handing it readable memory gives away the durable part of the relationship.
“You should own the context and rent the intelligence.”
Context persists across calls and becomes more valuable as it accumulates. Making it private is really the only way to efficiently own the context in this design. Memory privacy is therefore an ownership boundary, not a setting added after the database already sees every retrieval. The decision compounds with each new trajectory retained.
From a server view, other vector databases can see your queries, trajectories, memory, and any thoughts stored inside. XTrace's proposed distinction is that its view is all ciphertext. We began by looking at private-retrieval research such as TIPTOE and at systems based on fully homomorphic encryption. The practical question was how to compute semantic distance between two embeddings without revealing either one. Retrieval itself would otherwise expose both sides of the comparison. For a 1,024-entry vector, I described the naive path as requiring 1,024 FHE operations per vector, which is really expensive.
That operation count motivated a narrower representation rather than a universal cryptographic benchmark. The goal is stronger than encryption at rest or in transit: the query, stored memory, and distance calculation should remain protected from the service performing the search. The threat model includes that operator, not only a network observer.
Search can stay encrypted end to end
The key realization is that a floating-point embedding can be transformed to binary while preserving the distance needed by this search. When you're doing semantic search, the quantity that matters is distance. The design can therefore replace general computation over the original embedding with one ciphertext operation to calculate the difference between two vectors. Binary distance becomes the statistic used to rank nearby memories. The system trades encrypted computation for targeted retrieval. That binary representation permits additive homomorphic encryption, a partial form of homomorphic encryption.
I described it as almost like a thousand times faster than the FHE operation count in the earlier sketch. This is a talk-reported comparison, not an independent benchmark. On the client, the system applies AES and Paillier encryption before data reaches the server; during clarification, I said the binary vector itself is stored under Paillier encryption. The speed claim depends on this representation and operation count.
When you submit an encrypted query to the system, the server performs the additive homomorphic calculation and returns encrypted results. Only you can see it after local decryption. During clarification, I emphasized that the binary vector is stored under Paillier encryption and that one additive homomorphic operation computes the distance between two binary vectors. Neither the stored vector nor query appears in plaintext at the server. Plaintext arrives only after the result returns to the secret-key holder.
The same boundary supports selective sharing. A user can encrypt all the data, host it on the XTrace server, and share the key with a colleague or another agent. Those authorized clients can still do semantic search on ciphertext and decrypt the returned memory. The infrastructure supplies storage and computation, while readable access stays with the chosen key holders rather than the operator. This separates custody of ciphertext from authority to read it. Only the secret key holder will be able to access the data in the presented design.
I described that as a cryptographic guarantee that XTrace never sees the data, which remains the project's claim rather than an audit conclusion here. The architectural point is narrower: private storage can support useful retrieval and multi-agent access without making the underlying memory globally readable. Key distribution, rather than server visibility, grants access.
Agents need two kinds of memory
Encryption solves only half the problem. When you think about memory, it divides into two useful categories: semantic and procedural. Earlier systems emphasized semantic memory because the limited context window made facts and conversation retrieval the main challenge. Benchmarks such as LocoMo, LongMem, and EVON consequently evaluate user-and-agent chat rather than the complete trajectories produced by agents that act through tools. The distinction follows the move from conversational assistants to acting systems. Agents are doing a lot of tool calls, launching other agents, retrying failures, and producing long trajectories and rollouts.
“If you just directly use semantic search on procedural memory, it usually will not work.”
That's where procedural memory comes into play: it records how an agent accomplishes a task. A useful store needs both the facts an agent believes and the actions, triggers, corrections, and successful sequences through which it learned to operate. Failures and retries remain evidence about procedure even when the final answer is correct.
The semantic memory we build is based on what XTrace calls a belief-revision system. A resolving mechanism updates stored beliefs so that semantic memories remain mostly up to date instead of preserving every earlier statement as equally authoritative. Procedural memory has a different object: it distills agent-based trajectories, including tool calls, retries, and lessons learned during a run. Different memory objects require different update and retrieval rules. If you directly use semantic search on procedural memory, it usually will not work.
A request such as help me debug this or help me deploy this may not resemble the stored moment that matters. The useful cue is often the action the agent is about to take. Procedural episodes therefore need retrieval rules tied to execution context, not only semantic similarity between two pieces of text.
Procedural memory acts before tools
XTrace anchors the tool calls of the agent as trigger conditions. Calling a particular tool, editing certain files, or crossing another operational boundary can cause the memory layer to inject context and steer the agent before the tool call. If an agent is about to remove a database, for example, the trigger can surface a company rule while the decision can still change. Timing is the mechanism: the remembered instruction arrives before execution.
At the tool boundary, the system can stop the action and then inject the context inside the run. That makes procedural memory a guardrail rather than an explanation retrieved after the event. It can also share or distill lessons learned from failed retry loops so later runs avoid the same sequence.
The encrypted vector database is available through an API, and the encryption scheme is open source so users can verify it locally. A broader memory hub allows teammates to share their agent brain and context for a holistic view. I drew a product boundary, however: the consumer-facing hub is not encrypted in the same way, while the underlying vector database is encrypted. The two products should not inherit each other's privacy claims.
What does encrypted indexing cost?
On the server side, do you still have to do a linear scan, or does it really scale linearly with the size of the database?
Currently, you have to do the linear scan of the whole database. But because we're using binary embeddings, we can use an inverted file index. Essentially, we can have square-root-n search, so we don't have to search through everything. You can still have an encrypted index for it.
Appendix
The final deck frame points readers to XTrace; its official site describes shared, durable, searchable memory across agents, tools, sessions, and teammates.[2]
Glossary & further reading
homomorphic encryption — Homomorphic encryption permits specified computation on encrypted data without first revealing the plaintext. Fully homomorphic encryption supports arbitrary functions; the talk instead discusses an additive, partially homomorphic operation.[4]
Hamming distance — Hamming distance is the number of positions at which two equal-length binary strings differ. It can also be read as the number of bit changes required to turn one string into the other.[5]
vector search — Vector search maps material into numeric embeddings and retrieves items by a distance or similarity measure. Here, the proposed private database searches encrypted binary embeddings and returns results for local decryption.[3]