“Data never leaves the building” is one of the easiest things to put on a slide about private AI, and one of the hardest to actually deliver. Running the model on your own hardware is only the start. Prompts get logged, documents get turned into search indexes, tools send telemetry, models need updates, and backups go somewhere.
This post walks through what that promise really requires for a healthcare organization: where data goes in a typical private AI system, which of those paths are easy to miss, and how to check them. It is written by a team whose leadership has built healthcare platforms serving millions of users. It is not legal advice, and it does not replace your compliance team.
Short version: the model is the easy part. Logs, indexes, caches, backups, and outbound network traffic are where data actually escapes, so those are the places to design and to test.
Inference: the part everyone thinks about
Running the model on hardware you control, with no calls to outside AI APIs, is the foundation, and it is the part most vendors describe. Two details are worth getting right.
First, the model files had to come from somewhere. Downloading weights is an inbound transfer, so do it deliberately: from a source you chose, checked against the published checksums, with a record of what you installed. Second, make sure the application really calls the local model. A tool configured with a hosted fallback can send data out the first time the local model is slow.
Everything around the model
Most of the data in a private AI system is not in the model at all. It is in the things built around it:
- Prompts and responses. If they are logged, the logs contain patient or business data. Decide where they are stored, who can read them, and how long they are kept.
- Search indexes and embeddings. They are derived from your documents and can expose them, so treat an index as a copy of the documents: the same access control, the same backup rules, the same deletion.
- Caches, temporary files, and swap. Data passes through them even when nothing is ever saved on purpose.
- Backups. Know which of the above they include, and where they are stored.
Traffic you did not ask for
Many tools and frameworks send usage data or check for updates by default. In our own tooling, for example, ai-on-a-box disables Next.js telemetry in its web build and in its local stack, because a default that phones home is not something a private system should inherit. Defaults like that are the usual way data leaves a system that was designed not to let it.
The reliable answer is to block outbound traffic by default and allow only what you choose, including a deliberate window for software and model updates. Then verify it rather than assume it:
- Run a realistic workload with outbound traffic blocked at the firewall, and watch for failures and blocked-connection logs.
- Capture network traffic during a test run, and account for every destination you see.
- Repeat the check after every upgrade, since new versions bring new defaults.
Access control and audit trails
“Who can use it” and “who can see what it was asked” are separate questions, and the second is the one that matters when the history contains patient information. Give the system real access control, and keep an audit log that records who accessed what without copying patient content into the log itself.
HIPAA: what design can and cannot do
A private design supports the safeguards HIPAA is concerned with: access control, audit logging, encryption, and keeping data on hardware you control. It cannot make a system compliant on its own. Compliance depends on your policies, your agreements, and how the system is operated, and your compliance team makes that determination. We do not certify systems or guarantee compliance.
On business associate agreements: our engagements are designed so our team never accesses protected health information. Systems run entirely on your infrastructure, under your control.
A checklist before go-live
- Where does every copy of patient data live, including logs, indexes, caches, and backups?
- What is the system allowed to send outside, and what is the evidence that it sends nothing else?
- Who can use it, and who can read the history of what was asked?
- How long are prompts and logs kept, and how are they deleted?
- How do software and model updates arrive, and who approves them?
- What happens to access when someone changes roles or leaves?