On-Premise LLMs for Regulated Data: When the Model Has to Come to the Data
Healthcare, finance and HR data often can't leave the building. Here's how we put capable language models inside the client's network, and what actually decides whether it works.
Written byMuhammad Abdullah Chaudhary
Half the AI projects we are asked about start with the same constraint: the data cannot leave the network. Patient records, account-level financial data, HR files. For those clients, a hosted model is not a cost question. It is out of scope.
That used to mean "no AI." It doesn't any more. We have run language models entirely on client hardware, including a 30-billion-parameter model answering plain-language questions over a revenue-cycle database, and an on-premise lead system that reads business cards and transcribes conversations.
Hosted or on-premise?
| Hosted model API | On-premise model | |
|---|---|---|
| Data leaves the network | Yes | No |
| Upfront hardware | None | Server sized for the model |
| Model choice | Largest frontier models | Open-weight models that fit your hardware |
| Compliance review | Data-processing agreements, residency | Mostly internal |
| Typical fit | Public or low-sensitivity data | Health, financial, HR data |
Neither is better in general. The data decides.
For regulated data, the model has to come to the data, inside the client’s own network.
Settle the boundary before choosing the model
The first conversation is with compliance, not with a model leaderboard. What may cross which boundary? Who may ask what? Is voice input allowed? Only once that is written down do we size the model and the hardware. Picking the model first and negotiating the boundary later is how projects stall in month three.
The architecture we use
Grounding beats model size
A model that doesn't know your schema will invent columns. What made our natural-language data project useful was schema grounding, iterated against the real questions staff had previously sent to the data team. Those old tickets are the best test set you will ever get.
Constrain the blast radius
- Read-only paths for anything that queries operational data.
- The asking user's permissions, not a service account's.
- Every answer checkable: show the query or cite the record.
- Logs stay local too, because prompts often contain sensitive details.
A realistic timeline
From our natural-language data access project:
01Boundary and model selection
1–2 wks
Residency constraints, model choice, hardware sizing
02Infrastructure
2 wks
On-premise deployment, serving, monitoring
03Schema grounding
3–4 wks
Structure mapping, query generation, evaluation set
04Interface and permissions
2–3 wks
Voice and text input, read-only scoping
05Pilot
2–3 wks
Non-technical users, answers compared against analysts
Grounding is the longest single phase. Infrastructure, often the part people worry about, is one of the shortest.
Measure against the people it replaces
Our pilots compare the model's answers with what an analyst would have produced, using non-technical users. If it cannot match that, it does not ship.
What you get
Business users get answers in seconds, without SQL or a queue, and nothing sensitive leaves the building. See the full story in our case study on natural-language data access on-premise.
Sources and further reading
- Top 10 for Large Language Model Applications, OWASP
- AI Risk Management Framework, NIST
- vLLM documentation, vLLM project
- llama.cpp, ggml-org on GitHub
- BIRD: a big benchmark for text-to-SQL, BIRD-bench
- Spider: a cross-domain text-to-SQL benchmark, Yale LILY Lab
Related case study
Ask Your Data in Plain Language, a 30B LLM Running Entirely On-Premise
