Oriental Nippon Development
Back to blog
Technology

On-Premise LLMs for Regulated Data: When the Model Has to Come to the Data

Healthcare, finance and HR data often can't leave the building. Here's how we put capable language models inside the client's network, and what actually decides whether it works.

Half the AI projects we are asked about start with the same constraint: the data cannot leave the network. Patient records, account-level financial data, HR files. For those clients, a hosted model is not a cost question. It is out of scope.

That used to mean "no AI." It doesn't any more. We have run language models entirely on client hardware, including a 30-billion-parameter model answering plain-language questions over a revenue-cycle database, and an on-premise lead system that reads business cards and transcribes conversations.

Hosted or on-premise?

Hosted model APIOn-premise model
Data leaves the networkYesNo
Upfront hardwareNoneServer sized for the model
Model choiceLargest frontier modelsOpen-weight models that fit your hardware
Compliance reviewData-processing agreements, residencyMostly internal
Typical fitPublic or low-sensitivity dataHealth, financial, HR data

Neither is better in general. The data decides.

For regulated data, the model has to come to the data, inside the client’s own network.

Settle the boundary before choosing the model

The first conversation is with compliance, not with a model leaderboard. What may cross which boundary? Who may ask what? Is voice input allowed? Only once that is written down do we size the model and the hardware. Picking the model first and negotiating the boundary later is how projects stall in month three.

The architecture we use

Grounding beats model size

A model that doesn't know your schema will invent columns. What made our natural-language data project useful was schema grounding, iterated against the real questions staff had previously sent to the data team. Those old tickets are the best test set you will ever get.

Constrain the blast radius

  • Read-only paths for anything that queries operational data.
  • The asking user's permissions, not a service account's.
  • Every answer checkable: show the query or cite the record.
  • Logs stay local too, because prompts often contain sensitive details.

A realistic timeline

From our natural-language data access project:

Delivery timelineIndicative total: 10–14 weeks
  1. 01Boundary and model selection

    1–2 wks

    Residency constraints, model choice, hardware sizing

  2. 02Infrastructure

    2 wks

    On-premise deployment, serving, monitoring

  3. 03Schema grounding

    3–4 wks

    Structure mapping, query generation, evaluation set

  4. 04Interface and permissions

    2–3 wks

    Voice and text input, read-only scoping

  5. 05Pilot

    2–3 wks

    Non-technical users, answers compared against analysts

Grounding is the longest single phase. Infrastructure, often the part people worry about, is one of the shortest.

Measure against the people it replaces

Our pilots compare the model's answers with what an analyst would have produced, using non-technical users. If it cannot match that, it does not ship.

What you get

Business users get answers in seconds, without SQL or a queue, and nothing sensitive leaves the building. See the full story in our case study on natural-language data access on-premise.

Sources and further reading

  1. Top 10 for Large Language Model Applications, OWASP
  2. AI Risk Management Framework, NIST
  3. vLLM documentation, vLLM project
  4. llama.cpp, ggml-org on GitHub
  5. BIRD: a big benchmark for text-to-SQL, BIRD-bench
  6. Spider: a cross-domain text-to-SQL benchmark, Yale LILY Lab

Related case study

Ask Your Data in Plain Language, a 30B LLM Running Entirely On-Premise