Oriental Nippon Development
Back to portfolio
Case studyGerman enterprise client

Agentic AI on a Live CRM: An Evidence-Based Research Programme

Agentic AI claims were untested on real operational data, and hosted models were ruled out. Our R&D lab connected locally hosted models to the CRM through MCP tool servers and built a harness to compare architectures objectively.

Industry
Enterprise IT & Cloud
Region
Germany
Engagement
R&D Lab (joint innovation)

Outcomes

  • MCP-based CRM agent prototype
  • Reusable evaluation harness
  • Decision record incl. negative results

Stack

Odoo CRM · MCP · Local LLMs · LangChain · RAG · Python

Written byOriental Nippon Team

This was a research engagement, not a production build. A German enterprise client wanted to know whether everyday CRM work, such as finding, updating and acting on customer records, could be driven by a prompt instead of a form. And if so, which of the competing architectures was actually worth taking further.

Most claims about agentic AI had never been tested against real operational data. An informal comparison would produce an opinion. The client wanted a finding.

The question

  • Untested claims. Vendor demos had not been tried on the client's own records.
  • Opinions, not evidence. Informal comparisons do not survive a budget review.
  • Local models only. Data sensitivity ruled out hosted models, which narrowed the options.
  • Fail in the lab, not in production. Being wrong in a lab is cheap. Being wrong in production is not.
HypothesisWhat we compared
GroundingRetrieval improves accuracy on CRM tasksRAG vs no retrieval
OrchestrationPipeline design matters as much as the modelSeveral pipeline variants
ModelsA local model can be good enoughA range of locally hosted LLMs
StorageThe vector store choice affects resultsSeveral vector stores

What we built

  • A CRM-to-LLM integration

    Odoo CRM connected to locally hosted models through MCP tool servers, with explicit permissions on every action.

  • Retrieval over the CRM's own data

    Answers grounded in real records, not generated from nothing.

  • A comparison harness

    The same task set run across every architecture, model and store, so results are directly comparable.

  • A decision record

    What held, what didn't, and why.

Technology and the role it played

TechnologyRole in this project
Odoo CRMThe operational system the agents act on
MCP tool serversA permissioned tool layer between models and the CRM
Local LLMsModels hosted inside the client environment
LangChainOrchestration for the pipeline variants
RAG vector storesRetrieval over the CRM's own records

How we ran it

Our R&D lab agreed the success threshold before building anything: what "good enough to put in production" would mean, fixed in advance so the result could not be argued afterwards.

  1. Frame the hypothesis. Success threshold and time-box agreed up front.
  2. Build the integration. Against a masked copy of real data in an isolated environment.
  3. Define the task set. The operations an agent must perform, with the correct answer for each.
  4. Run the comparisons. Across architectures, models and stores.
  5. Write up the decision. Including the negative results.
Delivery timelineIndicative total: 11–17 weeks
  1. 01Framing

    1–2 wks

    Hypothesis, success threshold, time-box

  2. 02Integration layer

    3–4 wks

    CRM tool server, permissions, masked data environment

  3. 03Task set and harness

    2–3 wks

    Evaluation tasks, expected answers, measurement

  4. 04Comparison runs

    4–6 wks

    Retrieval vs non-retrieval, pipeline, model and store variants

  5. 05Write-up and decision

    1–2 wks

    Findings, recommendation, what to productionise

Risks and how we handled them

RiskHow we handled it
Results argued after the factSuccess threshold and time-box agreed before building
Real customer data exposedA masked copy of real data in an isolated environment
Agents taking unintended actionsExplicit permissions on every tool the model can call
Only reporting what workedThe write-up includes negative results

Outcome

The programme replaced assertion with structured evidence: a documented comparison across retrieval strategies, pipeline designs, local models and vector stores, on real operational tasks.

Not every variant was worth keeping, and that was the point. The value of a lab is cheap failure. A research programme that finds everything promising usually has not been run properly.

What the client keeps

  • The prototype and its source
  • The evaluation task set and harness
  • Results for every configuration tested
  • A written decision record

Sources and further reading

  1. Model Context Protocol, modelcontextprotocol.io
  2. LangChain introduction, LangChain
  3. Top 10 for Large Language Model Applications, OWASP
  4. AI Risk Management Framework, NIST

Facing something similar? This is the kind of work our AI, Data Science & Automation team delivers every day. Talk to us about your stack.