Agentic AI on a Live CRM: An Evidence-Based Research Programme
Agentic AI claims were untested on real operational data, and hosted models were ruled out. Our R&D lab connected locally hosted models to the CRM through MCP tool servers and built a harness to compare architectures objectively.
- Service
- AI Business Applications
- Industry
- Enterprise IT & Cloud
- Region
- Germany
- Engagement
- R&D Lab (joint innovation)
Outcomes
- MCP-based CRM agent prototype
- Reusable evaluation harness
- Decision record incl. negative results
Stack
Odoo CRM · MCP · Local LLMs · LangChain · RAG · Python
Written byOriental Nippon Team
This was a research engagement, not a production build. A German enterprise client wanted to know whether everyday CRM work, such as finding, updating and acting on customer records, could be driven by a prompt instead of a form. And if so, which of the competing architectures was actually worth taking further.
Most claims about agentic AI had never been tested against real operational data. An informal comparison would produce an opinion. The client wanted a finding.
The question
- Untested claims. Vendor demos had not been tried on the client's own records.
- Opinions, not evidence. Informal comparisons do not survive a budget review.
- Local models only. Data sensitivity ruled out hosted models, which narrowed the options.
- Fail in the lab, not in production. Being wrong in a lab is cheap. Being wrong in production is not.
| Hypothesis | What we compared | |
|---|---|---|
| Grounding | Retrieval improves accuracy on CRM tasks | RAG vs no retrieval |
| Orchestration | Pipeline design matters as much as the model | Several pipeline variants |
| Models | A local model can be good enough | A range of locally hosted LLMs |
| Storage | The vector store choice affects results | Several vector stores |
What we built
A CRM-to-LLM integration
Odoo CRM connected to locally hosted models through MCP tool servers, with explicit permissions on every action.
Retrieval over the CRM's own data
Answers grounded in real records, not generated from nothing.
A comparison harness
The same task set run across every architecture, model and store, so results are directly comparable.
A decision record
What held, what didn't, and why.
Technology and the role it played
| Technology | Role in this project |
|---|---|
| Odoo CRM | The operational system the agents act on |
| MCP tool servers | A permissioned tool layer between models and the CRM |
| Local LLMs | Models hosted inside the client environment |
| LangChain | Orchestration for the pipeline variants |
| RAG vector stores | Retrieval over the CRM's own records |
How we ran it
Our R&D lab agreed the success threshold before building anything: what "good enough to put in production" would mean, fixed in advance so the result could not be argued afterwards.
- Frame the hypothesis. Success threshold and time-box agreed up front.
- Build the integration. Against a masked copy of real data in an isolated environment.
- Define the task set. The operations an agent must perform, with the correct answer for each.
- Run the comparisons. Across architectures, models and stores.
- Write up the decision. Including the negative results.
01Framing
1–2 wks
Hypothesis, success threshold, time-box
02Integration layer
3–4 wks
CRM tool server, permissions, masked data environment
03Task set and harness
2–3 wks
Evaluation tasks, expected answers, measurement
04Comparison runs
4–6 wks
Retrieval vs non-retrieval, pipeline, model and store variants
05Write-up and decision
1–2 wks
Findings, recommendation, what to productionise
Risks and how we handled them
| Risk | How we handled it |
|---|---|
| Results argued after the fact | Success threshold and time-box agreed before building |
| Real customer data exposed | A masked copy of real data in an isolated environment |
| Agents taking unintended actions | Explicit permissions on every tool the model can call |
| Only reporting what worked | The write-up includes negative results |
Outcome
The programme replaced assertion with structured evidence: a documented comparison across retrieval strategies, pipeline designs, local models and vector stores, on real operational tasks.
Not every variant was worth keeping, and that was the point. The value of a lab is cheap failure. A research programme that finds everything promising usually has not been run properly.
What the client keeps
- The prototype and its source
- The evaluation task set and harness
- Results for every configuration tested
- A written decision record
Sources and further reading
- Model Context Protocol, modelcontextprotocol.io
- LangChain introduction, LangChain
- Top 10 for Large Language Model Applications, OWASP
- AI Risk Management Framework, NIST
Facing something similar? This is the kind of work our AI, Data Science & Automation team delivers every day. Talk to us about your stack.
