Oriental Nippon Development
Back to portfolio
Case studyUS healthcare debt-recovery operation

Predictive Account Prioritisation: From ~62% to ~87% Accuracy

Thousands of accounts, finite agent hours, and a bought-in credit signal that was right about six times in ten. Our ensemble model scores every account daily and retrains weekly on real outcomes.

Industry
Revenue Cycle & Finance
Region
United States
Engagement
Lab-type ODC (dedicated team)

Outcomes

  • ~87% vs ~62% accuracy
  • Daily scoring of every account
  • Weekly retraining on outcomes

Stack

Python · XGBoost · LightGBM · CatBoost · Random Forest · SQL · Scheduled pipeline

Written byOriental Nippon Team

A debt-recovery operation working on behalf of US healthcare providers had thousands of open accounts and a fixed number of agent hours. Each morning someone had to decide which accounts the agents would call first. That decision came from a credit-rating score bought from a third party, built for a general population, and it had never been checked against this operation's own results.

When we measured it, the score picked paying accounts about six times in ten. Nobody had measured it before.

Where the hours were going

  • Wrong signal. The external rating knew nothing about this operation's history, its clients or the way its accounts actually resolve.
  • Wasted calls. Every call to an unlikely payer was also a call not made to a better account.
  • No learning. Today's payments had no effect on tomorrow's list.
  • No baseline. Accuracy had never been tracked, so there was no way to tell whether any change helped.
BeforeAfter
Prioritisation signalBought-in credit ratingModel trained on the operation's own outcomes
Accuracy at finding payers~62%, never tracked~87%, tracked over time
Scoring frequencyStaticEvery account, every night
Learning from resultsNoneWeekly retraining

What we built

  • An ensemble model

    Several gradient-boosting and tree-based models (XGBoost, LightGBM, CatBoost, Random Forest) combined to give each account a probability of payment.

  • A nightly scoring pipeline

    Every active account is scored against current data, monitored, and the score is written into the tool agents already work in.

  • A weekly retraining loop

    The week's real outcomes go back into training, so the ordering keeps improving.

  • An accuracy record

    The baseline and every model since are measured the same way on held-out data.

The ensemble is only half the system. The loop around it is what keeps the ordering improving:

Technology and the role it played

TechnologyRole in this project
Python and SQLFeature engineering from the operation's own account history
XGBoost, LightGBM, CatBoostGradient-boosting members of the ensemble
Random ForestA tree-based model that adds diversity to the ensemble
Scheduled pipelineNightly scoring and weekly retraining, with monitoring

How we delivered it

Our dedicated ODC team started with the number nobody had: how good the existing approach actually was. Without that, a better model is only a claim.

  1. Baseline first. We measured the credit-rating approach and agreed the evaluation method with the client before building anything.
  2. Features from their own history. Signals were engineered with the operation's data team, from how its own accounts had behaved.
  3. Honest comparison. Candidate models were compared on a held-out period the models never saw in training.
  4. Into the workflow. Daily scoring runs on a schedule, is monitored, and lands where agents already look.
  5. Close the loop. Weekly retraining on outcomes, with accuracy tracked over time.
Delivery timelineIndicative total: 13–16 weeks
  1. 01Baseline measurement

    2 wks

    Accuracy of the existing approach, evaluation method agreed

  2. 02Feature engineering

    3–4 wks

    Signals built from the operation's own history

  3. 03Model development

    4–5 wks

    Ensemble build and held-out evaluation

  4. 04Scoring pipeline

    2–3 wks

    Nightly scoring, monitoring, delivery into the agent workflow

  5. 05Feedback loop

    2 wks

    Weekly retraining and accuracy tracking

Risks and how we handled them

RiskHow we handled it
A better model with nothing to compare it toThe existing credit-rating approach was measured first and became the baseline
Over-optimistic accuracy from leaked dataModels were judged only on a held-out period they never saw in training
Model drift as the account mix changesWeekly retraining on real outcomes, with accuracy tracked over time
Scores nobody usesScores are written into the tool agents already work in, not a separate report

Outcome

Over the five months after deployment, the model identified paying accounts with around 87% accuracy, against roughly 62% for the credit-rating approach it replaced.

In practice, agents now work a better-ordered list every morning, and the order gets better as the feedback loop runs.

That figure covers that period and that account population. It is a measured comparison, not a projection, and we would not extrapolate it forward.

What the client keeps

  • Model code and the training pipeline, in their own repository
  • Feature definitions
  • Held-out evaluation results and the original baseline measurement
  • Retraining configuration

Sources and further reading

  1. XGBoost documentation, XGBoost
  2. LightGBM documentation, Microsoft
  3. Ensemble methods, scikit-learn
  4. Cross-validation: evaluating estimator performance, scikit-learn

Facing something similar? This is the kind of work our AI, Data Science & Automation team does every day. Talk to us about your data.