Predictive Account Prioritisation: From ~62% to ~87% Accuracy
Thousands of accounts, finite agent hours, and a bought-in credit signal that was right about six times in ten. Our ensemble model scores every account daily and retrains weekly on real outcomes.
- Service
- AI Business Applications
- Industry
- Revenue Cycle & Finance
- Region
- United States
- Engagement
- Lab-type ODC (dedicated team)
Outcomes
- ~87% vs ~62% accuracy
- Daily scoring of every account
- Weekly retraining on outcomes
Stack
Python · XGBoost · LightGBM · CatBoost · Random Forest · SQL · Scheduled pipeline
Written byOriental Nippon Team
A debt-recovery operation working on behalf of US healthcare providers had thousands of open accounts and a fixed number of agent hours. Each morning someone had to decide which accounts the agents would call first. That decision came from a credit-rating score bought from a third party, built for a general population, and it had never been checked against this operation's own results.
When we measured it, the score picked paying accounts about six times in ten. Nobody had measured it before.
Where the hours were going
- Wrong signal. The external rating knew nothing about this operation's history, its clients or the way its accounts actually resolve.
- Wasted calls. Every call to an unlikely payer was also a call not made to a better account.
- No learning. Today's payments had no effect on tomorrow's list.
- No baseline. Accuracy had never been tracked, so there was no way to tell whether any change helped.
| Before | After | |
|---|---|---|
| Prioritisation signal | Bought-in credit rating | Model trained on the operation's own outcomes |
| Accuracy at finding payers | ~62%, never tracked | ~87%, tracked over time |
| Scoring frequency | Static | Every account, every night |
| Learning from results | None | Weekly retraining |
What we built
An ensemble model
Several gradient-boosting and tree-based models (XGBoost, LightGBM, CatBoost, Random Forest) combined to give each account a probability of payment.
A nightly scoring pipeline
Every active account is scored against current data, monitored, and the score is written into the tool agents already work in.
A weekly retraining loop
The week's real outcomes go back into training, so the ordering keeps improving.
An accuracy record
The baseline and every model since are measured the same way on held-out data.
The ensemble is only half the system. The loop around it is what keeps the ordering improving:
Technology and the role it played
| Technology | Role in this project |
|---|---|
| Python and SQL | Feature engineering from the operation's own account history |
| XGBoost, LightGBM, CatBoost | Gradient-boosting members of the ensemble |
| Random Forest | A tree-based model that adds diversity to the ensemble |
| Scheduled pipeline | Nightly scoring and weekly retraining, with monitoring |
How we delivered it
Our dedicated ODC team started with the number nobody had: how good the existing approach actually was. Without that, a better model is only a claim.
- Baseline first. We measured the credit-rating approach and agreed the evaluation method with the client before building anything.
- Features from their own history. Signals were engineered with the operation's data team, from how its own accounts had behaved.
- Honest comparison. Candidate models were compared on a held-out period the models never saw in training.
- Into the workflow. Daily scoring runs on a schedule, is monitored, and lands where agents already look.
- Close the loop. Weekly retraining on outcomes, with accuracy tracked over time.
01Baseline measurement
2 wks
Accuracy of the existing approach, evaluation method agreed
02Feature engineering
3–4 wks
Signals built from the operation's own history
03Model development
4–5 wks
Ensemble build and held-out evaluation
04Scoring pipeline
2–3 wks
Nightly scoring, monitoring, delivery into the agent workflow
05Feedback loop
2 wks
Weekly retraining and accuracy tracking
Risks and how we handled them
| Risk | How we handled it |
|---|---|
| A better model with nothing to compare it to | The existing credit-rating approach was measured first and became the baseline |
| Over-optimistic accuracy from leaked data | Models were judged only on a held-out period they never saw in training |
| Model drift as the account mix changes | Weekly retraining on real outcomes, with accuracy tracked over time |
| Scores nobody uses | Scores are written into the tool agents already work in, not a separate report |
Outcome
Over the five months after deployment, the model identified paying accounts with around 87% accuracy, against roughly 62% for the credit-rating approach it replaced.
In practice, agents now work a better-ordered list every morning, and the order gets better as the feedback loop runs.
That figure covers that period and that account population. It is a measured comparison, not a projection, and we would not extrapolate it forward.
What the client keeps
- Model code and the training pipeline, in their own repository
- Feature definitions
- Held-out evaluation results and the original baseline measurement
- Retraining configuration
Sources and further reading
- XGBoost documentation, XGBoost
- LightGBM documentation, Microsoft
- Ensemble methods, scikit-learn
- Cross-validation: evaluating estimator performance, scikit-learn
Facing something similar? This is the kind of work our AI, Data Science & Automation team does every day. Talk to us about your data.
