Oriental Nippon Development
Back to portfolio
Case studyUS revenue-cycle operator

Proactive Monitoring for Hundreds of Daily Secure File Transfers

Missing or late files were discovered only when downstream data broke or a client escalated. We wrote down what should arrive, built a watcher against it, and gave the operations team alerts, history and runbooks.

Industry
Revenue Cycle & Finance
Region
United States
Engagement
SES (specialist engineers)

Outcomes

  • Missing files caught before clients notice
  • Expectation register for every source
  • Runbook per alert type

Stack

Python · SFTP monitoring · Web dashboard · Alerting

Written byOriental Nippon Team

A US revenue-cycle operator exchanges hundreds of files a day with client facilities over secure file transfer. Those files carry the data the rest of the operation runs on.

When one went missing, nothing noticed. The gap showed up days later as bad downstream data, and then as a client escalation. At that volume, checking by hand was never realistic.

The problem

  • Silent failures. Missing or late files had nothing watching for them.
  • Found through consequences. A missing file appeared first as a data problem, then as a complaint.
  • No record. There was no history of what had arrived and what had not.
  • No written expectation. Nobody had a definitive list of which files should arrive, from whom, and when.
BeforeAfter
Files per dayHundreds, unwatchedHundreds, checked against a schedule
How a gap was foundClient escalationProactive alert
Expected-file listDid not existExpectation register
HistoryNoneDashboard with arrival history

What we built

  • A file watcher

    Monitors the transfer server and checks what arrived against what each schedule expects.

  • Missing-file alerts

    An absence becomes an event, raised the same day.

  • A monitoring portal

    Several dashboards showing arrival status, delays, patterns and history.

  • A response runbook

    What to do for each type of alert.

A typical day, from a facility sending its file to the right person hearing about a gap:

Technology and the role it played

TechnologyRole in this project
PythonThe watcher and its evaluation logic
SFTP monitoringReading arrivals on the transfer server
Web dashboardArrival status, delays, patterns and history
Scheduled alertingNotifications routed to the owner of each file

How we delivered it

Our NOC engineers started the watcher in observe-only mode and tuned it on real traffic. An alerting system that cries wolf gets muted within a fortnight.

  1. Write down the expectation. What should arrive, from whom, how often. This had never existed.
  2. Build the watcher. Running against that expectation, observing only.
  3. Tune on real traffic. Thresholds adjusted until false positives were rare.
  4. Dashboard and alerting. Status, history and notification routing.
  5. Runbook. A response procedure for each alert type.
Delivery timelineIndicative total: 8–11 weeks
  1. 01Expectation mapping

    1–2 wks

    Expected files, sources, cadences, owners

  2. 02Watcher build

    2–3 wks

    Evaluation logic, observe-only run

  3. 03Threshold tuning

    2 wks

    Real traffic, false-positive reduction

  4. 04Dashboard and alerting

    2–3 wks

    Status portal, history, notification routing

  5. 05Runbook and hand-over

    1 wk

    Response procedure per alert type

Risks and how we handled them

RiskHow we handled it
Alerts that cry wolfObserve-only mode first, then threshold tuning on real traffic
No definition of "expected"An expectation register of sources, files, cadences and owners
Alerts with no response planA runbook for each alert type at hand-over

Outcome

The operations team is told about a missing file instead of discovering it through its consequences.

The expectation register turned out to be as useful as the monitoring. A good deal of the risk came from nobody having a definitive list of what was supposed to arrive.

What the client keeps

  • Watcher and dashboard source, in their own repository
  • The expectation register
  • Alert thresholds and routing configuration
  • The response runbook

Sources and further reading

  1. What is SFTP?, AWS
  2. Monitoring distributed systems, Google SRE book
  3. Alerting on SLOs, Google SRE workbook

Facing something similar? This is the kind of work our NOC Services team delivers every day. Talk to us about your stack.