Proactive Monitoring for Hundreds of Daily Secure File Transfers
Missing or late files were discovered only when downstream data broke or a client escalated. We wrote down what should arrive, built a watcher against it, and gave the operations team alerts, history and runbooks.
- Service
- NOC Services
- Industry
- Revenue Cycle & Finance
- Region
- United States
- Engagement
- SES (specialist engineers)
Outcomes
- Missing files caught before clients notice
- Expectation register for every source
- Runbook per alert type
Stack
Python · SFTP monitoring · Web dashboard · Alerting
Written byOriental Nippon Team
A US revenue-cycle operator exchanges hundreds of files a day with client facilities over secure file transfer. Those files carry the data the rest of the operation runs on.
When one went missing, nothing noticed. The gap showed up days later as bad downstream data, and then as a client escalation. At that volume, checking by hand was never realistic.
The problem
- Silent failures. Missing or late files had nothing watching for them.
- Found through consequences. A missing file appeared first as a data problem, then as a complaint.
- No record. There was no history of what had arrived and what had not.
- No written expectation. Nobody had a definitive list of which files should arrive, from whom, and when.
| Before | After | |
|---|---|---|
| Files per day | Hundreds, unwatched | Hundreds, checked against a schedule |
| How a gap was found | Client escalation | Proactive alert |
| Expected-file list | Did not exist | Expectation register |
| History | None | Dashboard with arrival history |
What we built
A file watcher
Monitors the transfer server and checks what arrived against what each schedule expects.
Missing-file alerts
An absence becomes an event, raised the same day.
A monitoring portal
Several dashboards showing arrival status, delays, patterns and history.
A response runbook
What to do for each type of alert.
A typical day, from a facility sending its file to the right person hearing about a gap:
Technology and the role it played
| Technology | Role in this project |
|---|---|
| Python | The watcher and its evaluation logic |
| SFTP monitoring | Reading arrivals on the transfer server |
| Web dashboard | Arrival status, delays, patterns and history |
| Scheduled alerting | Notifications routed to the owner of each file |
How we delivered it
Our NOC engineers started the watcher in observe-only mode and tuned it on real traffic. An alerting system that cries wolf gets muted within a fortnight.
- Write down the expectation. What should arrive, from whom, how often. This had never existed.
- Build the watcher. Running against that expectation, observing only.
- Tune on real traffic. Thresholds adjusted until false positives were rare.
- Dashboard and alerting. Status, history and notification routing.
- Runbook. A response procedure for each alert type.
01Expectation mapping
1–2 wks
Expected files, sources, cadences, owners
02Watcher build
2–3 wks
Evaluation logic, observe-only run
03Threshold tuning
2 wks
Real traffic, false-positive reduction
04Dashboard and alerting
2–3 wks
Status portal, history, notification routing
05Runbook and hand-over
1 wk
Response procedure per alert type
Risks and how we handled them
| Risk | How we handled it |
|---|---|
| Alerts that cry wolf | Observe-only mode first, then threshold tuning on real traffic |
| No definition of "expected" | An expectation register of sources, files, cadences and owners |
| Alerts with no response plan | A runbook for each alert type at hand-over |
Outcome
The operations team is told about a missing file instead of discovering it through its consequences.
The expectation register turned out to be as useful as the monitoring. A good deal of the risk came from nobody having a definitive list of what was supposed to arrive.
What the client keeps
- Watcher and dashboard source, in their own repository
- The expectation register
- Alert thresholds and routing configuration
- The response runbook
Sources and further reading
- What is SFTP?, AWS
- Monitoring distributed systems, Google SRE book
- Alerting on SLOs, Google SRE workbook
Facing something similar? This is the kind of work our NOC Services team delivers every day. Talk to us about your stack.
