A data pipeline that takes bursts of readings from medical devices, drops impossible or duplicate values, and keeps the latest good reading ready for the app to show.
Overview
What this is and why it matters, in plain English.
A data pipeline that takes bursts of readings from medical devices, drops impossible or duplicate values, and keeps the latest good reading ready for the app to show.
Device readings like heart rate and blood pressure, cleaned up and shown live in under 100ms.
The situation
What was going on before, and why it needed fixing.
Device vendors posted vitals in bursts. Impossible samples, like a heart rate of zero or a duplicated timestamp, landed in the chart next to real readings.
Device vendors posted bursts of vitals. Bad samples (a heart rate of 0, a duplicated timestamp) landed in the chart next to real ones.
What it needed to do
The outcomes the project had to deliver.
- Absorb bursts without dropping data.
- Reject impossible and duplicate samples.
- Serve the latest reading without scanning the raw stream.
What I did
My part of the work and who I worked with.
I designed the pipeline on AWS and wrote the Python validation Lambda, the Redis latest-value store, and the REST read API.
How it works
The main building blocks, from the screen people use to the data underneath.
Kinesis took the burst, a Python Lambda dropped impossible values, and Redis held the latest accepted reading for the query API.
- 01
AWS Kinesis
Takes bursts from devices and buffers them for processing.
- 02
Python Lambda
Applies validation rules, drops impossible values and duplicates, and stamps server time.
- 03
Redis
Holds the latest accepted reading per patient and signal.
- 04
REST API
Reads the latest values from Redis instead of the stream.
Key choices and trade-offs
Every build involves trade-offs. These are the important ones and what each one cost.
| Decision | Trade-off |
|---|---|
| Server timestamp wins. | Device time is ignored for ordering, but out-of-order clocks cannot corrupt the chart. |
| Latest value in Redis. | History is served elsewhere, but the common read is instant. |
Under the hood
Implementation details for technical readers, with a code sample.
- Latest reads under 100ms.
- Rule-based sample rejection in Python.
- Kinesis absorbing ingest bursts.
def accept(sample: dict) -> bool:
hr = sample.get('heart_rate')
if hr is not None and not 30 <= hr <= 220:
return False
spo2 = sample.get('spo2')
if spo2 is not None and not 50 <= spo2 <= 100:
return False
return TrueProblems solved
What got in the way, and how each problem was fixed.
- 01
Out-of-order device clocks had to lose to the server timestamp.
The Lambda orders samples by server receipt time, so device clock drift cannot reorder readings.
- 02
A spike of identical payloads could not inflate the chart.
Duplicate payloads are detected and dropped before they reach storage.
- 03
The read API had to avoid scanning the raw stream on every page load.
The read API serves the latest value from Redis rather than scanning the stream on every page load.
Security, speed and growth
How data is protected, and how the system stays fast as usage grows.
Security
Device and API access are authenticated, and processing happens inside the AWS account without exposing the raw stream.
Speed & growth
Latest-reading queries return in under 100ms from Redis, independent of stream volume.
Quality and releases
How changes were tested and shipped safely.
Replay tests with recorded bursts, impossible values and duplicate payloads verify what is accepted and rejected.
Tech stack
The tools and technologies used, for recruiters and technical reviewers.
Results
Measured outcomes recorded for this project.
| Metric | Result |
|---|---|
| Latest read | <100 ms |
| Rejected samples | rules python |
| Ingest | Kinesis burst |
Questions
Plain answers first, then the detail a technical reviewer would ask about.
Common questions
- What happens to bad readings?
- Impossible values and duplicates are filtered out, so the app shows the last good reading, not the last packet.
- Is it fast enough for live monitoring?
- Yes. The latest accepted reading is served in under 100ms.
For your tech lead
- What counts as a bad sample?
- Impossible values, such as a heart rate of zero, and duplicate payloads.
- Why not read from Kinesis directly?
- Scanning the stream per page load is slow; Redis keeps the latest value ready.
Key takeaways
- •Device readings like heart rate and blood pressure, cleaned up and shown live in under 100ms.
- •Built with Python, AWS Kinesis, Lambda, Redis, REST.