Postmortem: one feed went silent and every dashboard stayed green

One of the platforms we run monitoring for ingests real-time event data — two streaming feeds writing rows into Postgres continuously, each handled by its own consumer inside the same ingestion service. The app on top of it stayed up the whole time. No error was thrown. Every dashboard was green. And for a stretch of the day, one of the feeds wrote nothing at all.

This is the write-up of that incident (feed and table names changed throughout), because it's a clean example of the failure that normal monitoring is structurally blind to: not a crash, not a spike in errors, but a partial stop where the surrounding system looks perfectly healthy. The check that caught it is boring SQL you could write yourself in ten minutes, so most of this post is about how it works and how to build your own version — the Alertee part is one line at the end.

What happened

The streaming ingestion service ran out of memory. Not cleanly — it didn't crash and restart, which would have been loud. Instead one of its consumers got wedged under memory pressure and stopped making progress, while the rest of the process kept running. The enriched feed — call its table enriched_events — stopped writing rows. The adjacent raw feed, handled by a different consumer in the same service, kept going fine.

So the shape of the failure was: one feed dark, one feed healthy, same service, same database, no restart, no exception bubbling up anywhere a human would see it. The process was "up" by every liveness measure you'd normally trust.

Why it was invisible to normal monitoring

Walk the usual signals and watch every one of them say "fine":

• App uptime. The web app never went down. It served reads out of Postgres the entire time — enriched_events was still there, still queryable, still full of rows from before the stall. Uptime monitoring is a check on whether the server answers, and it did.
• Error rates. Nothing threw. A consumer that stops consuming under memory pressure isn't an exception — it's an absence of work. There's no stack trace for a row that was never written. Error-rate alerting fires on things that happen; this was a thing that stopped happening.
• Dashboards. The dashboards were reading recent data and rendering charts, and they were cached. A cached green tile is the most dangerous object in an incident — it's showing you the last known good state with total confidence and no timestamp on its own staleness. Nobody looks twice at a green dashboard.
• Infra metrics. Memory was high, sure, but "high memory" is the steady state of a streaming ingestion box. There's no clean threshold that says this high-memory reading means one specific consumer has silently stopped writing one specific table.

Every layer of conventional monitoring watches the machinery. None of them watch the outcome — did rows for this feed actually land in the last few minutes — and the outcome was the only thing that had changed.

The check that caught it

The checks that caught this run outside the ingestion pipeline entirely: a separate scheduler, on its own clock, hitting Postgres over a read-only connection every few minutes. That independence is the whole point — a monitor living inside the service that died would have died with it. They were plain row-count and freshness checks, one per feed, along the lines of "enriched feed has recent rows" and "raw feed has recent rows".

The core of it is a query no more exotic than this — count recent rows per source in a short trailing window:

SELECT source AS name, COUNT(*) AS value
FROM enriched_events
WHERE ts > now() - interval '15 minutes'
GROUP BY 1;

Two things make this catch the partial failure that a whole-table count would miss.

First, it's windowed on ts, not a total. SELECT COUNT(*) FROM enriched_events would have returned a big, healthy-looking number the entire time — the table was full of older rows. It's only when you ask "how many rows landed in the last 15 minutes" that a stopped feed reads as zero. The freshness window is what turns "the table exists" into "the table is still being written."

The close cousin is the pure freshness version, if you don't want to count:

SELECT source AS name,
       EXTRACT(EPOCH FROM now() - max(ts)) AS value
FROM enriched_events
GROUP BY 1;
-- value = seconds since this source last wrote a row

Second, it's grouped by source (name/value — that column naming is how Alertee runs one check across many dimensions at once, but the idea is just GROUP BY). This is what let it distinguish "the enriched feed is down" from "everything is down." The grid of results showed the enriched feed's sources at zero while the raw feed's check sitting right next to it stayed entirely green. That contrast — this dimension quiet, that adjacent one fine — is the diagnosis, handed to you before anyone opens a query editor.

What the alert actually said

The part that made this a non-event instead of an incident was where the alert started from. It didn't say "something failed" or "OOM detected." It said, in effect: the enriched feed stopped writing rows at around the time the stall began; the raw feed is unaffected and still writing. The inbox item's diagnosis pointed straight at an ingestion gap on one consumer, named the window it started in, and noted the neighbouring feed was healthy — which immediately rules out "the database is down" and "the whole service died" and points at one wedged consumer.

So the person who picked it up didn't start from zero. They started from "the enriched feed stopped at roughly HH:MM, the raw feed is unaffected" — which is most of the way to "restart the ingestion service and watch the rows resume." It was fixed before any user noticed a stale chart. The difference between a five-minute fix and a customer email is entirely in how far the first sentence of the alert gets you.

Most monitors tell you something broke. The useful ones tell you what, where, and since when.

Building your own version

You do not need us to do any of this. Here's the transferable recipe, and it's genuinely worth doing even if you never touch Alertee.

1. Monitor the outcome, not the machinery. The question is never "is the service up" — it's "did the rows that should exist actually show up." Count or timestamp the data, on the table your users ultimately read.

2. Window on an event timestamp. A total row count hides a stopped feed behind history. Always constrain to a trailing window (ts > now() - interval '15 minutes') or measure now() - max(ts). If your ingest is bursty, size the window to the longest gap you'd consider normal plus a margin — too tight and quiet-but-fine periods page you.

3. Group by the dimension that can fail independently. Feed, source, tenant, region, integration — whatever the smallest slice is that can go dark while its neighbours stay healthy. GROUP BY that column and evaluate each group. A single aggregate is exactly the thing that let the failure hide; the per-slice breakdown is exactly what names it.

4. Run it from outside the pipeline. A cron job on a different host, a read-only connection, its own schedule. If the monitor shares the box, the process, or the memory of the thing it watches, it goes down with it — and that's precisely when you need it awake. Read-only, so the monitor can never itself be the thing that corrupts production.

5. Make the alert carry context. "0 rows" is a starting gun, not a diagnosis. At minimum have the alert say which slice hit zero and when it was last healthy, so the responder starts from "the enriched feed stopped at HH:MM" instead of "grep the last hour of logs." Even a hand-rolled Slack message can include the failing source and the last-seen timestamp — put them in.

The honest floor for all of this is a shell script: run the grouped query, check for any slice at zero after its expected window, post to Slack. For your first few feeds that's the right call, it's free, and it's yours. What it doesn't give you is memory of which alerts were real, a place to see the per-slice grid at a glance, a schedule that knows about weekends and quiet hours, or a diagnosis attached to the fire — and that's the gap that starts to hurt once you're running more than a handful.

That's what Alertee is: the same plain SQL you'd write above — row-count and freshness checks you own and can read — run on an independent scheduler against a read-only connection, across every dimension at once, with the incident state and diagnosis that turned this OOM into a five-minute fix. If you're monitoring streaming data into Postgres or ClickHouse, connect a database and turn one check on.