The problem
The job is to catch real-world trigger events in public Telegram communities while they are still happening. A flood in an apartment building is the textbook case. When it happens, the residential chat lights up in a particular way: a photo of water coming through the ceiling, neighbours confirming it reached their floor, someone asking whose fault it is. That mix of damage, emotion and mutual help is the signal, and for an insurance or marketing client it is worth money while the carpets are still wet, not three weeks later.
Keyword matching cannot find it. People mention water for a hundred reasons that have nothing to do with an incident: utility bills, rusty tap water, rentals with water included, plumbing ads, reposts about a flood three regions away. A keyword match is a candidate, and a weak one.
The obvious idea was to run every message through an LLM. It died on arithmetic. Tens of millions of messages a month, most of them noise any model would reject in its first token. Paying a model to read them is paying for confidence you already have.
The decision: treat it as a query, not an AI problem
A database planner does not run the most expensive operation on every row. It applies cheap filters first and saves the expensive work for what survives. I designed the product the same way: a funnel where every stage is more accurate and more expensive per message than the one before it, and sees far less data. That inverse relationship is the whole architecture.
How the cascade works
1. Postgres full text search. The first cut runs inside the database: no data leaves it, one query over tens of millions of rows. The Russian dictionary lemmatizes words, so one term catches every form of a verb. A message qualifies if it has an event word, or a leak word plus a location word, mentions a water object such as a pipe or a radiator, and does not contain billing or real estate vocabulary. That one exclusion group removed a huge share of noise, because in residential chats most water talk is money talk. The filter is deliberately wide: its job is never to miss a real incident while cutting millions of rows to thousands.
2. Regex scoring in Python. Unfashionable and one of the highest-leverage parts of the system. Explicit statements of fact take a fast path. Everything else runs a chain of drop filters, each built from a failure mode we found by reading real false positives: plumber ads written to look like incident stories, real estate listings, water quality complaints, scare posts about flood risk, fires where the water came from firefighting, long digests with no concrete fact. Every decision writes a score and a structured reasons object to the database, so when a client asks why a message is or is not in their report, we answer with specifics.
3. A cheap binary filter. A small model answers one question: is this a real, current incident? Output capped at ten tokens, a fraction of a cent per message.
4. A detailed classifier. The same cheap model with a serious prompt, worked examples and strict JSON: incident type, confidence, supporting quotes, reasoning. For most messages this is the final word.
5. A reasoning model, only for doubt. It wakes up on three triggers. Confidence between 0.40 and 0.75, because a confidence of 0.1 or 0.9 is already a decision and only the middle band is doubt. Messages over 500 characters, where small models lose the one fact buried in ten paragraphs. And advertising markers, because an ad that survived two filters is exactly the kind engineered to look like an incident.
Every level fails downward: if the reasoning call errors, the classifier's verdict stands, and results are saved after each level so any level can be rerun on its own. The pipeline degrades under failure instead of stopping.
How I measured it
I set the KPI and the pass bar, then ran five versions of the classification pipeline against roughly 1,070 hand-labelled messages. The error rate went from 7% to 1.6% across those rounds, and the production cascade holds 95% accuracy on a labelled holdout at tens of millions of messages a month.
We optimised for precision over recall on purpose. A false positive in a paid intelligence report damages trust in a way a quietly missed incident does not. That trade-off depends on the product; for this one it was clearly right.
What it changed
Volume and cost were decoupled. The expensive part of the system sits behind two layers of nearly free computation, so when message volume grew, the LLM bill mostly did not.
The output became explainable. The reasons log turned "the model said so" into an answer a client can check.
It became a platform. The same pipeline, running on our own embeddings across 12,000 topic categories, feeds three B2B product lines I took from zero: an ML interest profiler, behavioural profile history and channel audience insight. I owned go to market on each and made the post-launch call to continue, expand or cut. Next on top of it is a Telegram Ads targeting agent that turns a natural language brief into interest categories and the SQL underneath, traced in Langfuse.
What I would tell another product manager
The model is not the product. Prompts were maybe a tenth of the work. The product was the watermark table that stops rescans, the idempotent writes, the retry policy, the reasons log and the escalation rules. If you only review prompts, you are reviewing the decoration.
Match model cost to task complexity. A true-or-false question does not need a reasoning model.
Read your false positives. Every drop filter in stage two came from real garbage in the output. Plumbers writing fake flood stories was in nobody's planning document.
Before reaching for a bigger model, ask the query planner's question: what is the cheapest filter that removes half of this data? Then ask it again.
The longer essay on this system: Postgres First, GPT Last on Medium.