We should be defining success, not failure

For years, data reliability has started with one question: what do you want to monitor?
‍
It assumes someone can predict every way the data will break. That question still matters, especially when there are specific failures you know would be critical. But it can’t be the whole model, and with AI agents now querying data in new ways, it’s becoming much harder to keep monitoring coverage aligned with how the data is actually used.

Data reliability should start from a different place: understanding what each asset is for, who depends on it, and what has to be true for it to be trusted. Most of that can be worked out automatically and kept current as the data and its users change. People should only have to add what the business knows and the data can't show.

The hard question and the easy one

Try this. Pick any important table in your warehouse and answer two questions.

Question one: What are all the ways this table could be wrong? Late data. Partial loads. A renamed column upstream. Duplicates after a retry. A currency field that quietly goes null. A source that starts sending test records. You'll get to ten and still know you've missed some. Nobody can reliably imagine failures they haven't seen yet.

Question two: What is this table for, and what does the business expect from it? "It feeds the exec revenue dashboard. Finance needs it complete by 7am." Ten seconds.

The monitor-first model is built entirely on question one. Every test, threshold and anomaly rule is a guess about what might break, and a person has to make each guess by hand. The problem is relying on that question alone. Question two takes people seconds, and much of it can be answered without a person at all. That's the question reliability should be built on.

Start with what good looks like

You can't know what could break until you know what something is. So the right starting point isn't a monitor. It's an understanding of the asset, built from signals that already exist in every modern stack: lineage, transformation code, pipeline runs, query history, dashboards, owners, and the agents reading the data.

For every asset, a reliability system should be able to answer four questions:

What kind of asset is it? A source, a staging model, a fact table? Batch or streaming? What's its normal update rhythm?
What is it for? Which business process, metric or decision does it serve: revenue reporting, billing, a customer-facing feature, a model's training data? This is what tells you what a failure actually costs.
Who depends on it? Which dashboards, models, teams, workflows and AI agents use it, and how often?
What makes it reliable? Given what it's for and who uses it, what has to hold: arrival time, completeness, valid values, healthy upstream pipelines, a stable schema?

The answer should read like something a senior engineer would tell a new hire:

"fct_revenue is a daily production fact table, usually landing by 6:45, fed by Salesforce and billing. It's the source of truth for the daily revenue finance reports to leadership, and it powers the executive dashboard and the forecasting agent. It's reliable when it's on time, complete, and its upstream syncs succeed."

Most of that shouldn't need anyone to write it down. And once it's understood, the right checks follow from it, instead of from someone's best guess.

People should supply intent, not checks

Automating the work doesn't mean the system should guess everything. Some context only people have. Good reliability should ask for exactly that, and nothing more: not monitors, thresholds or test configs, but intent.

Four kinds of context fall into that category, because no amount of metadata can reveal them.

What matters most. A system can see how often a table is queried, but usage isn't importance. The quarterly regulatory report might be read four times a year and still be the one table that can never be wrong.
What was promised. History shows when a table usually lands. It can't show that finance was promised it by 7am, or that a customer contract depends on it arriving on time.
What "correct" means. Patterns show what's normal, not what's right. If a bug has been inflating revenue for three months, it looks perfectly normal. Rules like "revenue is never negative" or "every order has a customer" come from people who know the business.
Who owns it. Ownership can be suggested from code and commit history. But who is accountable, and who should hear about a problem, is a decision people make.

Even here, the effort should be small. The system should propose and people should confirm. "This looks like a critical finance table that usually lands by 6:45. Is it needed by 7?" is a yes-or-no question, not a configuration task. A team that answers none of these should still be covered by what can be inferred. Every answer they do give makes it sharper.

Signals are evidence. Reliability is the conclusion.

Once you know what an asset is for, you can do something monitors alone can't: judge. A late load, a volume dip, a failed test or a schema change is evidence. Whether the data can be trusted is the conclusion, and that conclusion is what people and agents actually need.

What you observe	What you should conclude
`fct_revenue` is 35 minutes late. On Mondays it usually varies by up to 45.	Still reliable. No incident, no noise.
3 hours late, the Salesforce sync failed, volume is down 43%.	Unreliable, high confidence. One incident.

And when it's real, the output should be the story, not a pile of alerts:

"Revenue reporting is unreliable. Data has been incomplete since 06:10 because the Salesforce ingestion failed. The executive revenue dashboard and the forecasting agent are affected. Severity: critical."

What happened, why, what's affected and how much it matters. That's how engineers already reason during an incident. The reasoning should happen before anyone opens Slack, not after.

Reliability has to improve itself

In the agentic world, the data estate doesn't sit still. New models ship every week. An agent starts joining tables nobody has touched in a year. A dashboard nobody used suddenly becomes the CFO's favorite. A monitoring config written by hand is out of date the day after it's written.

That's why reliability can't be a setup project. It has to be a loop that runs on its own:

Rediscover continuously. New assets and new consumers should be picked up and understood without anyone filing a ticket.
Check on demand. When a person or an agent queries something that isn't well covered, it should be validated at that moment, with a clear note on what couldn't be verified.
Keep what worked. An on-demand check that caught a real problem should become permanent coverage.
Learn from every incident. Was the root cause right? Was it a false alarm? Was a reliability dimension missing? Each answer should tune the next pass.

Done right, coverage follows how data is actually used, not how it was used when someone last had time to write tests.
‍