Research · Data Platform Engineering
Lakehouse Architecture Patterns for AI-Ready Data Platforms
Why bronze-to-gold alone isn't enough once agents are a consumer of your data platform, not just BI — and the patterns that make the difference.
The medallion pattern — bronze, silver, gold — is close to a default now for anyone building a lakehouse. That's a good thing; it's a genuinely sound shape for promoting raw data into something trustworthy enough to build on. But the pattern alone was designed for a world where the platform's consumers were dashboards and analysts. Once the consumer is an autonomous agent that acts on what it reads — files a remediation, approves a change, triggers a payment — a bad record isn't a wrong number on a chart. It's a wrong action in the world. That changes which parts of the medallion pattern are optional and which aren't.
The pattern itself, briefly
Bronze holds raw data close to its original form — whatever format the source system actually produced, unmodified, so nothing is lost before anyone's decided what matters. Silver is where records get cleansed and conformed into a consistent schema. Gold is where business logic turns conformed data into something a consumer can actually use without re-deriving it themselves: a data product, not a table. The promotion between layers is what's supposed to be doing the governance work — and in our experience, it only does that work if two things are true that the pattern itself doesn't guarantee.
Quality and governance belong at the gate, not after it
The first is that quality and privacy checks have to run at the Bronze→Silver boundary itself, in real time, not in a nightly batch job that catches the problem a day after agents have already acted on it. In the reference architecture we run for clients, that boundary performs quality scans, PII masking, and policy-rule evaluation on every record as it's promoted — not sampled, not batched — and a real share of incoming records (on the order of 2% in the environments we've built this for) gets quarantined rather than silently passed through. See the full reference architecture for the diagram this is drawn from.
That quarantine rate is the point, not a flaw to optimize to zero. A gate that never rejects anything isn't checking anything. The number that should worry a platform team is a silently dropping quarantine rate with no corresponding change in the source systems — that's usually a sign the gate stopped looking, not that the data got cleaner.
Metadata is what makes gold discoverable, not just curated
The second thing the pattern alone doesn't guarantee is discoverability. A beautifully curated gold-layer table nobody can find is no more useful to an agent than raw bronze — an agent can't reason over a data product it doesn't know exists or doesn't understand the shape of. An active metadata layer spanning bronze through gold, built into a catalog rather than maintained as a separate wiki page, is what turns a pile of curated tables into something a consumer — human or agent — can actually discover and trust the meaning of.
This is also the exact surface our own OmniMesh Compliance product watches: its Lineage Agent reasons over this same metadata boundary to score the blast radius of a schema change before it merges. A lakehouse with strong promotion gates but no metadata layer is legible to the people who built it and opaque to everyone — and everything — else.
Closed-loop observability, because the consumer can act
The third pattern is the one that matters specifically because the consumer is agentic, not just analytical. A BI dashboard that's fed bad data produces a wrong chart someone eventually notices. An agent fed bad data takes a wrong action, possibly before anyone's looking. That asymmetry is why observability in an agent-ready lakehouse can't stop at "is the pipeline running" — it has to capture telemetry from every hop, every rule, and every agent decision, and feed that telemetry back into tuning the gates and retraining the agents that depend on them. The loop only closes if what the agents did downstream actually changes what the platform does upstream.
What this changes in practice
None of this replaces the medallion pattern — bronze, silver, and gold are still the right shape. What changes is which parts of the pattern you can treat as optional. A warehouse serving dashboards can get away with batch quality checks, a thin metadata layer, and observability that's really just uptime monitoring. A lakehouse serving agents can't, because the cost of a bad record stopped being a wrong number and started being a wrong action — and an architecture built for the first case will not, by itself, protect against the second.
Where this connects
This is the data half of the loop we describe in The AI Development Life Cycle: A Spec-Driven Loop for Shipping Production AI: the ADLC's "specify the data before touching it" stage is only credible if there's a real platform underneath it enforcing that specification at promotion time, not just a document asserting it.
Changelog
- Initial publication.
Want a lakehouse your agents can actually trust?
We'll assess your current data estate against this reference architecture and show you the gap.