Explain the medallion architecture pattern. How do the bronze, silver, and gold layers work, and what are the design principles behind each?

Question

Accepted Answer

The medallion architecture is a data design pattern that organizes data into progressively refined layers: bronze, silver, and gold. Each layer serves a distinct purpose and applies increasing levels of quality, structure, and business logic. This pattern is widely used in lakehouse architectures and promoted by platforms like Databricks.

The bronze layer captures raw data exactly as it arrives from source systems. No transformations are applied. Data is stored in its original format, whether that is JSON from APIs, CSV from file exports, Avro from Kafka topics, or change data capture events from databases. The key principles are completeness and immutability. Every record is preserved, including malformed or duplicate records. This creates an audit trail and enables reprocessing when transformation logic changes. Metadata like ingestion timestamp, source system identifier, and batch ID is added to support lineage tracking.

The silver layer applies cleaning, standardization, and conforming transformations. This is where data quality work happens. Deduplication removes repeated records. Data types are enforced and standardized. Null handling, format normalization, and basic validation occur here. Related data from different sources is joined and conformed to common schemas. The silver layer represents a trusted, cleaned version of the data that is suitable for most analytical use cases. Think of it as the enterprise-wide data foundation that multiple teams can build on.

The gold layer contains business-level aggregations, metrics, and datasets optimized for specific use cases. These are the curated datasets that directly serve dashboards, reports, machine learning models, and data products. Gold tables often align with business domains: revenue metrics, customer segmentation, product analytics. They are denormalized for query performance, pre-aggregated where appropriate, and documented with clear business definitions.

Design principles across the layers include incremental processing wherever possible. Rather than reprocessing all data on each run, each layer processes only new or changed records. This requires tracking watermarks or change timestamps. Schema evolution should be handled gracefully at each layer. The bronze layer accepts any schema since it stores raw data. The silver layer must handle evolving source schemas through mapping logic. The gold layer shields consumers from upstream changes.

Testing and validation should be embedded at each layer transition. Validate record counts, check for unexpected null rates, verify referential integrity, and alert on anomalies. dbt is commonly used for silver-to-gold transformations with built-in testing. Great Expectations or Soda can validate data at any layer boundary.

The medallion pattern can be extended with additional layers when needed. Some organizations add a "platinum" layer for highly curated, externally shared data products, or a "raw" layer before bronze for completely untouched landing zone data. The core principle remains the same: progressive refinement from raw to business-ready.

One common mistake is blurring the boundaries between layers. When transformation logic creeps into the bronze layer or raw data leaks into gold tables, the architecture loses its value. Maintain clear contracts between layers about what guarantees each provides.