Trusting and Governing: Monte Carlo, Great Expectations, Soda, Collibra, Alation, Atlan, Unity Catalog, DataOS

Week 3 wrote SLOs as queries. These tools run them for you, tell people about breaches, and keep the definitions and access rules in one place.

Quality and observability

Tool	What it does	How you use it
Monte Carlo	Data observability: automatically monitors freshness, volume, schema changes, and distribution anomalies across the warehouse; shows lineage so you know which dashboards a broken table feeds.	Connect the warehouse; it learns normal patterns and alerts on deviations (your partial-file day would trip a volume monitor). Add custom SQL rules for business checks.
Great Expectations	Open-source framework of "expectations" (expect_column_values_to_be_unique) run inside pipelines, producing data docs.	Define expectation suites per table; run them as a pipeline step; fail the run on breach.
Soda	Checks written in a YAML-like language (SodaCL) run against the warehouse on a schedule, with a cloud UI for incidents.	checks for transactions: - duplicate_count(deal_id, amount, transaction_date) = 0.
dbt tests	The simplest option if you already use dbt: unique, not_null, accepted_values, relationships, plus custom SQL tests.	Declare in the model's YAML; dbt test runs them.

The difference: Monte Carlo discovers problems you didn't write checks for; Great Expectations, Soda, and dbt tests enforce the ones you did. Mature teams use both.

Catalog and governance

Tool	What it does
Collibra	Enterprise data governance: business glossary, data ownership, policies, workflows for approvals; strong in regulated industries.
Alation	Data catalog with search, popularity signals from query logs, stewardship, and lineage; the "Google for your data" pitch.
Atlan	Modern collaborative catalog: active metadata, lineage, Slack-style collaboration, embedded in the tools people already use.
Unity Catalog (Databricks) / Snowflake Horizon	Platform-native governance: one place for permissions, lineage, tags, and discovery inside the warehouse itself.

Data product platforms

DataOS (The Modern Data Company, the publisher of the Playbook and the State of Data Products reports) is a platform built around the data product idea itself: a self-serve layer where an engineer declares input ports, transforms, output ports, and SLOs in a spec, with contracts, a semantic model, and governance as first-class resources. Competing ideas appear as "data product" features in Databricks, Snowflake, and catalogs like Atlan. Whatever the vendor, the test is the same: can you see the owner, the contract, the SLOs, and the consumers of a product in one place?

Questions a Data PM asks

• Which SLOs are monitored, and who gets the page? A check nobody receives is documentation.
• Can a new analyst find the right table in five minutes? That's what the catalog is for; popularity and ownership matter more than descriptions.
• Where is the access policy, and is it enforced or advisory? PII masking in Unity Catalog is enforced; a Collibra policy document is advisory unless wired to the platform.
• Does the ontology from week 6 live in a tool, or in a slide?