The Modern Data Stack: a Map

Vendors change; the layers don't. Read the stack left to right, the way data flows, and every tool name becomes a position rather than a brand.

Layer	What it does	Typical tools	At Meridian
Sources	Operational systems that create data	CRM (Salesforce, HubSpot), ERP (SAP, NetSuite), product events, files	The CRM, the billing system
Ingestion / integration	Move data from sources into storage; batch or streaming	Fivetran, Airbyte, Informatica, Talend, Ab Initio, Kafka, Kinesis	crm_deals_ingest, transactions_ingest
Storage + compute	Warehouse or lakehouse where data lands and SQL runs	Snowflake, Databricks, BigQuery, Redshift, Microsoft Fabric	The database behind the labs (SQLite stands in)
Transformation + orchestration	Turn raw tables into modelled, tested tables on a schedule	dbt, Spark, Airflow, Dagster, Prefect	gold_sales_metrics and its daily run
Semantic layer + BI	Define metrics once; explore and visualise	Looker, Tableau, Power BI, Qlik, Metabase, Redash	The Sales Funnel Accelerator dashboard
Quality, observability, catalog, governance	Know it's fresh, complete, unique; find it; control access	Monte Carlo, Great Expectations, Soda, Collibra, Alation, Atlan, Unity Catalog, DataOS	The week 3 SLOs, the ontology in week 6
ML / data science	Train, track, and serve models	MLflow, SageMaker, Vertex AI, Dataiku, Jupyter	The agent pilot in week 6

Three things a Data PM keeps straight

1. ETL vs. ELT. Classic tools (Informatica, Ab Initio) transform before loading into a warehouse. The modern pattern loads raw data first (Fivetran) and transforms inside the warehouse (dbt): ELT. That's why Bronze/Silver/Gold exists.
2. Batch vs. streaming. A nightly load is batch (Airflow-scheduled). Events arriving continuously are streaming (Kafka, Kinesis). Most sales analytics is batch; fraud and operations are streaming.
3. Where the definition lives. If "revenue" is computed in five dashboards, you have five definitions. Semantic layers (Looker's LookML, dbt's metrics, DataOS contracts) exist to define it once, in one place, and have every consumer read it.

How to read Gartner's market categories

Gartner splits this into Cloud Database Management Systems (warehouses and lakehouses), Data Integration Tools, Analytics and BI Platforms, Data Quality Solutions, Metadata Management / Data Catalogs, and Data Science and ML Platforms. The categories map one-to-one onto the rows above. When someone says "we're evaluating a Magic Quadrant leader," ask which row.