Moving and Shaping Data: Fivetran, Airbyte, dbt, Airflow, Kafka, Informatica, Talend, Ab Initio

Two generations coexist. Enterprise ETL suites (Informatica, Talend, Ab Initio) transform data on their own engines before loading it, with heavy lineage and governance built in; you'll meet them in banks, telecoms, insurers, and anywhere with decades of batch jobs. The modern ELT stack (Fivetran or Airbyte to load, dbt to transform in the warehouse, Airflow to schedule) is what most new data teams assemble.

Tool	Role	How you use it	Watch out for
Fivetran	Managed connectors: SaaS and databases → warehouse, on a schedule, with automatic schema handling.	Pick a source (e.g. Salesforce), authorise, choose tables, set sync frequency. Done.	Priced on rows changed per month (MAR); a wide CRM sync can get expensive. You don't control the transformation.
Airbyte	Open-source alternative to Fivetran; hundreds of community connectors; self-host or cloud.	Same flow; more connectors, more maintenance.	Connector quality varies.
dbt	Transformations as versioned SQL SELECTs ("models") that run inside the warehouse, with built-in tests, documentation, and lineage.	Write models/gold/sales_metrics.sql, declare tests (unique, not_null, accepted_values), run dbt build.	It's the "T" only; it needs a loader before it and a scheduler around it. Model sprawl without ownership.
Apache Airflow	Orchestration: define DAGs (task graphs) in Python, schedule them, retry, alert.	A DAG runs Fivetran → dbt → dashboard refresh at 02:00; the Airflow UI shows every run's status and duration.	Airflow knows a task ran, not that its data is right: week 3's whole lesson. Dagster and Prefect are the newer alternatives.
Apache Kafka	Distributed event streaming: producers write events to topics; consumers read them in order; partitions scale it. Confluent sells the managed version; AWS Kinesis is the equivalent.	Model each business event (deal_stage_changed) as a message; downstream systems subscribe.	Exactly-once, ordering, late events, schema evolution (schema registry). Streaming is where "events" in your PRD live.
Informatica	The long-standing enterprise data-integration and management suite (PowerCenter on-prem, IDMC in the cloud): integration, quality, MDM, catalog.	Graphical mappings; strong governance and lineage; typical in large regulated enterprises.	Cost and specialist skills; often the incumbent being migrated from.
Talend	Open-source-rooted ETL/ELT suite with data quality and governance; now part of Qlik.	Graphical job design generating Java; on-prem or cloud.	Mid-market fit; product direction after the Qlik acquisition.
Ab Initio	High-performance enterprise data processing platform: a graphical development environment (GDE), a parallel execution engine (the Co>Operating System), and a metadata hub with end-to-end lineage.	Build "graphs" of components (read, transform, join, write) that run in parallel across many CPUs; very fast for huge batch volumes.	Proprietary, expensive, rarely seen outside large banks, telecoms, and government; skills are scarce. When you meet it, the question is usually "what would it take to move this?"

How this maps to Meridian

crm_deals_ingest is a Fivetran-style connector; gold_sales_metrics is a dbt model run by Airflow; the stage-history log is what a Kafka topic of deal_stage_changed events would give you for free.

Questions a Data PM asks

• ETL or ELT? If the transform happens before the warehouse, the raw data isn't available to re-derive metrics when the definition changes.
• Where are the tests? dbt tests, Great Expectations, or nowhere?
• What happens when a run fails at 02:00? Who's paged, and how long until recovery (week 3's failed-recovery query)?
• How does a schema change upstream reach us? A renamed CRM field can silently zero a metric.