Build an ELT pipeline from Postgres to a warehouse

Project Description

Analytics currently runs against a read replica of production, which is slow for us and occasionally unpleasant for production. We want a proper warehouse: ingestion from Postgres plus two SaaS sources, dbt models with tests, orchestration in Airflow, and documented lineage. We care that the models are testable and that a failed load is loud rather than silent. We have been bitten by a pipeline that half-succeeded for a week. About 40 source tables. Please tell us how you would structure staging and marts before you tell us which tool you like.

Skills Required