Pipelines
Building, scheduling, and maintaining automated workflows that move data reliably from source to destination.
11 articles
-
Bronze, Silver, Gold: The Medallion Architecture Explained
The medallion architecture organizes a data pipeline into three refinement layers — bronze, silver, and gold — so raw data, cleaned data, and business-ready data never get confused with each other.
🍎 Low Hanging Fruit -
Building an Agent Harness for Data Engineering
A harness is what turns an LLM from a chat window into something that can safely operate on your pipelines — scoped tools, approval gates, sandboxing, and an audit log.
🌳 High Hanging Fruit -
Operational Telemetry, Explained for the Person Reading the Dashboard
Why that little dashboard you check for two seconds a day exists — and how it lets one person keep tabs on far more systems than they could ever check by hand.
🍎 Low Hanging Fruit -
How to Build a Data Pipeline
Every data pipeline comes down to three decisions: where the work runs, what triggers it, and where the data lands. Get compute, a scheduler, and storage right, and you can reframe the whole thing as a textbook ETL.
🍎 Low Hanging Fruit -
Start With Data Stakeholders Already Trust
Building analytics on well-known public datasets earns stakeholder credibility before you write a single custom pipeline. Once the framework proves itself on familiar ground, automation and analytics compound the value without the credibility risk.
🍎 Low Hanging Fruit -
Stop Forcing Tools Into Jobs They Weren't Built For
Most data pain is not a bad-tool problem. It is a phase-transition problem. A decision framework for four forks where teams pick wrong and stay wrong.
🌳 High Hanging Fruit -
The Silent Death of Orphan Data Pipelines
A data product loses value the moment active engagement stops — not because the pipeline breaks, but because data drifts, definitions shift, and timeliness erodes with no one in the loop to notice. The pipeline is the easy part. Keeping the output aligned with business reality is the ongoing obligation most organizations never plan for.
🌳 High Hanging Fruit -
ETL vs ELT: Choosing the Right Pipeline Pattern
ETL and ELT both move data from source to destination, but they make very different trade-offs between cost, flexibility, and setup complexity.
🍎 Low Hanging Fruit -
Change Data Capture Requires an ROI to Be Taken Seriously
CDC is powerful infrastructure, but it carries real costs in complexity, maintenance, and operational overhead. If you cannot articulate the return, you will not get buy-in — and you probably should not build it.
🌳 High Hanging Fruit -
Scheduling and Automating Data Pipelines
Move from manually running scripts to pipelines that run on a schedule — using cron, Python schedulers, and lightweight orchestration tools.
🌳 High Hanging Fruit -
Building Your First Data Pipeline
Combine data collection, cleaning, and storage into a repeatable automated workflow that keeps your data fresh and organized.
🌳 High Hanging Fruit