Keeping small data pipelines boring

Most teams we work with do not need a data platform. They need three scripts that run every night and tell someone when they fail.

Most of the teams we work with do not need a data platform. They need three or four jobs that run every night, move numbers from one place to another, and tell a human when something went wrong. The hard part is not building those jobs. The hard part is keeping them understandable a year later, after the person who wrote them has moved on to something else.

Over the past few years we have settled on a handful of habits that keep small pipelines boring in the best sense of the word. None of them are new, and none require special tools.

One job, one purpose

A job should do one thing that can be described in a single sentence: "copy yesterday's orders from the shop into the reporting database", or "build the weekly stock summary". When a job starts to need the word "and" in its description, it is usually time to split it. Smaller jobs fail in smaller ways, and the error message tells you where to look.

Make every run repeatable

The most useful property a pipeline can have is that running it twice gives the same result as running it once. If a job is interrupted halfway, you should be able to simply run it again. In practice this means writing to a staging table first and swapping it in at the end, using the date being processed as a parameter rather than "now", and avoiding steps that append without checking what is already there.

Repeatable jobs also make backfills trivial. When the accounting team asks for last March to be recalculated, you run the job with a different date and go back to your coffee.

Plain files beat clever infrastructure

For a small team, a scheduled task on a single server, a version-controlled folder of scripts and a log file is often enough. A queue, a cluster and a dashboard all have their place, but each one is another thing to update, monitor and explain. We add infrastructure only when a concrete problem demands it, and we write down which problem that was.

Failures should be loud and specific

A pipeline that fails silently is worse than one that does not exist, because people keep trusting the numbers. Every job we build ends with a check: did we load roughly the number of rows we expected, are the totals within a sensible range, did the newest record arrive. If not, the job exits with an error and sends a short message that says which check failed and what the numbers were.

The message matters more than the alerting tool. "Order import loaded 12 rows, expected at least 200" can be acted on immediately. "Job failed" sends someone digging through logs.

Write the runbook while you still remember

For each job we keep a short note next to the code: what it does, where its inputs come from, who uses the output, how to rerun it, and what to do when each check fails. It rarely runs longer than a page. Writing it takes twenty minutes on the day the job is finished, and saves hours the first time something breaks at an inconvenient moment.

When to stop being boring

There is a point where a handful of scripts stops being enough: when jobs start depending on each other in complicated ways, when data volumes outgrow a single machine, or when several teams need to share the same definitions. That is the right moment to look at orchestration tools and proper warehouses. Until then, boring is a feature.