Stop Building Workflow Plumbing

Most teams building operational automation spend the majority of their engineering effort on things that are not their business logic: retrying failed steps, scheduling when work runs, tracking what state a long-running process is in, making sure an action does not fire twice, and recovering cleanly when something crashes halfway through. None of that is the rule that decides what to do. It is the plumbing that gets the rule executed reliably, and hand-building it is one of the most common ways automation projects run over budget.

The plumbing tax

Picture a team automating a simple operational rule: when a customer’s order sits unshipped for more than 48 hours, send an alert and open a support ticket. The rule itself is one sentence. Building it reliably requires a scheduler to check the condition periodically, a way to track which orders have already triggered an alert so the same order does not fire twice, a retry path for when the ticketing API is briefly down, a mechanism to recover the in-progress check if the process restarts mid-run, and logging good enough to answer “did this actually fire for order 4471.” That is five separate engineering problems supporting one sentence of business logic, and every one of them is a place a bug can hide.

Most teams underestimate this because the plumbing does not show up as a line item until it breaks. It breaks quietly, and it breaks in three specific, expensive ways.

The three failure modes that make this expensive

Silent partial failure. A workflow does step one and two, then step three throws an error that gets logged but not surfaced anywhere a human will see. The system now believes the task finished, or it never checks again. Nobody notices until a customer complains about something that should have been automatic weeks earlier.

Duplicate side effects. A retry fires because a response was slow, not because the first attempt actually failed, and now the charge, the email, or the ticket gets created twice. Idempotency, making sure an operation has the same effect whether it runs once or five times, sounds like a minor detail until a payment or a customer-facing notification fires twice and someone has to explain why.

Lost state on restart. A long-running process gets interrupted by a deploy, a crash, or a scaling event, and the in-memory record of where it was in a multi-step sequence is gone. The workflow either restarts from zero, redoing work and risking duplicate side effects, or it silently stops, and nobody notices until a downstream report looks wrong.

Each of these is solvable. None of them is trivial, and all three recur across every workflow a team builds, which means the cost of building them once badly is paid again on every new automation.

Push the plumbing onto infrastructure

The argument here is not “never write orchestration code.” It is that retries, scheduling, state tracking, and idempotency are solved problems with mature infrastructure behind them, and re-solving them per project is a poor use of engineering time that should be going into the actual domain rules: what counts as a stalled order, what the right escalation threshold is, what “resolved” means for this specific business. That is the logic only your team understands, and it is the part worth protecting engineering time for.

A team that keeps its custom code to “what should happen” and leans on infrastructure for “make sure it reliably does happen” ships faster and debugs less, because the hard, general-purpose failure modes above are handled by something battle-tested rather than something built once under deadline pressure and never revisited.

When to build it yourself

This is not a blanket argument against custom orchestration. If your process genuinely does not need retries, because every step is instant and idempotent by nature, plumbing is not a real cost and building it yourself is fine. If your reliability requirements are unusually specific, such as strict ordering guarantees across steps that a general tool does not model well, a hand-built solution tuned to that exact requirement can outperform a generic one. And a small, single-purpose script that runs once a day with a human checking the output does not need the same rigor as an unattended, customer-facing process; adding infrastructure there is over-engineering, not diligence.

The judgment call is volume and consequence: the more often a workflow runs unattended and the more it touches things customers or money can see, the more the plumbing tax matters, and the stronger the case for pushing it onto something designed to handle failure, not something assembled to make a demo work.

Business logic is the only part of an automation project a competitor cannot easily copy. Everything else is worth treating as a commodity, because it already is one.