Weekly releases across AWS and the rack
Shipping low-latency products when half the footprint is cloud and half is yours taught me that identical pipelines beat heroic SSH. Notes from Nomad, Terraform, and boring Fridays.
I used to think “hybrid” meant clever architecture diagrams. In practice it meant two places where things could go wrong, and one team on the hook for both.
When you are shipping interactive surfaces that care about milliseconds, the worst failures are not mysterious algorithm bugs—they are drift: a Terraform apply that never reached the cage, a Nomad job spec that only existed in someone’s notes, a certificate that aged out because nobody owned renewal on the metal side.
One pipeline shape, two substrates
The habit that saved us was refusing to treat on-prem as a special snowflake. Same job spec shape, same health checks, same artifact promotion story—only the scheduler target and secrets injection changed. If your rack path needs a bespoke bash script every release, you have not automated a hybrid system; you have invented two products.
- Promote artifacts, not branches. Tag immutable builds once, let both environments pull the same digest.
- Make rollback a first-class button, not a runbook paragraph. Panic is when people skip steps.
Terraform state is a team sport
Splitting state files without discipline turns “who touched prod?” into folklore. I am biased toward environment-scoped workspaces or clearly named backends, with CODEOWNERS on the modules that touch networking. Small PRs on infra beat big-bang refactors—especially when DNS and load balancers are involved.
Observability that crosses the fence
If your dashboards stop at the VPC boundary, you will debug half a story. I want traces or request IDs that survive edge → origin → worker even when one hop is not HTTP-shaped. When that is painful, it usually means the integration contract was fuzzy—fix the contract, not just the dashboard.
What I still do on new engagements
- Draw the promotion path on one page before writing more YAML.
- Agree who owns TLS, DNS, and firewall tickets—in writing.
- Schedule a dry-run deploy that includes the unhappy path (rollback, partial failure).
Hybrid is not exotic. It is two environments that both deserve the same seriousness you would give a single-region cloud launch.
If you are tightening a similar setup, I am happy to compare notes—contact details are on this site.