Turbine: zero-downtime deploy orchestrator

The problem

deploys across a 40-node fleet meant someone watched dashboards for twenty
minutes and prayed. Rollbacks were tribal knowledge. Turbine replaced that
with a single state machine.

Design

• Health gates: every deploy stage polls /readyz against an SLO window;
  a regression below the gate aborts and flips traffic back atomically.
• Single source of truth: deploy intent is a versioned record, so
  "what is running where" is a query, not an archaeology project.
• Crash-only design: the orchestrator is stateless; a killed process
  resumes mid-deploy from the record.

Results

Deploys went from 25 minutes of human babysitting to 4 minutes fully
automated. Rollbacks — previously "ask Dave" — are now a single command
that has been exercised weekly in game-days.

func (o *Orchestrator) Promote(ctx context.Context, d Deploy) error {
    if err := o.gate(ctx, d, StageCanary); err != nil {
        return o.rollback(ctx, d) // health gate failed: flip back
    }
    return o.shiftTraffic(ctx, d, 100)
}

What I'd do differently

gRPC streaming for gate telemetry was overkill; a 1Hz polling loop would
have shipped a month earlier. Building the dashboard last, not first, was
the right call — the CLI was the real product.