feat(health): serve healthz and readyz

[bdchatham]

Stacked on #71. Base is brandon2/amm-swap-scenario, so this branch carries the
AMM scenario, the ERC721 gas fix and these endpoints together, and the image it
builds is the one to deploy.

Why

sei-load serves /metrics and nothing else. A deployment therefore has no way
to tell a run that is still starting from one that is stuck, and a pod counts as
available the moment its container starts.

That blocks the load-generator deployment in the platform repo, whose pod spec
has a startupProbe, a readinessProbe and a livenessProbe pointing at
/readyz and /healthz. Against the current binary every one of them fails.

What the two endpoints mean

/healthz answers as soon as the HTTP server binds. It never reads the startup
sequence.

/readyz refuses until the dispatcher is running, and while it refuses it names
the phase.

Keeping them separate is the point rather than a detail. Funding, deployment and
prewarm take minutes against a cold chain. A liveness probe that reported the run
dead for that window would restart the pod before it sent a transaction, then
restart the next attempt at the same place. The run would never happen, and the
cause would read as a crash loop rather than a slow start.

Measured against the binary, polling both endpoints across a real startup:

t+3s    healthz=200   readyz=prewarming accounts [503]
t+6s    healthz=200   readyz=prewarming accounts [503]
...
t+18s   healthz=200   readyz=prewarming accounts [503]
t+21s   healthz=200   readyz=running [200]
SIGTERM -> exit 0

The phase in the body is there for the operator watching that window. A
ten-minute startup that answers only 503 says nothing about which step is slow.

One design note

The ready flag and the phase are stored as a single value, not as two atomics.
Two would leave a window where a writer has set the flag but not yet the phase,
so a reader sees the run serving while the body still names the step it left.
The status line and the body would then disagree about the same instant.

Phases

phase	reported while
starting	before the run reaches its first step
deploying contracts	the generator deploys what the profile names
funding accounts	the funder pays the account pool
prewarming accounts	prewarm sends one transaction per account
running	ready
shutting down	after SIGTERM, through the post-summary scrape hold

shutting down drops readiness while /healthz keeps answering. The run holds
the pod open on purpose for that scrape window, and a liveness probe that failed
during it would kill the process before its final metrics were read.

Verification

Five mutations, five caught:

• liveness made to wait for startup
• readiness made to always pass
• readiness made to stop naming the phase
• NotReady made a no-op
• the single stored value split into two atomics — a reader observed a serving
  status carrying funding accounts

The fifth is worth naming. The first version of that guard passed against the
split-atomics mutation, so its failure message claimed something it could not
detect. It was rewritten to widen the window before it was believed.

gofmt, go vet and golangci-lint run are clean. The full suite passes, and
the health package passes under -race.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Operational probe endpoints on the metrics port with no auth or transaction-path changes; incorrect wiring could affect pod scheduling but behavior is heavily tested.

Overview
Adds a health package that exposes /healthz and /readyz on the existing metrics HTTP server so Kubernetes startup, readiness, and liveness probes can succeed without treating long cold-chain startup as a crash.

/healthz always returns 200 while the process is listening and does not consult startup state. /readyz returns 503 with a plain-text phase (starting, deploying contracts, funding accounts, prewarming accounts) until the run calls Ready(), then 200 with running. NotReady("shutting down") is deferred when the load test is up so traffic stops during the post-summary scrape hold while liveness stays OK.

State is a single atomic pointer pairing the ready flag and phase so probe responses cannot disagree. main.go creates probes before the metrics server starts, registers them on the mux, advances phases through generator deploy, funding, and prewarm, marks ready when the dispatcher starts, and drops readiness on every exit path.

Tests lock in the split between liveness and readiness, phase bodies, shutdown behavior, and concurrent consistency (including race-sensitive storage).

^{Reviewed by Cursor Bugbot for commit d555a26. Bugbot is set up for automated code reviews on this repo. Configure here.}

[seidroid]

Adds a small, well-tested health package serving /healthz and /readyz on the existing metrics mux, with the ready flag and phase stored as one atomic value. The separation of liveness from readiness is correct for a run with a minutes-long startup; the only gap is that readiness is dropped on the signal path only, so a duration-bounded or error exit leaves /readyz reporting running through the shutdown/flush window.

Findings: 0 blocking | 2 non-blocking | 1 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] Probes.Enter and Probes.NotReady have identical bodies; the distinction is documentation-only. Consider having NotReady delegate to Enter (or drop one) so the two cannot drift apart.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[suggestion] NotReady only runs on the signal path. When --duration expires (or a background worker fails), utils.Recv returns ctx.Err() at line 396 and returns early, so this line never executes. The run then proceeds through LogFinalStats, EmitRunSummary and the PostSummaryFlushDelay sleep (25s by default) while /readyz still answers 200 running — exactly the window readiness is meant to cover, and duration-bounded runs are the common deployment shape. Registering it once for every exit after Ready() covers all paths, e.g. defer probes.NotReady("shutting down") placed right after probes.Ready() (line 371), keeping the log line where it is.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 137675a. Configure here.}

[bdchatham]

Correct, and confirmed by running it rather than by reading. Fixed in d555a26.

NotReady ran on the signal path alone, so a duration-bounded run returned early at utils.Recv and never reached it. Readiness then stayed true through the final statistics, the run summary, and the post-summary hold — which defaults to 25 seconds and is exactly the window readiness exists to cover.

Took your suggested shape: defer probes.NotReady("shutting down") immediately after Ready(). That covers every return by construction rather than by remembering to call it at each one.

Measured on the same duration-bounded run, before and after:

                 t+1..4      t+5 (deadline)    t+6..14 (scrape hold)
  pre-fix        running     running           running        <- never drops
  post-fix       running     shutting down     shutting down

/healthz answered 200 throughout in both, which is the half that has to keep working so the kubelet does not read a deliberate hold as a hang.

One thing worth stating plainly: the guard here is structural, not a test. A defer at the top of the scope covers every return path by construction, and this package has no harness that drives the run's lifecycle — main_test.go holds two unit tests and nothing that starts a run. The before/after measurement above is what stands in for one. If that trade is wrong I would rather add the harness than leave a claim resting on a comment.