Goodput, not uptime

AUGUST 2026 · CARLOS ECKERT

My hands-on scale is three production clusters I administer solo, plus a summer on Mount Sinai’s Minerva across 356 H100 and A100 nodes. Most of my time goes to the layer that actually breaks: InfiniBand fabric faults, driver and NCCL failures, node-level hardware. My Kubernetes experience is real but not at 10k nodes, so I’ll be concrete about detection, attribution and remediation policy, and treat the orchestration layer as a set of requirements rather than claiming design experience I don’t have.

I’m also building a diagnostic tool for this problem class at smaller scale — it normalizes Slurm accounting, journald, DCGM and InfiniBand counters into one event stream keyed on (node, jobid, time) and classifies failure root cause under a default-deny action policy. It is early and nothing is shipped. Most of what follows is that architecture scaled up and moved onto a different scheduler.

The objective is goodput

The first move is refusing “auto-healing” as the objective. Healing is a mechanism; the objective is goodput, the fraction of wall-clock GPU time that produces training steps surviving into the final model. Goodput decomposes into losses you can attack independently: detection latency from fault to recognition, decision latency from recognition to correct action, replacement time to drain and schedule and warm and rejoin, rollback waste since the last checkpoint, the straggler tax paid to degraded-but-alive nodes, and the false-positive tax — work destroyed by healing actions that were wrong.

That last term matters more than people expect. At 10k H100s you’re burning roughly $20–30k an hour at market rates, so an automated system that cordons the wrong node and kills a run costs more than the failure it was chasing. Every decision below is a tradeoff between detection latency and false-positive rate, and being explicit about that tradeoff is what separates a real design from a monitoring wish list.

For calibration, Meta’s Llama 3 405B run on 16k H100s reported over 400 unexpected interruptions across 54 days, roughly one every three hours, overwhelmingly hardware-attributed with GPU issues the largest bucket. At 10k GPUs failure is the steady state. You are not preventing failures; you are minimizing the cost of each one.

Kubernetes fights synchronous training

Kubernetes makes this harder than it needs to be, because it was built for stateless replicas that fail independently and synchronous training is the inverse — stateful, gang-scheduled, all-reduce coupled, so one sick rank stalls 9,999 healthy ones. Several defaults are actively harmful. kubelet restarting a container is meaningless once the collective is already dead, so pod restart policy has to be subordinated to job-level restart. Default NotReady tolerations of five minutes are roughly $2k of idle fleet. The scheduler is topology-blind and will scatter a job across leaf switches and NVLink domains, which on rail-optimized fabrics is a large permanent throughput tax that looks like nothing on any dashboard. And eviction logic assumes pods are fungible, when ranks are not — a replacement has to land in the same topological position.

The requirements that follow are gang scheduling so a job either gets a coherent allocation or waits, topology-aware placement that understands rail and leaf and spine and NVLink domains, GPU-level telemetry wired into node conditions, and a job controller that owns restart policy rather than delegating it to kubelet. Volcano and Kueue are the obvious candidates for the first of those, and I’d want to evaluate rather than assert which one fits.

Single-tenancy is a genuine simplification here and I’d lean on it hard: no fairness constraints, no isolation boundary, no competing workloads, which means you can take aggressive automated action because there’s exactly one stakeholder and one workload to protect.

The best signal is in-band

Out-of-band telemetry tells you a node is broken; in-band telemetry tells you the training run is degraded, which is what you actually care about and which catches failure classes no node metric will ever surface. The primary signal is the per-rank step time distribution — every rank reporting step time and collective wait time, compared against the fleet median rather than a fixed threshold. A rank consistently 8% slow over 50 steps is sick regardless of what DCGM says, and that one signal catches thermal throttling, silent downclocking, a degraded NIC rail, a noisy storage path and PCIe link degradation, none of which reliably trip a hardware alert.

The critical corollary is that GPU utilization is a liar. A NCCL collective deadlock spins in a busy-wait loop at 100% utilization with zero forward progress, so any health system keyed on utilization will report a hung 10k-GPU run as perfectly healthy. Progress is steps committed, never utilization.

Read Xid semantically

Layered underneath that, the out-of-band signals need to be read semantically rather than as a single alarm. A system that treats “Xid fired” as one class will cordon healthy nodes for user bugs:

Beyond hard failures, the leading indicators matter more: ECC volatile counts trending, pending row remaps, retired pages, PCIe replay counters, thermal margin and clock-throttle reason bitmasks. Pending remap is the highest-value early signal available, because it predicts a fail-stop you can schedule around instead of absorb.

On the fabric side I’d watch per-port symbol errors, link-downed counters, recovery events, per-QP retransmits and congestion notifications. Link flaps are the classic gray failure, where NCCL silently falls back to a slower path, throughput drops 30%, and nothing alerts. That’s the class I’ve spent the most of my own debugging time on, and the one I’d most expect a naive system to miss entirely. Storage needs checkpoint write latency and dataloader stall time, which masquerade as compute problems constantly.

All of it correlates on (node, gpu, rank, jobid, time), and adding rank to that key is what connects hardware topology to the collective’s dependency graph.

Attribution first, then confidence tiers

Detection produces signals; the hard part is deciding what’s true and what to do about it. Attribution comes first, always — every incident classifies as hardware fault, fabric fault, storage fault or user code, because draining a node for a user’s OOM is expensive and erodes trust in the automation faster than anything else. From there, confidence tiers drive action authority:

Blast radius

Blast radius controls are non-negotiable, because this is where auto-healing systems cause the outages they were built to prevent. Remediation gets rate-limited to N nodes per hour, with a circuit breaker on drain rate: forty nodes failing at once is a PDU, a leaf switch, a bad driver rollout or a monitoring bug, and essentially never forty bad GPUs. Auto-healing correlated failures is how one rack outage becomes a fleet-wide cordon.

There’s a hard floor on healthy capacity so nothing drains below the job’s minimum viable size, and every action is reversible, dry-runnable and logged with the evidence that triggered it. The failure mode I’d design against isn’t a missed bad GPU; it’s an automation that eats the cluster at 3am.

Hot spares and the restart path

On the action side, hot spares are the single biggest lever. Hold 2–5% of capacity genuinely warm — image resident, container pulled, IB configured, NCCL topology mapped, validated within the last N hours — because warm spare versus cold node is 30 seconds versus 15 minutes, and at this scale that difference is the whole game. Spares have to be topology-aware, since a replacement landing outside the failed node’s rail and leaf position fixes availability and breaks throughput.

The restart path in preference order is in-job rank replacement, swapping a spare into the failed rank’s position without changing world size, because under 3D parallelism you can’t reshape cheaply and same-shape substitution is the only fast path; then elastic restart from memory, where surviving ranks hold state and only the replaced rank reloads; and only then checkpoint restart, which is the expensive fallback.

Checkpoint interval should follow from measured MTBF rather than instinct — async tiered checkpointing to local NVMe first and object store second, with the interval scaling as roughly the square root of checkpoint cost times MTBF. At a three-hour MTBF that’s far more frequent than teams typically choose.

Reintegration is the step everyone skips

A node coming back from repair is guilty until proven innocent, and a vendor’s “fixed” is not evidence. Before rejoining the spare pool it should run deep-level DCGM diagnostics, an HBM test, a thermal soak under sustained load, and most importantly a NCCL all-reduce bandwidth test against known-good peers — single-node health tells you nothing about the collective path, and the collective path is what failed.

Then verify the healing action itself: after replacement, confirm step time actually returned to baseline. If it didn’t, you drained the wrong node, which happens more than people admit, and without this check you never find out.

The learning loop

Underneath all of it runs a learning loop. Every incident writes a record of signature, attribution, action, outcome and whether the fix verified. Across one 10k-GPU campaign that’s hundreds of labeled incidents, and the classifier improves sharply within a single run. Keyed by serial number it also tells you which chassis fail repeatedly, whether failures cluster by rack or manufacturing batch, and which RMAs come back bad.

A single-tenant 10k fleet is the best possible environment to build this: enormous event volume, one workload shape, no cross-site privacy negotiation, and immediate ground truth on whether the fix worked.

Where LLM agents fit

Not in the fast path. Hard failures need sub-second deterministic decisions, and that’s policy code, not inference. Agents belong in post-incident synthesis, assembling the correlated bundle across DCGM, fabric counters, NCCL logs and rank timings into a readable root cause; in gray-failure triage, the ambiguous tier where a human currently spends 40 minutes reading counters; and in fleet-scale pattern discovery, surfacing that fourteen incidents over three weeks share a signature nobody named.

Anyone proposing an LLM in the remediation fast path at this scale hasn’t priced a wrong action.

What I’d build first

Not the healer. The detection and attribution layer, running in shadow mode with a human approving every remediation, for long enough to measure the false-positive rate against real incidents. At $20–30k an hour, the cost of finding that out in production is the entire argument for building it in this order.

You can’t know whether the automation is safe until you’ve watched it be wrong.