Skip to content

S3 outage

Trigger: readiness/degraded reason, S3 operation error, artifact/R2/snapshot timeout. Blast radius includes Worker bundle cold activation, R2, immutable backup, and full-platform snapshots. Already cached and verified objects may keep serving.

Read-only diagnosis:

sh
/opt/open-compute/ocd --config /etc/open-compute/config.toml doctor --json
/opt/open-compute/ocd --config /etc/open-compute/config.toml backup list --json

Allowed mutation: restore network or credential access to the same endpoint/region/bucket/prefix, then explicitly run doctor --full --json as the canary. Do not temporarily switch provider/prefix, and do not treat a failed upload as committed. Expect stable errors, bounded retries, and no premature manifest. Authority fingerprint change, missing bucket, short read, or checksum error are stop conditions. Rollback is restoring the original credential/config. Verification: S3 canary, immutable sample, R2 marker, and snapshot verify.