Certificate authority ceremonies

This section is the canonical, executable record of every CCAT root-CA ceremony — the procedures, paper artefacts, threat model, and post-ceremony lessons. The documents here are loaded onto the supplies USB by step-ca/prepare-ceremony-usb.sh so the air-gapped ceremony laptop has the same content this Sphinx build does.

When to run a ceremony

• Phase 2 commissioning — the original root + intermediate setup before any CCAT client trusts anything (this is what the playbook below is written for; first run was 2026-04-29).

• Root rotation — every ~10 years, or sooner if the threat model changes. The playbook is reusable; only PINs and HSM serials change.

• Post-incident re-key — if HSM #1 (root) leaves the safe under uncontrolled conditions, treat it as compromised and rerun the ceremony with fresh HSMs.

The documents

• CCAT CA — Ceremony Supplies USB
  • Contents
  • What to do first, at ceremony time
  • Physical discipline reminder
  • After the ceremony
  • Provenance
• Ceremony Live USB — Preparation Guide
  • Why two USBs instead of one persistent stick
  • Threat model this setup addresses
  • USB #1 — Ubuntu LTS Live (boot media)
  • USB #2 — Ceremony supplies
  • Ceremony-time verification
  • Pre-ceremony physical discipline
  • Post-ceremony hygiene
  • Updating pinned versions
• CCAT CA — Offline Root Ceremony Playbook
  • §0 — Before you sit down
  • §1 — Boot, air-gap, sanity-check
  • §2 — Plug in supplies USB and verify the manifest
  • §3 — Install software from the supplies USB
  • §4 — Verify each HSM serial against its masking-tape label
• Phase A — HSM #1 (root) operations
  • §5 — Initialise HSM #1 as the root
  • §6 — Generate the root key on HSM #1
  • §7 — Create the self-signed root certificate
• Phase B — HSM #2 (intermediate) operations
  • §8 — Initialise HSM #2 as the intermediate
  • §9 — Generate the intermediate key on HSM #2
  • §10 — Create the intermediate CSR
  • §11 — Generate SSH user/host CA keys on HSM #2
• Phase C — back to HSM #1 to cross-sign
  • §12 — Sign the intermediate CSR with the root
• Phase D — paper finalisation (no HSM)
  • §13 — Record the root fingerprint on paper
  • §14 — Copy public artefacts to the export USB
  • §15 — Unplug HSMs, label, power off
  • §16 — Hand-off
  • If something goes wrong mid-ceremony
• Lessons learned — root ceremony 2026-04-29
  • 1. OpenSC virtual-token quirk: the label gets renamed
  • 2. step certificate create URI shape
  • 3. step certificate create reflex password prompt
  • 4. step certificate sign argument order
  • 5. step certificate sign writes to stdout
  • 6. SSH CA pubkey export — version-portable conversion
  • 7. Stick-swap gate procedure
  • 8. Phase grouping reduces 7 swaps to 2
  • 9. Read-aloud rule scoped to non-secrets only
  • 10. PIN format, length, and retry counters
  • 11. PKCS#11 slot indices are non-stable
  • 12. Single USB at install vs three USBs at ceremony
  • What the playbook does not yet capture
• CCAT CA — HSM Cutover Playbook (post-ceremony)
  • §0 — Preconditions
  • Stage A — Verify HSM #2 is functional on input-b
  • Stage B — Commit ceremony artefacts to the repo
  • Stage C — Vault the intermediate user PIN
  • Stage D — Build a step-ca image with opensc-pkcs11
  • Stage E — Compose changes for HSM-backed mode
  • Stage F — Write the HSM-aware ca.json
  • Stage G — The cutover (downtime window, ~5 minutes)
  • Stage H — Test cohort re-bootstraps
  • §I — After the cutover settles
  • Roll-back / what-if
  • Appendix — files this playbook touches
• Lessons learned — Phase 2 HSM cutover 2026-05-04
  • 1. The architectural multi-pivot
  • 2. step-cli 0.30.2 has no working offline provisioner add
  • 3. enableSSHCA defaults off in Phase 2
  • 4. Volume pre-population needs a running helper container
  • 5. Host pcscd must be stopped post-cutover
  • 6. Plumbing bugs that ate real hours
  • 7. Verifying HSM-backed signing without physical access
  • 8. Open follow-ups (not blocking; tracked as GitHub issues)

Document	Audience	Purpose
USB readme	Operator at ceremony start	“Start here” entry point printed on the supplies USB.
Live USB setup	Prep operator (pre-ceremony)	How to build the boot + supplies USBs, threat model, hash-verify the manifest.
Playbook	Operator + witness during ceremony	Numbered, executable steps §0 through §16. The single source of truth.
Lessons learned 2026-04-29	Future ceremony operators	Post-ceremony retrospective: PKCS#11 URI shape, OpenSC virtual-token quirk, step-cli argument-order issues, and other surprises that the playbook now bakes in.
Cutover playbook	Operator + witness during HSM cutover	Stages A–H executable for the post-ceremony cutover on input-b: HSM verification, root rotation, compose changes, volume bootstrap, provisioner sync, test cohort re-bootstrap.
Lessons learned cutover 2026-05-04	Future cutover operators	Phase 2 cutover retrospective: the libusb-vs-pcscd architectural multi-pivot, step-cli’s missing offline mode, host pcscd contention, the udev/GID alignment dance, and the plumbing bugs that ate hours.

Adjacent documents (background, not procedure)

• Certificate authority architecture — long-form explanation of the trust model, two-HSM layout, GitHub-team gate, lifetimes, and Pattern A. The Phase 2 ceremony itself is no longer narrated here; the executable procedure lives in playbook.md and cutover-playbook.md.

• CCAT provisioner set — reference tables for the six provisioners, SSH access tiers, and the Ansible role tags exposed by ca_trust and hsm_host.

• CA day-to-day operations, CA provisioner management, CA rotation and recovery — operator how-to guides for routine work, provisioner changes, and rotation/DR scenarios.

• Certificate authority threat model — the broader threat-modelling document this ceremony’s discipline derives from.

• TLS and PKI primer — vocabulary refresher.

Paper artefacts

The PIN sheet (step-ca/ceremony-pin-sheet.tex → .pdf) is the paper companion the operator fills in by hand during the ceremony. It captures the four PINs, two HSM serials, and the root-CA fingerprint twice. The LaTeX source is committed; the rendered PDF should be printed on a trusted printer immediately before the ceremony.

To rebuild the PDF from a checkout:

cd step-ca && pdflatex ceremony-pin-sheet.tex && pdflatex ceremony-pin-sheet.tex

(Twice, so the LastPage reference resolves.)