Phase A — HSM #1 (root) operations#

The next three steps (§§5–7) all run with HSM #1 plugged in. Do not unplug between them. Set up a working directory in RAM that all ceremony output files will land in:

mkdir -p ~/ceremony && cd ~/ceremony

PKCS#11 URI shape for step-cli (verified 2026-04-29).

The commands in §§7, 10, 11, 12 use this URI structure:

--kms "pkcs11:module-path=...;serial=$SERIAL?pin-value=$PIN" — module path + serial of the currently-plugged HSM (captured at the gate) + the user PIN inline.

--key "pkcs11:id=01" — minimal object selector. Adding object= or token= to --key made step’s getPublicKey URI matcher reject the lookup (“uri not matching”) in our run.

Why serial= instead of token=?

The opensc-pkcs11.so driver presents one PKCS#11 token per PIN type, and renames it: when you init with --label "ccat-root" the URI-layer token name becomes ccat-root (UserPIN), which has to be percent-encoded (ccat-root%20%28UserPIN%29) to feed step. Stable across re-inits, no encoding gymnastics, already on the paper sheet — serial=$SERIAL (e.g.\ serial=DENK0107520) is just cleaner. The token form still works if you prefer it; both identifiers point at the same physical card.

Why pin-value and not pin-source? URI-embedded pin-source=/path/to/file did not resolve reliably in this run. pin-value=$PIN (with $PIN captured via read -rs PIN so the literal value never lands in shell history) was the form that consistently let step log into the token.

§5 — Initialise HSM #1 as the root#

sc-hsm-tool --initialize --label "ccat-root" \
    --so-pin <ROOT-SO-PIN> --pin <ROOT-USER-PIN>

Expect a “Token initialized” success line. If it errors, stop and do not continue. A wrongly-initialised HSM is recoverable, but you do not want to compose ceremony state under uncertainty.

§6 — Generate the root key on HSM #1#

pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so \
    --token-label ccat-root --login --pin <ROOT-USER-PIN> \
    --keypairgen --key-type EC:secp384r1 \
    --label "ccat-root-key" --id 01

Expect “Key pair generated” output. The private key is now on HSM #1 and will never leave it.

§7 — Create the self-signed root certificate#

The --kms URI tells step to use the HSM key by reference; the certificate is public output, the private key stays inside HSM #1.

Working URI shape (verified 2026-04-29 ceremony). After extensive trial and error against step-cli + the OpenSC virtual-token model, the following two-part URI structure is what step’s PKCS#11 driver actually accepts on Ubuntu 24.04 Live:

--kms carries module-path + token (or serial) + pin-value (in-line, not pin-source — the latter did not resolve reliably in our run).

--key is minimal — just pkcs11:id=01. Adding object= or token= to --key made step’s getPublicKey URI matcher fail.

The token form token=ccat-root%20%28UserPIN%29 (URL-encoded ccat-root (UserPIN)) works. So does serial=<DENK0...> if you recorded the serial in §4 — that variant is more robust to label changes but requires the operator to type the serial.

Read the root user PIN into a shell variable so it’s not in your shell history (read -rs reads silently; the URI gets the PIN via shell expansion at run time):

read -rs PIN

step certificate create \
    --profile root-ca \
    --not-after 240960h \
    --no-password --insecure \
    --kms "pkcs11:module-path=/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so;serial=$SERIAL?pin-value=$PIN" \
    --key "pkcs11:id=01" \
    "CCAT Observatory Root CA" \
    root_ca.crt
unset PIN

--no-password --insecure suppresses step’s reflex prompt for an output-file password. Since we’re using an existing HSM key, no private-key file is written to disk; the cert itself is public PEM that doesn’t need a passphrase. --insecure is just step’s required acknowledgement that the operator intentionally chose --no-password.

step certificate create takes 3 positionals <subject> <crt-file> <key-file>, but the <key-file> slot is for writing a freshly-generated key. To use an existing HSM key, set --key <pkcs11:URI> and omit the third positional. --kms carries the token + PIN context; --key selects the object by id.

240960h ≈ 27.5 years. The lifetime should outlive the next planned rotation; rotation is an event, not a schedule.

Witness verifies the file appeared:

ls -la root_ca.crt
step certificate inspect root_ca.crt --short

Expect Subject: CN=CCAT Observatory Root CA, Issuer: same, validity window ~27y from today.

Phase B — HSM #2 (intermediate) operations#

Gate: only HSM #2 (intermediate) plugged in. Apply gate procedure (§4 sidebar). The next four steps (§§8–11) all run against HSM #2. Do not unplug between them.

§8 — Initialise HSM #2 as the intermediate#

sc-hsm-tool --initialize --label "ccat-intermediate" \
    --so-pin <INT-SO-PIN> --pin <INT-USER-PIN>

Same expectation as §5. Witness confirms the success message.

§9 — Generate the intermediate key on HSM #2#

pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so \
    --token-label ccat-intermediate --login --pin <INT-USER-PIN> \
    --keypairgen --key-type EC:secp384r1 \
    --label "ccat-intermediate-key" --id 01

§10 — Create the intermediate CSR#

Read the intermediate user PIN (a different secret from the root user PIN used in §7):

read -rs PIN

step certificate create \
    --csr \
    --no-password --insecure \
    --kms "pkcs11:module-path=/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so;serial=$SERIAL?pin-value=$PIN" \
    --key "pkcs11:id=01" \
    "CCAT Observatory Intermediate CA" \
    intermediate.csr

--csr switches the output from a self-signed certificate to a CSR. Same --kms + --key pattern as §7, just pointing at HSM #2 and the intermediate key generated in §9. Token name is ccat-intermediate (UserPIN) (URL-encoded). Keep $PIN in the shell variable for §11 below — it uses the same intermediate user PIN; do not unset PIN until after §11 is done.

step certificate inspect intermediate.csr --short

Expect a CSR with subject CN=CCAT Observatory Intermediate CA and an EC P-384 public key.

§11 — Generate SSH user/host CA keys on HSM #2#

step-ca uses separate keys for SSH certificates (distinct from the X.509 chain). Generate both on HSM #2 so step-ca can sign SSH certs at runtime:

pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so \
    --token-label ccat-intermediate --login --pin <INT-USER-PIN> \
    --keypairgen --key-type EC:secp384r1 \
    --label "ccat-ssh-user-ca-key" --id 02

pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so \
    --token-label ccat-intermediate --login --pin <INT-USER-PIN> \
    --keypairgen --key-type EC:secp384r1 \
    --label "ccat-ssh-host-ca-key" --id 03

Export the public parts in OpenSSH format:

(reuses the $PIN from §10 — the same intermediate user PIN.)

step-kms-plugin key outputs the public key in PKCS#8 PEM by default. We convert to OpenSSH wire format via step crypto key format --ssh, which is part of step-cli (already on the supplies USB):

step-kms-plugin key "pkcs11:module-path=/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so;serial=$SERIAL;id=02?pin-value=$PIN" > /tmp/.ssh_user_ca.pem
step crypto key format --ssh /tmp/.ssh_user_ca.pem > ssh_user_ca.pub
rm /tmp/.ssh_user_ca.pem

step-kms-plugin key "pkcs11:module-path=/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so;serial=$SERIAL;id=03?pin-value=$PIN" > /tmp/.ssh_host_ca.pem
step crypto key format --ssh /tmp/.ssh_host_ca.pem > ssh_host_ca.pub
rm /tmp/.ssh_host_ca.pem

Sanity-check format:

head -c 30 ssh_user_ca.pub
head -c 30 ssh_host_ca.pub

Both should start with ecdsa-sha2-nistp384 ….

Fallback if step crypto key format --ssh is missing in this step-cli version: replace the conversion line with ssh-keygen -i -m PKCS8 -f /tmp/.ssh_*_ca.pem > ssh_*_ca.pub. Same result, uses standard OpenSSH instead of step-cli.

If step-kms-plugin key --help shows --format ssh (or --ssh) in newer releases, you can collapse each conversion to a single command — but the two-step PEM→ssh path above works on every version and matches the rest of the ceremony’s belt-and-suspenders style.

After both .pub files are written, clear the intermediate PIN before the swap to HSM #1:

unset PIN

Phase C — back to HSM #1 to cross-sign#

Gate: only HSM #1 (root) plugged in. Apply gate procedure. This is the final HSM swap. §12 below uses HSM #1 to sign the intermediate CSR; §13 (fingerprint) needs no HSM.

§12 — Sign the intermediate CSR with the root#

Read the root user PIN back into a shell variable (the intermediate PIN was unset at the end of §11):

read -rs PIN

step certificate sign \
    --profile intermediate-ca \
    --not-after 87600h \
    --kms "pkcs11:module-path=/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so;serial=$SERIAL?pin-value=$PIN" \
    intermediate.csr \
    root_ca.crt \
    "pkcs11:id=01" \
    > intermediate_ca.crt
unset PIN

(step certificate sign writes the cert to stdout — no output-file password prompt, so --no-password --insecure aren’t needed here.)

For step certificate sign the three positionals are <csr> <issuing-ca-cert> <issuing-ca-key> — so root_ca.crt is the signing certificate (used to derive issuer info), not an output, and the third positional is the root-key URI on HSM #1. We use the same minimal pkcs11:id=01 form that worked in §7.

Argument order matters. This step-cli version requires all flags before all positionals. With positionals first the parser rejected the cert positional with scheme is missing (it was trying to apply URI parsing to it because a --kms argument followed). Putting --kms and the other flags before intermediate.csr resolves that. A bare relative path root_ca.crt is correct — do not add a file:// prefix; that errors out with scheme not expected.

87600h = 10 years.

Verify the chain:

step certificate verify intermediate_ca.crt --roots root_ca.crt
step certificate inspect intermediate_ca.crt --short

The verify command should print nothing (success). Inspect should show issuer = root subject.

Phase D — paper finalisation (no HSM)#

§13 — Record the root fingerprint on paper#

This is the canonical value every CCAT client will check at step ca bootstrap. If this paper is lost, recovery is painful.

step certificate fingerprint root_ca.crt

Operator transcribes the fingerprint from the screen onto the paper — twice, on two different sheets, side by side. Witness visually verifies each character on the screen against each character on the paper, on both sheets. Do not read the fingerprint aloud — same audio-leak hygiene as PINs (ambient mics in phones / watches / voice assistants). The fingerprint itself becomes public information once clients deploy, but visual verification against the screen is strictly stronger than aural confirmation (which only validates what the operator typed, not what is actually on screen vs paper), and keeping a uniform “all values are checked silently” rule prevents the operator developing speak-aloud habits that would leak the PIN/SO-PIN values on the same sheet. The serial check rule (spoken aloud) is the deliberate exception, scoped to non-secret device IDs. Both papers go in the envelope along with HSM #1 and the PIN sheet.

Also save it to the export USB later (§14) as a tamper-evident cross-check.

§14 — Copy public artefacts to the export USB#

Plug in the export USB (“CCAT CA EXPORT”). Do not unplug the supplies USB yet — the manifest from §2 is your audit trail.

EXPORT=/media/ubuntu/CCATEXPORT   # adjust to the actual mount-point
mkdir -p $EXPORT/ceremony-$(date +%Y-%m-%d)
cd ~/ceremony

cp root_ca.crt           $EXPORT/ceremony-$(date +%Y-%m-%d)/
cp intermediate_ca.crt   $EXPORT/ceremony-$(date +%Y-%m-%d)/
cp ssh_user_ca.pub       $EXPORT/ceremony-$(date +%Y-%m-%d)/
cp ssh_host_ca.pub       $EXPORT/ceremony-$(date +%Y-%m-%d)/
step certificate fingerprint root_ca.crt > $EXPORT/ceremony-$(date +%Y-%m-%d)/FINGERPRINT.txt

ls -la $EXPORT/ceremony-$(date +%Y-%m-%d)/
sync
umount $EXPORT

Five files must be on the export USB:

root_ca.crt
intermediate_ca.crt
ssh_user_ca.pub
ssh_host_ca.pub
FINGERPRINT.txt

There is no private file to copy. The private keys are inside the HSMs. If you find yourself reaching for an intermediate_key.pem, stop — something has gone catastrophically wrong.

Witness compares FINGERPRINT.txt content against the paper from §13, character by character. Equal? Good.

§15 — Unplug HSMs, label, power off#

Unplug HSM #1 first. Apply a permanent label sticker: CCAT root — DO NOT INSTALL — YYYY-MM-DD. Place it in the envelope with the root PIN paper and the fingerprint paper.

Unplug HSM #2. Label: CCAT intermediate — input-b — YYYY-MM-DD. Place in a small sealed bag for transport to the server room.

Unplug the supplies USB. Do not plug it into a networked machine afterwards — the manifest in §2 was the tamper-detection control; that seal can only be checked again before another ceremony, not after casual reuse.

Power off the laptop normally (Power → Shut Down, not Suspend). After it’s off, leave it for ~30 seconds before removing the boot USB, to be sure nothing is still flushing to it.

§16 — Hand-off#

The ceremony is done. The post-ceremony cutover happens elsewhere on the network. The next operator picks up at docs/COMMISSIONING-TODO.md § “HSM cutover on input-b” and the executable docs/source/ceremony/cutover-playbook.md. They will need:

HSM #2 (intermediate, in its labelled bag)
The intermediate user PIN (memorised by, or shared with, the cutover operator — not the root PINs, which stay in the safe)
The export USB
The fingerprint paper, for cross-checking the committed root_ca.crt in ansible/roles/ca_trust/files/ and the step ca bootstrap --fingerprint <…> value the test cohort will run

The boot USB and the supplies USB are now ceremony-archive media. Put them in the safe with HSM #1, separately from the root PINs.

If something goes wrong mid-ceremony#

Before §5 (HSM init): stop, power off, retry on a different day. Nothing has been written to HSMs yet.
Between §5 and §12: the HSMs hold partial state. You can re-sc-hsm-tool --initialize an HSM (this wipes it) and start over from §5 — but only do this with the witness in agreement, and write down what happened. If you wipe HSM #1 you must also discard root_ca.crt from §7 if it was already created; if you wipe HSM #2 you must discard intermediate.csr and ssh_*_ca.pub from §§10–11.
After §12 (root has signed the intermediate): if the signed output is bad, do not retry by re-initialising HSM #1 unless you also agree the root cert from §7 is to be discarded. The whole ceremony is atomic; you either commit to the root cert from §7 or you wipe both HSMs and rerun from §5 with new PINs.
At any point: if you suspect the ceremony has been observed (someone walks in, an unexpected light blinks, a USB device is inserted that you did not authorise) — stop, power off, treat the HSMs as compromised, replan with fresh PINs and fresh HSMs.

CCAT CA — Offline Root Ceremony Playbook#

§0 — Before you sit down#

§1 — Boot, air-gap, sanity-check#

§2 — Plug in supplies USB and verify the manifest#

§3 — Install software from the supplies USB#

§4 — Verify each HSM serial against its masking-tape label#