Client setup — SSH with step-ca certificates#

This page is a how-to for setting up your laptop or workstation to authenticate to CCAT hosts with a step-ca-issued SSH certificate. The short version of this is also covered in SSH Access for Operators (the operator-facing entry point); this document is the longer manual walkthrough that the ccat ssh setup automation collapses into one command. For the design context — why we chose this model, how the GitHub-team gate works, what cert lifetimes mean — see CCAT Certificate Authority — Architecture and Design.

Who this page is for

You’re a partner or new user whose source CIDR is already on the CCAT proxy allowlist (Uni Köln /16 by default). If it isn’t, the CCAT operator runs Adding a partner subnet in CA day-to-day operations first — you don’t. Off-network laptops have separate options below in Bootstrapping from off-network.

Tip

The entire setup below is automated by a single command:

pip install -e .   # if you haven't installed the ccat CLI yet
ccat ssh setup

It is idempotent — safe to run again if you want to update the renewal helper or re-issue a certificate. Run ccat ssh status afterwards to verify everything is in order. The manual steps below document what the command does under the hood.

This walkthrough takes about 10 minutes. After setup, your daily workflow is just ssh <host> — the certificate renews automatically when it expires.

Prerequisites#

  • Member of the ccatobs/datacenter GitHub team (Dex rejects everyone else at the authentication step).

  • A web browser on the machine you SSH from (the GitHub OAuth flow opens a browser tab).

Step 1 — Install step-cli#

You need the step binary from Smallstep:

# Ubuntu/Debian
wget https://dl.smallstep.com/gh-release/cli/docs-cli-install/v0.28.5/step-cli_0.28.5_amd64.deb && sudo dpkg -i step-cli_0.28.5_amd64.deb

# macOS
brew install step

# Other platforms: https://smallstep.com/docs/step-cli/installation/

Step 2 — Bootstrap against the CCAT CA#

(CCAT-ROOT-FINGERPRINT-CANONICAL)

This downloads the CA’s root certificate and saves it to /home/<user>/.step/:

step ca bootstrap --ca-url https://ca.ccat.uni-koeln.de --fingerprint da7e917de3ed65a373872374ed981bc46fd81aed609d531ca133a3c1369f6255

The fingerprint above is the HSM-backed CCAT root (Phase 2, established 2026-04-29). To verify against an independent source, check the ceremony record in the git log:

git log --grep="ca_trust: commit" --oneline

Verify the bootstrap worked:

step ca health

Should return ok.

Step 3 — Issue your first certificate#

mkdir -p ~/.step/ssh
step ssh certificate $(id -un) ~/.step/ssh/id_ccat --provisioner CCAT-GitHub --no-password --insecure --force

A browser tab opens, you authenticate with GitHub, Dex checks your team membership, and a 16-hour SSH certificate is written to disk. Three files appear:

  • ~/.step/ssh/id_ccat — private key (unencrypted, short-lived)

  • ~/.step/ssh/id_ccat.pub — public key

  • ~/.step/ssh/id_ccat-cert.pub — the signed certificate (16h)

Note

The private key is unencrypted on disk (--no-password --insecure). This is the right tradeoff: the cert expires in 16 hours, and an unencrypted key enables transparent renewal without password prompts. The real security boundary is the OIDC flow (GitHub team membership), not the key file. Your disk encryption (FDE) provides the at-rest protection layer.

Step 4 — Install the renewal helper#

The renewal helper is a 15-line script that checks cert freshness before each SSH connection and opens a browser for re-auth when the cert has expired. It lives in the system-integration repo:

mkdir -p ~/.local/bin
cp tools/step-ssh-renew-ccat ~/.local/bin/
chmod +x ~/.local/bin/step-ssh-renew-ccat

Make sure ~/.local/bin is on your PATH. Most Ubuntu/Debian systems include it by default; if not, add to your shell rc:

export PATH="$HOME/.local/bin:$PATH"

Step 5 — Configure SSH#

Add this to ~/.ssh/config. The Match exec block runs the renewal helper before every CCAT connection; the Host block sets up the jump host, identity ordering, and fallback.

For operators WITH a Nitrokey (Tier 1 admins):

# CCAT cert auto-renewal — opens browser if cert expired
Match host "input-?,input-?-staging,*.data.ccat.uni-koeln.de,*.staging.data.ccat.uni-koeln.de" exec "step-ssh-renew-ccat"
    IdentityFile ~/.step/ssh/id_ccat
    CertificateFile ~/.step/ssh/id_ccat-cert.pub

# CCAT hosts — cert first, Nitrokey fallback
Host input-? input-?-staging *.data.ccat.uni-koeln.de *.staging.data.ccat.uni-koeln.de
    IdentityFile ~/.ssh/id_ed25519_sk_datacenter
    IdentityFile ~/.ssh/id_ed25519_sk_datacenter_backup
    IdentitiesOnly yes

For operators WITHOUT a Nitrokey (Tier 2 staff):

# CCAT cert auto-renewal — opens browser if cert expired
Match host "input-?,input-?-staging,*.data.ccat.uni-koeln.de,*.staging.data.ccat.uni-koeln.de" exec "step-ssh-renew-ccat"
    IdentityFile ~/.step/ssh/id_ccat
    CertificateFile ~/.step/ssh/id_ccat-cert.pub

# CCAT hosts — cert only (no hardware fallback)
Host input-? input-?-staging *.data.ccat.uni-koeln.de *.staging.data.ccat.uni-koeln.de
    IdentitiesOnly yes

How the config works:

  1. Match exec fires first. If the cert is valid, the Match block applies and its IdentityFile + CertificateFile are added to the identity list. If the cert is expired, the browser opens for renewal; if renewal succeeds, same result. If renewal fails (CA down), the Match block is skipped entirely.

  2. Host block always applies. It adds IdentitiesOnly yes and for Tier 1 admins the Nitrokey IdentityFile entries. (User, ProxyJump, and HostName come from your own existing config blocks — this block handles identity/cert only.)

  3. OpenSSH offers identities in order: cert first (from Match), then Nitrokeys (from Host). If the cert was accepted, done. If not (expired, CA down), Nitrokey takes over — the dongle blinks, you tap it, you’re in.

  4. IdentitiesOnly yes ensures only these 1-3 identities are offered. The other 10+ keys in your agent are never sent. No fail2ban triggers, no MaxAuthTries exhaustion.

Step 6 — Test it#

ssh -v input-a 2>&1 | grep -iE 'offering|accepted|authent'

You should see the cert offered first and accepted. If the cert has expired, a browser tab opens first, you click authorize, then the connection proceeds.

Renewal and expiry#

Certs expire after 16 hours. This is by design — see CCAT Certificate Authority — Architecture and Design § “Why these lifetimes” for the rationale. You never need to think about it:

  • Cert valid: ssh input-a connects instantly. No browser, no prompt, no delay.

  • Cert expired: ssh input-a opens a browser tab for GitHub OAuth (~3 seconds), then connects. You see the browser — that’s your awareness signal that the infrastructure is working.

  • CA down (Tier 1 admin): ssh input-a skips the cert, offers your Nitrokey instead. Dongle blinks. Report the CA failure to the team.

  • CA down (Tier 2 staff): ssh input-a fails with “Permission denied.” Contact an admin. This is the intended failure mode — the CA being down is an incident, and Tier 2 staff not being able to SSH is how the team notices.

Bootstrapping from off-network#

The CA vhost (ca.ccat.uni-koeln.de) is locked to Uni Köln /16 at the proxy (proxy/data/vhost.d/ca.ccat.uni-koeln.de). If you hit the bootstrap from a non-uni IP you’ll see a TLS handshake that never completes (the proxy returned 403 Forbidden before negotiating). There are two supported options:

  1. Ask a CCAT operator to add your CIDR to the allowlist. Best choice for stable workstations. The operator follows Adding a partner subnet in CA day-to-day operations.

  2. Tunnel the bootstrap through hera. Hera is on Uni Köln, so traffic exiting hera reaches the proxy from an allowlisted IP. Open a local-forward in one shell:

    ssh -N -L 8443:ca.ccat.uni-koeln.de:443 hera.uni-koeln.de

    Then in another shell, point step at the local end and pass the real SNI hostname so the proxy serves the CA vhost:

    step ca bootstrap \
    --ca-url https://ca.ccat.uni-koeln.de:8443 \
    --fingerprint da7e917de3ed65a373872374ed981bc46fd81aed609d531ca133a3c1369f6255

    Once the root is installed locally you can drop the tunnel; daily ssh works through hera as a normal ProxyJump.

There is no client-side trust-bundle workaround. Off-uni clients are stopped at the proxy IP allowlist, before TLS ever negotiates, so no trust trick on your end can help.

Optional: cert auth through hera#

Hera is a university jump host used as a TCP tunnel via ProxyJump. Cert auth on hera is not required — ProxyJump uses -W mode, which forwards the connection without authenticating to hera in any special way beyond your normal credentials.

If you want cert auth on hera too (so the step-ca cert works for the jump as well), add the CCAT SSH user CA to your authorized_keys on hera:

# Get the CA public key from the repo
cat ansible/roles/ca_trust/files/ssh_user_ca.pub

# On hera, append to your authorized_keys (replace <github-username>):
echo 'cert-authority,principals="<github-username>" <paste the whole ssh_user_ca.pub line>' >> ~/.ssh/authorized_keys

Warning

The principals="<github-username>" restriction is mandatory. Without it, any cert signed by the CCAT user CA — for any principal — could log in as you on hera. Always restrict to your own principal.

Troubleshooting#

x509: certificate signed by unknown authority after bootstrap

The CA vhost (ca.ccat.uni-koeln.de) is fronted by nginx-proxy with a CCAT-rooted cert. If you see this error from a uni-Köln IP the proxy is serving the wrong cert — verify what it’s presenting:

echo | openssl s_client -connect ca.ccat.uni-koeln.de:443 -servername ca.ccat.uni-koeln.de 2>/dev/null | openssl x509 -noout -issuer
# Expect: issuer=O=CCAT Observatory, CN=CCAT Intermediate CA

If the issuer is Let's Encrypt, the renewal/re-issue runbook in CA rotation and disaster recovery § “Vhost cert emergency re-issue” needs to run on input-b. From off-uni you’ll see the same TLS error because the proxy 403s your request before negotiating — see Bootstrapping from off-network above.

Browser opens but Dex says “access denied”

You are not in the ccatobs/datacenter GitHub team. Ask a team maintainer to add you at https://github.com/orgs/ccatobs/teams/datacenter/members.

Cert issued but SSH says “Permission denied”

Check principals: ssh-keygen -L -f ~/.step/ssh/id_ccat-cert.pub | grep Principals. Your GitHub username must be listed. Then check the target host has ca_trust applied — specifically that /etc/ssh/auth_principals/<your-username> exists and contains your GitHub username.

SSH works but offers Nitrokey first instead of cert

The Match exec block must appear BEFORE the Host block in your ~/.ssh/config. OpenSSH applies blocks in order; if the Host block’s IdentityFile entries are seen before the Match block’s, they’re offered first.

step-ssh-renew-ccat: command not found

The script is not on your PATH. Either move it to a directory that is (e.g. /usr/local/bin/), or add ~/.local/bin to your PATH in your shell rc.

For deeper troubleshooting (CA-side issues, provisioner errors, rotation procedures), see CA day-to-day operations and CA provisioner management.

See also#