SSH Key Rotation Without the Lockout Risk
Why rotation keeps getting deferred, how to make it fail safely, and what post-quantum key exchange changes about the job
Why Rotation Gets Deferred
Almost every organisation knows its SSH keys are older than they should be. Keys outlive the people who generated them, get copied between laptops, and sit in authorized_keys long after the contractor who owned them moved on. Rotation is on everyone's list.
It stays on the list because the failure mode is so unforgiving. Get it wrong and you do not get a failed task and a retry: you get a host you cannot log into. Recovery means console access, a rescue image, or a support ticket with your cloud provider, per host, at the worst possible time. Against that, an old key that still works does not feel like an emergency.
The way out is not more care. It is a process where the worst realistic mistake leaves you exactly where you started, still holding a working key.
A Rotation That Fails Safely
The single rule that makes rotation survivable: the old key is never removed until the new key has proved it can authenticate. Everything else follows from that. Split into three phases, a mistake at any point fails safely instead of leaving a host unreachable.
Phase 0: Validate
Runs entirely on the control node, before a single host is touched. Checks the required variables are set, the new key is a recognised and strong type, and the new private key genuinely matches the public key file. A typo in a path fails here, not halfway through the fleet.
Phase 1: Install
Connects with the old key and installs the new one alongside it. Both keys now work. Public key authentication is confirmed as enabled. Nothing has been taken away, so there is nothing yet to recover from.
Phase 2: Verify
Reconnects using the new key to prove it works, and only then removes the old key and disables legacy authentication. If the reconnection fails, the run stops and makes no further changes to that host.
Two further habits matter as much as the phases. Apply configuration changes with an sshd reload rather than a restart, so existing sessions survive the change. And validate every sshd_config edit with sshd -t before writing it, keeping a backup of both sshd_config and authorized_keys, so any change can be reverted by hand from the console if it comes to that.
What Silently Goes Wrong
The phases protect you from lockout. They do not, on their own, protect you from a rotation that reports success while leaving the host less secure than you think. These are the failures worth designing against, because none of them announce themselves.
Drop-in files override your edits
Modern distributions read /etc/ssh/sshd_config.d/*.conf, and on many cloud images the first match wins over anything written further down the main file. Ubuntu ships 50-cloud-init.conf, which on 26.04 sets PasswordAuthentication yes. Edit sshd_config, reload, see no error, and password login is still enabled.
Match blocks re-enable what you disabled
A Match User or Match Address block further down the config can re-enable password authentication for exactly the account you just locked down. A global check passes; the account you care about is still reachable by password.
Success is assumed rather than verified
The fix for both of the above is to stop trusting the write. After disabling password and keyboard-interactive authentication, re-read the effective configuration with sshd -T and fail the run if either is still enabled, checking it for the rotated user specifically rather than globally.
Crypto-policy rejects your key type
On RHEL-family hosts running the FIPS crypto-policy, ssh-ed25519 is not accepted. A perfectly good ed25519 key is refused for reasons that have nothing to do with the key. Use ECDSA or RSA on those hosts, and catch it in validation rather than mid-rollout.
Weak keys pass unnoticed
Rotation is the moment to raise the floor, not to carry the old standard forward. Deprecated key types, weak types, and undersized RSA keys should be rejected before anything is rotated, not discovered at the next audit.
The blast radius is the whole fleet
The same automation that rotates one host rotates a thousand. Test against a disposable host with a snapshot you can roll back before pointing it anywhere that matters, and stage the rollout so a surprise costs you one batch rather than the estate.
Post-Quantum: What Actually Changes
The reason SSH key rotation has moved up the agenda is not that keys got weaker. It is harvest now, decrypt later: traffic captured today can be stored until a cryptographically relevant quantum computer exists, and then decrypted retrospectively. For anything with a long confidentiality lifetime, the exposure starts when the traffic is captured, not when the quantum computer arrives. That makes key exchange the urgent half of the problem, ahead of signatures.
The practical difficulty is that post-quantum SSH is not one setting. Both ends of the connection need matching algorithm lists across several independent layers, and missing one produces a connection that quietly negotiates down to classical algorithms while everything appears to work:
On the target
KexAlgorithmsinsshd_configPubkeyAcceptedAlgorithmsHostKeyAlgorithmsCASignatureAlgorithms, for SSH CA setups
System-wide
- RHEL and Fedora crypto-policy
DEFAULT:PQorFIPS:PQsubpolicy modules- Policy applied on targets and control node alike
On the client
- The control node's own ssh client
- Matching algorithms on outbound connections
- Otherwise the automation itself cannot connect
Distribution behaviour differs in ways that catch people out. On RHEL-family 9.x the FIPS policy has no post-quantum key exchange at all and needs the PQ subpolicy module added; on 10.x it is already included. Write directives with OpenSSH's +algorithm syntax so you append to the compiled-in defaults rather than replacing them outright, and clients that do not speak post-quantum yet can still fall back to a classical algorithm instead of failing to connect.
Treat post-quantum as opt-in and staged. Enable it where the traffic justifies it, verify it actually took effect on the wire rather than trusting the config, and keep a rollback path.
The Collection We Built for This
Everything above is implemented in ssh_key_rotation, an MIT-licensed Ansible collection we publish and maintain. It came out of client work, was useful enough to generalise, and is free to use.
It runs the three phases described here, validates and backs up every sshd_config edit, reloads rather than restarts, warns about drop-in files and verifies the lock-down applies to the rotated user specifically, and supports post-quantum and hybrid negotiation across all the layers listed above as an opt-in. It has been run end to end against Ubuntu 22.04 and 26.04, Rocky Linux 9, AlmaLinux 9 and 10 under FIPS, and openSUSE Leap 15.6, with the distribution-specific gotchas documented rather than glossed over.
As with anything you download, test it against a disposable host before running it anywhere that matters. A VM snapshot you can roll back is ideal.
Planning a Rotation Across a Real Estate?
Doing this on one host is a morning. Doing it across a mixed fleet, with FIPS hosts, cloud images that fight you, and a post-quantum migration to plan, is a different exercise. That is the part we help with, and we offer ongoing support for the collection once it is in place.
Talk to Us