Automate SSH Key Rotation With Ansible and Vault

SSH Key Rotation allows you to manage your Unix account private keys and passphrases as well as their passwords. With key rotation, whenever the password is changed on the secret (manually, during a scheduled auto-change, or when checking in a secret that changes the password on check-in), the public/private key pair will be regenerated and the private key encrypted using a new passphrase. The public key will then be updated on the Unix machine referenced on the secret.

Secret Server provides new secret templates and password changers for SSH Key Rotation.

How do we solve this?
Well you can write a bash script that is going to go through a list of hosts and then do an *ssh-copy-id* to each host, but this is not going to invalidate the old keys it is just going to dd the new ones. Only solving half of the issue. None the less for the initial setup of the solution this command is pretty nice so here it is.

ssh-copy-id -i ~/.ssh/mykey user@host

More at: https://www.ssh.com/ssh/copy-id

Soultion: Anisible
Ansible is nothing new, it has been around for a while but it is surprising the number of folks that do not use it. Here is some info for ansible http://docs.ansible.com/.
The feature that we are going to focus on is the authorized_key module http://docs.ansible.com/ansible/latest/authorized_key_module.html

Start Setup:

Needed are, SSH keys already setup on each of the hosts that you are going to change.
Ansible running on your laptop, which is where we are going to ssh from.
Ansible inventory of the hosts with the shared ssh key.
— for my lab Iam using a bunch of EC2 in AWS, because I like them and it makes me feel like a tiny mad scientist having instances running on AWS.

Here is an inventory example — simple enough:

[centos]
192.168.1.6 user=centos
192.168.1.7 user=centos
192.168.1.8 user=centos
192.168.1.9 user=centos

Ansible is going to generate a new ssh key at a different location than my primary key because right now I am labbing this out and don’t want to break all the things. It is easy enough to use your central location, once we are out of lab stage. A future implementation is going to have a KMS of sorts so I can securely store all the keys but right now lets do this insecurely and we can discuss the best practice later. Vault: https://www.vaultproject.io/docs/secrets/ssh/index.html

Great we have an inventory and our ssh key on each of the hosts. Lets say hello world to our hosts to make sure the lab is configured

ansible centos -a “/bin/echo hello aws” -i hosts -u pi — private-key=”~/.ssh/id_rsa

10.0.202.6 | SUCCESS | rc=0 >> hello aws
10.0.202.7 | SUCCESS | rc=0 >> hello aws
10.0.202.8 | SUCCESS | rc=0 >> hello aws
10.0.202.9 | SUCCESS | rc=0 >> hello aws

Great it worked, next lets change the key and hope to not break everything.
We are going to create an ansible playbook with a few tasks in it. First task is to generate a new ssh-key locally at our new location.

Playbook ansible:

name: “Set up authorized_keys for the root user”
hosts: centos
user: centos
tasks:
— name: Create new ssh key-pair
local_action: command ssh-keygen -t rsa -N “” -q -f
~/test/id_rsa

Next we will have a task that takes our new keyfile and pushes it to the hosts, using an exclusive property. The exclusive property is important because without it, we can retain the old keys. If we wanted to have multiple keys deployed we can do multiple steps, the first being the exclusive push and then appending our keys as we go without having exclusive set.

— name: Set up authorized_keys for the centos user
authorized_key: user=centos key=”{{ item }}” state=present exclusive=yes
with_file:
— ~/test/id_rsa.pub

Finally we have two steps to move the newly generated keys to our super secret archive, I have not worked this one out yet but cleanup is needed somewhere. Currently it is trying to run multiple times, and its a bug right now.

–name: move key-pair
local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak
run_once: true
— name: move key-pair
local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak
run_once: true

Putting it all together we get.
I realize that this formatting is terrible so here is a gist:

  • name: “Set up authorized_keys for the root user”
    hosts: centos
    user: centos
    tasks:
    — name: Create new ssh key-pair local_action: command ssh-keygen -t rsa -N “” -q -f ~/test/id_rsa
    — name: Set up authorized_keys for the centos user
    authorized_key: user=centos key=”{{ item }}” state=present exclusive=yes
    with_file:
    — ~/test/id_rsa.pub
  • name: move key-pair
    local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak
    run_once: true
  • name: move key-pair local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak
    run_once: true

Lets fire it up and change some keys!

$ ansible-playbook -i hosts playbook/ssh-key-push-revoke-old.yml
PLAY [Set up authorized_keys for the root user] ********************************
TASK [setup] *******************************************************************
ok: [10.0.202.6]
TASK [Create new ssh key-pair] *************************************************
changed: [10.0.202.6 -> localhost]
TASK [Set up authorized_keys for the pi user] **********************************
changed: [10.0.202.6] => (item=ssh-rsa xxxxx xxx@xxx)
TASK [move key-pair] ***********************************************************
changed: [192.168.1.9 -> localhost]
TASK [move key-pair] ***********************************************************
changed: [10.0.202.6 -> localhost]
to retry, use: — limit @playbook/ssh-key-push-revoke-old.retry
PLAY RECAP *********************************************************************
10.0.202.6 : ok=1 changed=1 unreachable=0 failed=0

Also you can check more links:

https://derpops.bike/2014/06/07/ssh-key-rotation-with-ansible/

Ansible docs:
http://docs.ansible.com/ansible/latest/playbooks_intro.html
http://docs.ansible.com/ansible/latest/authorized_key_module.html

Manual ssh-copy-id
https://www.ssh.com/ssh/copy-id