A while ago it was requested that developers should be able to login, upon request, on our production servers via SSH. Normally this is restricted to SysOps and DevOps operators.
Developers have their DTA(P) to do their thing, but nevertheless management decided that they need CLI access to our live systems. Needless to say, we need to secure this and set it up so that it remains manageable.
I’m happy to say this can be done with the standard SSH tools. This can be done on any Linux distro, as long as you have SSH key based access configured.
With SSH CA signed keys we can give temporary access to our live systems.
At the production server we’ve got the SSH keys stashed of every developer in the /root/.keys directory. We’re deploying this and keeping this up-to-date in Ansible (and we’ve got some light Ansible content as well). You can put them anywhere on the server, just take care its secure (root:root 0400) and Ansible isn’t a must.
When these public keys are in place, we can configure the CA keys, per server:
# cd /etc/ssh # ssh-keygen -f ssh_ca -t rsa -b 8192 -C firstname.lastname@example.org # chmod 400 ssh_ca*
As you can see, we’re putting them in /etc/ssh. The above command creates the ssh_ca file (private key) and the ssh_ca.pub file (public key).
We will add the public key to a normal users authorized_keys file, which is the same user the project is running as (as the user ‘project’, and we’re serving files from this users home directory):
# vi /home/project/.ssh/authorized_keys
In every /etc/ssh/sshd_config file on every server we have to add the below line to recognise and allow the CA key which we’ve configured above:
For our developers we’ve also created a read only user on the databases, but this completely optional of course. At every server:
# mysql > CREATE USER 'develop'@'localhost' IDENTIFIED BY 'yourstrongpass'; > GRANT SELECT ON *.* TO 'develop'@'localhost'; > flush privileges; > exit
Now for the access. When access is requested, DevOps has to perform the below on the production server in the keys directory. We’re using user ‘John Smith’ as an example.
# ssh-keygen -s /etc/ssh/ssh_ca -I johns -n project -V +1d johns.pub
You can probably understand where the flags are for, but just to be sure:
- -s is for the private key to sign with
- -I is for the name / principle
- -n is for the user we want access to
- -V is for the validity
Here we’re granting access for a day (+1d). We can also grant it for a couple of hours (+4h) or weeks (+2w), etc. The above command will generate the johns-cert.pub file. Provide this key to the developer and remove it from the server once provided.
Now that the CA signed SSH key is in possession of the developer, he should perform the following steps to connect. First, make sure the below settings are in the ~/.ssh/config file:
Use the provided key to connect to the production system. In our example the key is in the Downloads directory.
$ cd ~/Downloads $ chmod 400 johns-cert.pub $ ssh -i johns-cert.pub email@example.com
The developer should be connected. If you’d like, you can check expiration of the key on your local machine with:
$ ssh-keygen -L -f johns-cert.pub
They can connect to databases via the cli or a database tool of their choice, when using the provided key to login with.
This setup does not prevent that an evil developer puts his or her own public key in the /home/project/.ssh/authorized_keys file, thereby granting his or herself unlimited access. This can be monitored and prevented in multiple ways. We decided to make the file and directory immutable with Ansible. You can do this manually with:
# chmod 400 /home/project/.ssh/authorized_keys
# chattr +i /home/project/.ssh/authorized_keys
# chattr +i /home/project/.ssh/