VictorOps is now Splunk On-Call! Learn More.
It’s easy to talk about DevOps at a high level by emphasizing how great it is when developers and IT Ops teams collaborate. But, how do you actually implement DevOps? Which types of specific tools and practices are required to enable collaboration between developers and IT operations?
Those questions have long answers but one example is SSH (Secure Shell). Giving developers SSH access to production systems is one basic requirement for enabling more efficient collaboration between IT Ops and development teams.
But, of course, this doesn’t mean every developer should have SSH access to every production system, all the time. You need to be strategic about assigning SSH access to developers. To that end, check out these best practices to learn how and why giving developers SSH access can help you put DevOps into practice.
Typically, SSH is a tool that’s used more frequently by IT Ops than developers. For IT engineers who need to administer systems remotely (especially systems that don’t have graphical interfaces, like many Linux servers), SSH is a fast and easy way to connect.
In contrast, developers are less likely to use SSH day-to-day. They typically do their work on local machines or in cloud IDEs (integrated development environments). They might occasionally log in to testing servers using SSH but that’s the only obvious reason for developers to use the tool.
It was, at least, the only reason prior to the rise of DevOps. For organizations that have embraced DevOps, developers working closely with IT Ops to troubleshoot problems and guarantee software quality is critical to reliability and speed. In many cases, this means developers need SSH to connect to the production applications and logs in order to access the information they need and help IT Ops do their work.
The wrong way to go about giving developers SSH access is to give every developer an account on every system enabled with SSH. That’s a huge security risk. Even worse would be to create one account that developers share for SSH access across all systems. Instead, as a matter of basic security, you need to ensure each developer has access only to the systems that he or she needs.
At the same time, from a manageability perspective, it’s important to streamline the way you manage SSH configurations. The last thing you want is lack of access when a developer’s on-call and needs SSH in order to fix an issue quickly.
You don’t want to have to set up a user account and enable SSH access for it on each individual server and then manually update the account’s passwords periodically. You also don’t want to require each developer to have a different password for each server; he or she should be able to use the same credentials to SSH into any server to which they require access.
Addressing the challenges described above requires some creativity, but there are a few possible approaches:
IaC (infrastructure as code) tools like Puppet, Chef and Ansible all provide ways to manage SSH keys and configurations. Using one of these tools to configure which developers can access what servers via SSH may not be the most obvious use case. But, improved SSH configurations will go a long way toward solving both the security and manageability challenges described above.
Keep in mind that you’ll need to use SSH keys instead of passwords to make this work. But SSH keys are typically more secure anyway, in most respects.
A second approach is to use a tool designed specifically for managing secrets, like HashiCorp Vault or CyberArk Conjur. This strategy will require more custom coding and/or integration with a third-party tool. But it provides a somewhat more secure solution than just giving each developer an SSH key and expecting him or her to store and manage it securely on a local device.
You may not think of Active Directory (a Windows-oriented tool) as a good way to manage SSH access. But it can be used for that purpose. Join your Linux boxes to your AD domain, then create a group of users (your developers) who should have SSH access.
Obviously, this solution would work only if you have an Active Directory domain set up, which you probably don’t if you maintain a Linux-only infrastructure.
A final workable strategy is to go ahead and give all developers access to all servers, but use a firewall configuration to allow SSH requests on each server, only from hosts that should be able to access it. This approach would work well if, for example, each of your development teams is on its own subnet, because you could give each subnet access to the servers it requires while blocking SSH requests from other subnets.
Or, you could block all SSH requests from developer boxes by default, then manually enable them temporarily in the firewall on a case-by-case basis, when developers need to connect to a certain server as part of DevOps work.
This solution is definitely a hack, but (like most good hacks) it’s fast and easy. It doesn’t require you to manage SSH keys or even use them at all (you could simply stick to passwords). It only requires some basic firewall configuration. Of course, it comes with a considerable security risk, which is that someone could gain unauthorized SSH access to a server by finding a way to appear to connect from an IP address or subnet that is allowed in the firewall configuration.
Giving developers SSH access to production servers is often a necessary part of practicing DevOps. And, while achieving this in a scalable, secure way may seem hard, the right tools can simplify the job.
Developers with SSH access can take more accountability for the services they build and respond to issues faster. Sign up for a 14-day free trial or request a free personalized demo of VictorOps to improve collaboration and visibility between IT Ops and developers during a firefight, leading to on-call incident management that doesn’t suck.
Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open-source, agile infrastructure and networking. He is the Senior Editor of content and a DevOps Analyst at Fixate IO. His latest book, For Fun and Profit: A History of the Free and Open Source Software Revolution, was published in 2017.