VictorOps is now Splunk On-Call! Learn More.
In a perfect world, you would instantly fix your application every time a relevant CVE (Common Vulnerability and Exposure) was issued. (In an even more perfect world, there would be no security incidents, and hence, no CVEs in the first place.)
But, in the real world, reacting to CVEs requires a careful calculation. You need to assess whether each CVE is serious enough to warrant the rejection of a build and a delay of a release. That’s a careful balancing act – on the one hand, you don’t want to release code with serious known vulnerabilities, but on the other, you want to avoid constant delays caused by unexpected CVE announcements.
While there’s no one-size-fits-all answer about how to decide which CVEs are serious enough to warrant the rejection of a build, there are some key considerations that can help you make this decision. Keep reading for an overview of how to assess CVE severity.
First, though, a brief definition of CVEs.
A CVE (which stands for Common Vulnerabilities and Exposures) is a known vulnerability in a certain piece of software.
CVEs were introduced in 1999 in an effort to provide a universal framework for identifying security issues. The idea is that whenever a vulnerability is discovered, a CVE entry will be created for it. The CVE entry can then be accessed and shared by the public.
The official CVE list is maintained by MITRE, a nonprofit group, but that list doesn’t include an assessment of how serious each CVE is. However, other organizations that maintain their own lists of CVEs, most notably NIST’s National Vulnerability Database, assign severity categories (such as “medium,” “high” and “critical”) and/or numerical severity scores to CVEs.
There are a variety of tools that will automatically monitor and scan your application code, dependencies or environments in an effort to detect components that are subject to known CVEs, then alert you to them. Thus, it’s relatively easy to find CVEs that are relevant to an application you develop or manage.
The harder task is what to do when you discover a CVE that impacts your application. As noted above, it’s not always practical to delay the release of your application until the CVE is addressed. Although, patches to correct CVEs sometimes appear almost as quickly as a CVE is announced. In some cases, it can take months or longer before a CVE solution becomes available (and it’s not always clear how long it’s going to take for a fix to arrive). You may not be able to wait that long.
Also, you may simply get so many CVE alerts that delaying a release in response to each one is not feasible. If your application is complex and has lots of dependencies, you could get multiple CVE alerts each week. Delaying your release pipeline in response to each one would create a lot of confusion and impact your ability to deliver continuously.
So, how should you react when a CVE is announced? There are two main approaches:
The first approach to CVE management is to maintain a consistent policy where, based on a CVE’s severity rating, you always handle a CVE announcement in the same way. For example, you could enforce a rule where any CVE with a severity ranking of medium or greater means your delivery pipeline will be put on hold until the issue is addressed.
This approach is conservative because it involves applying a blanket policy to all CVEs. You assume that every CVE of a certain type is serious until proven otherwise.
The upside of this approach is that it’s consistent and easy to automate. It also allows you to err on the side of prioritizing security over delivery speed, which may be desirable if you’re particularly risk-averse.
The downside to strict policy-based CVE management is that you’ll likely end up delaying your releases in response to CVE announcements that don’t actually impact you. The severity assessments from NIST and other organizations are useful but they’re also generic and subjective. Just because a given CVE is deemed critical, for example, doesn’t mean it’s actually critical for your particular configuration. Keeping this all in mind, if you require all CVEs with a certain severity level to be addressed before you resume software delivery, you may end up with unnecessary delays.
The other CVE management strategy is to assess each CVE on a case-by-case basis, then determine a course of action. This strategy is more flexible but it also involves taking more risks.
This strategy can’t really be automated. It requires your security engineers to read each relevant CVE alert, no matter what time of day or night it arrives, then make a decision about whether it’s serious enough to delay a release. This approach may also be inconsistent – one security engineer may have a different take on the seriousness of a given CVE for your project than another.
On the other hand, a case-by-case approach is more flexible and provides the ability to maintain release speed even in the face of recurring CVEs. It’s the only way to react to CVEs based on how much they impact your particular application or configuration, rather than on the basis of generic assessments from third-party organizations.
It’s worth noting, of course, that the two strategies described above don’t have to be an either/or proposition, you can combine them to develop a hybrid CVE management strategy. For example, you could use a policy-based approach where you delay the release for all CVEs with a critical severity ranking, but use case-by-case assessment to make determinations about those that fall between medium and critical. Or, you could provisionally delay releases in response to all CVEs with a severity ranking of medium or higher, but then assess them individually in order to decide what you want to do on a case-by-case basis.
At the end of the day, your team must manage CVEs in their own special way. There’s no universal advice about which of the strategies described above works best.
However, here are some factors you might want to consider when deciding which approach to take:
Security issues are always bad. But, they tend to be worse for some organizations and industries than others. If you’re developing a finance app, for example, you’ll probably need to be more conservative in the way you handle CVEs than you would if your vertical is less regulated.
Some delivery pipelines make it easier to roll back a problematic release than others. If you use immutable infrastructure, for example, it’s probably easier to replace one version of your app with another. If you can roll back quickly and easily, you may be able to take more risks when it comes to CVEs because you have a greater ability to pull a release quickly in the event a CVE you chose to ignore turns out to be serious.
Not everyone is equally qualified to interpret the severity of a CVE. If you don’t have any security experts in house (or you don’t have enough to be able to assess a CVE’s impact quickly), you may be better off taking a conservative approach and assuming that all CVEs should be treated seriously. If, on the other hand, you have the expertise to interpret CVEs quickly and meaningfully, you’re better positioned to assess them on a case-by-case basis.
Last but not least, consider how quickly you typically release application updates. Continuous delivery means wildly different things to different teams. You might only be releasing once a week in which case delaying a release by a day or two in response to a CVE isn’t a big deal. On the other hand, if you release every hour, hitting the pause button due to a CVE is much more disruptive to your routine (and to your users, who are probably accustomed to rapid updates).
CVEs are serious business. But, just how serious a CVE is for your particular team and project can range widely. Having a healthy perspective on your ability to tolerate CVE-related delays is critical for ensuring you successfully navigate the straits separating CVE obsessiveness from a fundamentally insecure CVE policy.
Learn how VictorOps is building a collaborative approach to on-call incident response and alert automation, drastically speeding up incident remediation. Sign up for a 14-day, free trial or reach out for a personalized demo to learn more.
Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open source, agile infrastructure and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO. His latest book, For Fun and Profit: A History of the Free and Open Source Software Revolution, was published in 2017.