Security Vulnerabilities: Discover, React, Deploy – in 72 Hours

There is a new vulnerability in a common piece of enterprise networking, from the industry leader in the class of device called a “load balancer” that is particularly nasty: https://www.wired.com/story/f5-big-ip-networking-vulnerability/

The vendor is a well known, generally well respected supplier, and reacted quickly and effectively to the exploit, producing a patch quickly and correctly advising its customers that this patch was necessary to implement immediately.  The referenced article notes that the patch was released on June 30, and if it wasn’t implemented over the weekend, it may well be too late.  So, 4 days or less from notice to deployment.

You might ask your current or proposed NG9-1-1 NGCS vendor how they would handle a vulnerability like this.

It’s pretty typical to see 30, 60 or even 90 day commitments for patches, if there is even a commitment at all.  Usually, vendors claim that they must perform extensive testing on the patch to make sure it doesn’t cause more problems, and often, it can take 20-30 days to roll out a patch to all deployments.

Compare that typical response to the need here. 

It would be great if the bad guys would let us have a few months to patch a serious vulnerability.  Unfortunately, they just aren’t that nice.  Darn.

To be able to react this fast requires a few things:

  1. An automated test that can quickly determine if there are problems with the patch affecting the services
  2. An expedited approval process that takes hours to run
  3. A deployment plan for emergency patches that takes at most a couple days to implement across all deployments

A really important thing that a vendor needs to make this work is practice.  This is the kind of thing that goes well only if everyone who has to do their part has practiced it successfully a few times.  It’s almost guaranteed not to work if it’s never been tried.  And refresher practice every year or so is also important.

It gets even more complex, and requires even more practice when there is more than one company involved.  If the NGCS vendor purchases, say, an ECRF from another company, and, let’s say, a serious vulnerability in a library used by the ECRF supplier is discovered.  Then both the ECRF vendor and the NGCS vendor’s processes have to work in a couple of days, and have to be practiced together a couple times to make sure they do.

How about yours?

72 hours from release of patch to deployment is about all we can reasonably expect, but we should demand that.  You can make it dependent on the severity of the exploit.  The Common Vulnerability Scoring System (CVSS) is one common way to measure the severity of a vulnerability.  Scores of 9 or 10 are usually deemed critical.  Certainly, a CVSS score of 10 would necessitate the fastest response.  Whether a score of 9 requires 72 hour response is something to discuss with vendors or potential vendors.

I wish I knew how to write software that would never have bugs like this one.  I wish there were ways to give vendors more time to respond when this class of bug is discovered.  Alas, no such magic.  So, we have to assume this will happen, plan for it and practice our plan.

Leave a comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.