[Spoiler alert – if you have lived under a rock most of your life and aren’t yet familiar with Spiderman’s origin story but wanted to read it one day, you probably want to do that before reading this post. You can actually see Amazing Fantasy #15 online at Archive.org. Go ahead and read it regardless; I’ll wait. It is a great story and a magnificent piece of comic book history.]
I have bad news for security folks reading this blog post – security in DevOps pipelines isn’t like the comic books and if you want developers to adore you like Spiderman you might have to let a couple of killers go free – for a little while at least.
Peter Parker failed to stop a thief when he had the chance and this thief – being a degenerate and apparently escalating career criminal – ended up killing Peter’s beloved Uncle Ben. Peter is deeply moved by this and goes on to become your friendly neighborhood Spiderman and a scourge of criminals everywhere. And I’ll bet if you asked him, Spidey would err on the side of caution – stopping folks he thought might be criminals because he didn’t want to let another one go free just in case they, too, were to then commit a horrible crime.
But that approach isn’t going to work if you’re integrating security testing into a Continuous Integration/Continuous Delivery (CI/CD) pipeline. Organizations use CI/CD pipelines so they can go faster. They want to know if the last set of incremental changes that were made put the software in a state where they know they couldn’tdeliver it without major issues. All software has problems, and for most organizations, the CI/CD pipeline needs to answer the question “Did I really mess something up with that last set of updates?” Breaking builds slows things down and forces one or more folks to go back and undo or redo something that they have recently done because it is perceived to have changed the quality of the software in a way that makes it unacceptable to deliver. As a result, in a CI/CD pipeline, the ability to break a build represents tremendous power. And – as our friend Peter Parker learned – with that great power comes great responsibility.
At the beginning of his career, Spiderman didn’t use his powers to stop a crime, and that failing resulted in his uncle’s death. From then on he’s roaming around New York City stopping every crime he can. Security teams, however, need to approach their newfound powers in a different manner. Breaking builds slows things down, so the real crime in a CI/CD pipeline is breaking a build when it should have gone forward. If security teams get invited to participate in the CI/CD pipeline they need to recognize the implications of the power they have and wield that power responsibly, which means only stopping builds when they have strong evidence that the last round of changes has truly introduced a serious and verifiable security problem. Anything more heavy-handed than that and the security team is going to be viewed like a villain. Nobody wants the bad guys weighing in on their CI/CD pipeline, and security teams that take this approach will quickly be marginalized and removed from the process.
From a practical standpoint, if you want to add security checks into a CI/CD pipeline, those checks need to be carefully curated, and when they trigger on builds they need to be handled thoughtfully. In short – security findings from testing done in CI/CD pipelines need to be:
- Available quickly– The speed of CI/CD builds is important, but that is out of scope for this current post.
- High value– They need to be things that developers recognize as being important enough to justify slowing down the process – SQL injection, cross-site scripting (XSS) – the kind of stuff that too easily gets organizations into headlines.
- Low (NO!) false positives– They need to be correct. The CI/CD pipeline isn’t the time to slow stuff down because you think you have a problem. You have to know you have a problem. And the developers need to believe you.
None of this is to say that security in DevOps environments has to be limited to this sort of fast and high-confidence testing. You can – and should – still kick off asynchronous processes and manual reviews to do deeper testing to look for more subtle and harder-to-find issues. But this can’t hold up pipeline builds. Instead, security teams should set up checkpoints where the results of these out-of-band assurance activities can be aggregated, analyzed, and presented to the development team for remediation. But in the CI/CD pipeline – keep it fast, keep it high-impact, and keep it high-confidence. Failing to exercise this power responsibly will get you kicked out of the CI/CD pipeline, and being on the outside looking in is a horrible place for a security team to find themselves.
We’ve been doing this for a while in ThreadFix – helping security and development teams to work together to craft sensible security testing policies within CI/CD pipelines. ThreadFix CI/CD integration lets you set up both synchronous and asynchronous orchestrations – running security testing that can either be used to make go/no-go decisions about CI/CD builds, or to provide information that can be acted on later – all reported into the tools that your development teams are already using.
To all the security folks out there – when you get invited to help evolve your DevOps teams to be DevSecOps teams – use your powers wisely and be willing to let some things pass in the short term because some security testing in CI/CD is better than no security testing in CI/CD. Save your battles for the really important stuff you identify later on. Uncle Ben would have wanted it that way.