By Dan Cornell
One of the things I do is answers questions as an "expert" for the web site SearchSoftwareQuality.com and they recently posted an answer I gave to a question about how to get developers and quality assurance folks to do security testing. You can see my blog post about this here as well as the original question and answer here. After I posted this online, Aaron Weaver on Twitter asked a great question: "What else beside management buy-in?"
I had to admit - this was a great question and my original response did kind of boil down to:
- Decide if you really want developers and QA to be performing security testing
- If so, you have to get management support to make it happen
That's all well and good, but doesn't really address the whole issue which could be summed up in the question "Assuming I want developers and QA to be performing security testing AND I have management buy in how do I do this successfully?" So I figured I would provide some more thoughts based on our experience working to get developers and QA folks to help out with security testing.
In short:
- Understand your goals for dev/QA security testing and let the involved parties know what the goals are
- Learn about what they are currently doing so you can set proper expectations
- Provide them context-specific training and support so they can be successful
- Follow up later to see if you are getting what you wanted
Let's look at each of these in a little more detail:
1 - Know and Communicate Your Goals
What exactly is it that you want to get from having developers or QA folks perform security testing? If you're hoping that providing them some testing tools will magically solve your software security problem you are in for a rude awakening so it is important to be realistic. Are you hoping that developers can catch certain types of security bugs early so you don't have to deal with them later? Is your security group understaffed and you're hoping that QA can help shore you up with some manual application penetration testing? You need to know your goals up-front, be realistic about what you can achieve and make sure that the developers and QA testers know their role in the process. If you can't tell these folks what you want them to do - management buy-in or not - they aren't going to be successful in doing it. Also it is probably a good idea at this point to take some measurements of the current state of affairs - how many security bugs are making it past development and QA? How long - calendar time and level of effort - does it take to fix vulnerabilities once they're identified? This will be important later.
2 - Learn About Current Development and QA Processses
Once you know the outcomes you want to see, it is really important to understand what practices, tools and processes developers are QA already have in place for development and testing. Is the team doing Test-Driven Development (TDD)? Are they using a Contininuous Integration (CI) environment with an existing set of unit and functional tests? Or is the situation, candidly, a mess with no processes, little or not automation and no real framework for making changes. Our sample size isn't really big enough yet to state this quantitatively, but one of the things we found doing some stastistics on software security remediation projects was that teams that had good, "modern" development practices in place were able to fix vulnerabilities with less disruption and effort. The same goes for integrating security testing into the activities of development and QA teams - the maturity of their overall environment is going to be an indicator of what you can expect these folks to be able to do on the security front.
3 - Provide Context-Specific Training and Support
After you know what you want developers and testers to do and have reality-checked these goals against the ground truth of what the teams are capable of, you need to show them what you want them to do and the more specific and context-relevant you can make this education, the more successful you are going to be. I used to think that teaching the principles of secure development was sufficient - if developers know they need to do input validation and contextual encoding they'll go out and find the appropriate resources for their particular environment, right? I also used to think that all developers loved to spend their free time looking at different tools and figuring out how they could be useful in their environment. In short, I was an idiot. (Arguably I'm still an idiot, but that is probably something to explore in other blog posts...) Developers and testers already have jobs to do and, prior to your current initiative, those jobs probably didn't require them to do security testing. Management buy-in is helpful to spur actions, but development and non-security testing still has to get done. Assuming you want to developers and testers to adopt new practices and new tools at any sort of scale you need to show them what they need to do and be as specific and prescriptive as possible so that you minimize the impact on ... all the other stuff the developers and QA testers are supposed to do.
4 - Follow Up
If you've crafted your developer and QA security testing program along these lines so far hopefully you're in a position where you can see some results. So next you need to check and see if you're actually getting the results you want. If you wanted developers to start running a static analysis tool and catching things like SQL injection and cross-site scripting (XSS) before it got to pre-deployment testing you should check to see if that is actually happening (you did remember to check and see how many SQLi and XSS you were finding before you made this change, right?) If it is - fantastic! If not - why not? This post is predicated on the assumption that you have enough management buy-in to get these teams to change their behavior. If you like having management buy-in and want to continue to have it in the future you'd best have answers to questions like "I authorized you to spend $X on application security testing - what did we get from that?"
Watching This In Action
I had the opportunity to run through this process with a group who was rolling out a static analysis tool to their developers recently. They had purchased enough licenses for the majority of their developers and wanted to use the static analysis tool to decrease PCI-relevant findings before applications got in front of PCI auditors. We worked with them to understand their development environment and how code was developed, tested and ultimately deployed to production. Based on this we crafted a very specific rollout and training plan. Part of this program development was a conversation like this:
Me: "... and we can show them the different types of analysis the tool can perform - it has some really cool capabilities"
Security Rep: "Don't care"
Me: "... and we can show them how to write their own custom rules - that can be really helpful tuning the tool to work for applications in your environment"
Security Rep: "Don't care"
Me: "... and ..."
Security Rep: "I just want you to show them how to install the IDE plugin, how to click the button to run a scan, and how to interpret and address the results the scan kicks out. Any other problems - they can come to me for help."
And that's exactly what we did. Did we strike a "grand bargain" to radically change the way this organization develops software? No. Did we turn all of their developers into secure software development gurus? No. But what we did accomplish was:
- Took an investment of money and staff time
- Acquired a new technology and rolled it out by changing the organization's processes and training the relevant staff on what they need to do to be successful
- Put the developers in a position to find and address important vulnerabilities earlier in their development process
- Reduced the vulnerability count in their software and made their PCI audits less painful (with graphs to prove it!)
Was this a "home run" for the cause of software security in this organization? No. Was it a solid single or double? Yeah I think it was. And, possibly more important, it provides a track record of success that can be used to make further advancements. I'll take it.
Contact us to talk about the roles development and QA can play in building secure software.
--Dan
dan _at_ denimgroup.com
