There was a question on an application security mailing list I am on about the current landscape of application security tools – what were they good for, is static analysis better than dynamic analysis, etc. Here is my unedited reply:
We get asked that question quite often and here is my (short) answer:
Application security tools are really good at what they do and they are getting better. Also they are pretty much useless for the things that they don’t do and that isn’t likely to change any time soon.
Tools are great for finding technical flaws in applications – the types of flaws that deal with configuration, input handling and output formatting issues. Things like SQL injection and cross site scripting. Tools are typically really bad at finding logical flaws in applications – these are the issues that deal with business logic, authorization and trust issues. Examples would be insecure direct object references or passing the price to an e-commerce application in a hidden variable. Tools have no way to gauge intent so there are flaws that slip by.
Another good thing about tools is that they don’t get tired or sloppy – they can chew through large applications and large code bases and apply rules on a consistent basis. Organizations would do well to use tools as an initial step in assessing the security of an application but they MUST realize that no tool is going to be able to demonstrate that an application is secure. To get to any reasonable level of assurance, tools must be augmented by manual testing and inspection.
It has also been interesting to watch the tools popular in the industry evolve over time. The early tools were black box scanners (Watchfire, SPI Dynamics). These look and act pretty much like Nessus or ISS but for web applications. This makes sense because the first folks to really get concerned about application security were the information security folks in an organization and these people tended to have network and sysadmin backgrounds. They are used to scanning an operating environment to find flaws.
As the “problem” of software security has started to be adopted by the software development groups in an organization we have seen an upsurge in code-level analysis tools (Fortify, OunceLabs). That is because software development folks have software development backgrounds (naturally) and they are comfortable dealing with code and compilers.
As for what tools an organization should use – it is important to look at what group is going to be running the tools. If the security group is full of folks with network and system admin backgrounds it is unlikely that they are going to become proficient enough .NET or JEE coders to get a lot of utility out of a source code analysis tool. Also it might be hard for them to get the software development groups to turn over source code (for some reasons most enterprises we work with are a little sensitive about shipping that all over the place…) For groups like that the dynamic analysis/scanner tools allow them to get a solid analysis of the security state of the tools in their portfolio.
If the application development group is going to be running the tool, using tools that can take advantage of having full access to the source code makes a lot of sense. The scanner tools have to make a guess as to whether or not the request/response pattern they have seen is an actual vulnerability. The source code analysis tools have a much more clear view of what the software is actually going to do. For groups like these the source code analysis tools make a lot of sense. At the end of the day for large organizations it often makes sense to have both types of tools in use to get a “second opinion” of the security state of their applications.
Obviously this answer is full of simplifications and generalizations and solutions for one organization won’t necessarily work for others. But I did say this was my short answer ;)
I forgot a couple of points so I had to post a follow-up:
Couple of things I forgot to mention:
The tools don’t run themselves. If you buy a tool you should know how you are going to modify your processes to make use of it. Then you need to make sure the folks who will be using the tool know how it actually works and can operate it. And it pays to check back with those people after a while and make sure that the tool is actually being used and is being used as it should. Otherwise you can spend as much money as you like but your applications will not get any more secure. You might fool a couple of auditors – but you won’t fool the attackers ;)
I suppose I could shrink that down to an even shorter answer:
Use application security tools to apply consistent checks for technical application flaws. Do not expect to be finished after you run one. Be sure that you know how you are going to use the tool before you buy it and train the people who are going to use it.
Perhaps a post at a later date will examine why I seem to end all of my emails with the silly winky emoticon ;)
dan _at_ denimgroup.com