AppSec Blog

AppSec Blog

Seven Tips for Picking a Static Analysis Tool

Stephen J, who is a member of our software security mailing list, asked a while back, "Do you have any recommendations on static source code scanners?" James Jardine and I started talking and came up with the following tips.

There are so many commercial static analysis tools from vendors like Armorize, Checkmarx, Coverity, Fortify (HP), Klocwork, IBM, and Veracode that it's hard to recommend a specific product. Instead we'd like to focus on seven tips that can help you maximize your selection.

1) Test before you buy


This probably sounds obvious but, assuming you haven't purchased anything yet, definitely do a bake off and have the vendor run the code against your actual apps. Do *not* simply run the tool on a vendor supplied sample app as the quality of the results, surprisingly, can vary quite a bit across different tools and code bases. Just keep in mind that some vendors will try to avoid this so they can better control the test or, simply, to keep you from getting a free scan.

2) Determine your budget


It's really important to understand your budget. The cost of commercial tools varies greatly between vendors and you should be aware of how much you are willing to spend. This includes the cost of the tool, ongoing maintenance, and even things like additional RAM for your build servers (yes, it can take a lot of RAM). And don't forgot to budget for the time people spend setting up, configuring, and reviewing results from the tool.

3) Prepare to spend a lot of time on it


Usually, anything worth doing takes time. Test-driven development and writing thorough unit tests don't happen on their own. Similarly, using a static analysis tool to find security issues is an investment. It needs both management and developer buy-in to succeed when time consuming issues like false positives, tweaking rules, and product upgrades come up.

4) Cloud or Internal


Are you ok with uploading your source code, or binaries, to a third party site to do the analysis? This is only a question you and your company can answer, but it is an important one. Not only do you have to consider sharing your code, but if the solution is in the cloud, what about integration with your build process. Do you want to have a solution that can be built into your nightly build process? Tools that are housed internally usually lend themselves better to customization. Do you have the staff and capabilities to extend the tool so that it works better for your applications?

5) What type of support do you need?


Some tools come with much more support than others. One tool may have support personnel available to go over the results of your scan with you, so you understand them, whereas others may not have this type of support. Depending on your in-house skill set this could be a big deciding factor.

6) Check references


Ask other colleagues (not the vendors) who use the tool and get their feedback. These are the ones that will give you honest answers. Keep in mind that even though a specific vendor works well for one application or company it doesn't mean that it's the right fit for your situation.

7) Research the products


Make sure that the solution you are considering is the best fit for you. You will definitely want to make sure that the vendor/product you choose supports your development platform and your development process.

Every application is different and it's impossible for an outsider to recommend the correct static analysis tool without having intimate knowledge of your application and processes. Given the investment in time and money, choosing a vendor and product is a difficult yet important decision. Do your due diligence and research before making a choice.

Take some time to lay out your current landscape and answer some of these questions:

- What language(s) do you develop in?
- What source control system do you use?
- What other types of technologies are used in your application (Ajax, jQuery, Silverlight, Flash, mobile, etc)?
- What skill sets do you have in house when it comes to application security?
- Do you have people that can quickly learn how to operate the scanner? Can they understand the output?
- What type of support do you want from the vendor?
- Are you willing to submit your code to an external site, or does your policy require that it not be shared outside the company boundaries?
- What is your budget for a static analyzer?
- How will the technology work with the people and processes that you have in house?

Hopefully, all these questions don't change your opinion about needing a static analysis tool and make it a little easier to actually select a tool to use.

Thanks for reading!

Frank Kim & James Jardine

5 Comments

Posted December 14, 2011 at 8:10 PM | Permalink | Reply

Dr. Michael Zhao

Great summary and thanks for sharing.
I used Klocwork on a large code base that had over 300 software engineers working on it. Great experience.
Static analysis tools are powerful tools that are very important for large scale software project.

Cheers,
Michael

Posted December 15, 2011 at 4:05 PM | Permalink | Reply

Jim Bird

Good points. Another question is: What is more important to you, code quality or code security? Klocwork and Coverity, for example, started life as code quality scanners and most of their checkers are for code quality issues, not security issues, although they continue to add security-specific checkers. Fortify (HP) is a security scanner that also has some quality checks.

And you may want to check into how extensible the scanner is - how easy can you add your own rules checking to the scanner?

Another criterion is whether you are already using other tools from the same vendor. Fortify (HP) can tie their static source code analyzer in their dynamic vulnerability scanner for hybrid code analysis making both types of scanning more intelligent and useful.

And finally, remember that these tools all use different techniques and often find different problems. You may want to consider whether it is worth using more than one scanner (we do) to make sure that you catch as many problems as you can as soon as you can.

Posted October 02, 2012 at 4:01 PM | Permalink | Reply

Mitchell

Great article. It touches on many points that people consider when choosing a static analysis tool for their company. Parasoft, the company I work for, wrote a guide with a somewhat different take on how to approach static analysis evaluations. It's available at http://alm.parasoft.com/static-analysis-tool-evaluation-guide/ I'd be interested in hearing what you think about it.

Posted January 03, 2013 at 6:47 PM | Permalink | Reply

CAST User

Hi, Nice article. I wonder why "CAST" software is not mentioned anywhere. It is the leading company in this market and the products you mentioned only cover a part of what CAST does. So for me, anybody, who has not experienced CAST, shall not say that it is a great experience:)

Posted April 26, 2013 at 6:39 PM | Permalink | Reply

Ram Cherukuri

Nice article. One additional consideration especially with safety critical software is the need for proof of absence of errors. Polyspace is a formal methods based tool that is ideal for such applications. You can read more about that at http://www.mathworks.com/discovery/formal-methods.html

Post a Comment






Captcha

* Indicates a required field.