New Web Application Scanner Technology Guarantees NO False Positives

By Dan Cornell

Recently I’ve been working with some other Denim Group folks to do our regular internal benchmarking of various application security scanners.  Last week we got into a deep discussion of false positives, how many scanners claim to reduce or eliminate them, and different techniques to make this happen.  During the talk we came across an idea for a surefire way to completely eliminate false positives and this weekend I had enough free time to put together a proof of concept.  I’ve included the proof of concept code at the bottom of this post and you can consider it to be released under the Apache 2.0 license.

This is revolutionary enough that we are currently trying to make testing with this technology required for Scanless PCI Certification and we want to make sure that all Certified Application Security Specialists are required to be conversant with it.  We’re considering shifting all our use of scaning tools to use our new technology.

Below is the proof of concept code:

package com.denimgroup.nofalsepositives;

import java.util.ArrayList;

import java.net.MalformedURLException;

import java.net.URL;

public class DynamicAnalyzer

{

  public static void main(String[] args)

  {

    ArrayList vulnerabilities = new ArrayList();

          

    String sUrlToScan;

    URL urlToScan = null;

          

    long scanStart;

    long scanEnd;

          

    // Must at least enter a URL for the site to scan

    if(args.length < 1) {

      usage();

      System.exit(1);

    }

          

    // Make sure the URL is valid

          

    sUrlToScan = args[0];

    try {

      urlToScan = new URL(sUrlToScan);

    } catch (MalformedURLException e) {

      System.out.println("Provided URL was invalid.  Unable to scan.");

      System.exit(2);

    }

          

    // Kick off the scan

          

    scanStart = System.currentTimeMillis();

    System.out.println(String.format("Starting scan of %s at %d", urlToScan.toString(), scanStart));

          

    // Finalize scan and report findings

          

    scanEnd = System.currentTimeMillis();

    System.out.println(String.format("Finished scan of %s at %d", urlToScan.toString(), scanEnd));

    System.out.println(String.format("Found %d vulnerabilities with NO false positives", vulnerabilities.size()));

  }

    

  public static void usage()

  {

    System.out.println("usage: java com.denimgroup.nofalsepositives.DynamicAnalyzer <SITE_URL>");

  }

}

In case anyone hasn’t figured it out by now this whole post is completely facetious.  Automated scanning has its place in any credible application security program, but no credible application security program consists only of automated scanning.  And you will always get false positives as well as results that need to be re-prioritized based on the business context of the vulnerability.  Automation is great, but you can’t automate everything.

Contact us if you would like more info on constructing your application security program.

dan _at_ denimgroup.com

@danielcornell

Posted via email from Denim Group's Posterous

Comments

[Robert Sullivan]

Good bit, and entertaining. Is Dan nibbling at the edges of a serious hypothesis that deserves consideration? "When the ratio of false-positives to positives goes below 50% it is time to get new tools."

[mls]

Dan, I fear you may have been scooped:

From http://www.mavitunasecurity.com/

"So we developed a new technology which can confirm vulnerabilities on demand, which allowed us to develop the first false positive free web application security scanner."

However their claims are neither ironic nor spoofed. While there is enough wiggle room and caveats in the surrounding paragraphs for the Mavituna guys to make their case, the phrase "false positive free web application security scanner" should cause some consternation.

[Dan Cornell]

Robert:

I suppose it depends what you are looking to use the automated scanner for. If you want it to be point-and-click then looking at the ratio of false positives to actual results makes sense. However if you are using it as an information gathering tool to focus follow-on manual efforts the ratio is less important than the time required to validate results and cull out the false positives.

We are getting ready to release some technology we put together that lets you import results from multiple scanning tools and then merge them together. One item of interest is what sort of metrics we can track, and looking at false positive ratios per tool might be interesting.

--Dan