44% of security products contain security problems

Slashdot linked to an interesting analysis of an ISCA Labs report, done by Help Net Security, about the underperformance of various network security products. The meat of the analysis focused on how most products fail to achieve certification on the first test, but I found this particular statistic incredibly enlightening:

Rounding out the top three is the startling finding that 44 percent of security products had inherent security problems. Security testing issues range from vulnerabilities that compromise the confidentiality or integrity of the system to random behavior that affects product availability. Even though it can be a demanding process, certification with a trusted, established third party is critical to verifying product quality, states the report. Product categories studied were: anti-virus, network firewall, Web application firewall, network IPS, IPSec VPN, SSL VPNs and custom testing.

The report has some caveats. For example:

Even the technology used to store and access test data has seen substantial change. We certainly cannot make the claim that a single, consistent data collection method was employed across all products throughout the timeframe of this study.

Check out the rest of the report; it’s a good read. I’ve long been of the belief that most high-end security products (beyond typical endpoint stuff) are snake oil and don’t provide any kind of real ROI; this report does nothing to change my opinion, especially in the IPS space, where a really remarkably huge portion of the sampled products failed to achieve certification.