Tomas' Labroratory

Rubber Stamp from a Big Name or Real Security?

This week I challenged a client (and myself) to a test.  The client went out to get a vulnerability assessment of their SaaS web application from a North American firm who is recognized as one of the top IT security companies in the field.  Let’s call them “H”.  I was sympathetic when my client explained that the reason they picked H.  It was precisely because H was widely recognized and it would be easier to “sell” the result of the assessment to their downstream customers.  In other word H would provide a superior rubber stamp. This bothered me a bit.  So I offered the client the following challenge: If I do a second vulnerability assessment on the same web application will I find more vulnerabilities than H can? Long story short.  I found more vulnerabilities. H found -              1 High risk vulnerability -              3 Medium risk vulnerabilities -              1 Low risk vulnerability -              5 Total I found: -              4  High risk vulnerabilities -              5  Medium risk vulnerabilities -              8  Low risk vulnerabilities -              17 Total Quantity isn’t everything of course.  I also supplied proof of concept code for key vulnerabilities that were not easily reproducible through the application’s GUI.  My client shared with me that H charged more than $15,000 for their work.  In this case, my work was pro-bono, but If I were to charge the client next time, it would have cost them approximately $3000 including preparation and follow-up.  If I crunch the numbers (vulnerabilities per $) I figure that in this case I was about 20x more efficient than H. I have considered whether luck played any role in this difference.  When it comes to finding vulnerabilities I can’t deny that luck does play a role.  But luck without skill will yield absolutely nothing useful.  With a 20x difference in value, and the fact that H must have surely used a top notch analyst, with top notch tools, it still doesn’t add up well for H. While I believe I successfully answered the question whether it’s better to use an IT security freelancer such as myself versus a “Big Name” in security, there is a big question that remains: Do you want a shiny rubber stamp? Or do you need real security?