This week I challenged a client (and myself) to a test.  The client went out to get a vulnerability assessment of their SaaS web application from a North American firm who is recognized as one of the top IT security companies in the field.  Let’s call them “H”.  I was sympathetic when my client explained that the reason they picked H.  It was precisely because H was widely recognized and it would be easier to “sell” the result of the assessment to their downstream customers.  In other word H would provide a superior rubber stamp.

This bothered me a bit.  So I offered the client the following challenge: If I do a second vulnerability assessment on the same web application will I find more vulnerabilities than H can?

Long story short.  I found more vulnerabilities.

H found

–              1 High risk vulnerability

–              3 Medium risk vulnerabilities

–              1 Low risk vulnerability

–              5 Total

I found:

–              4  High risk vulnerabilities

–              5  Medium risk vulnerabilities

–              8  Low risk vulnerabilities

–              17 Total

Quantity isn’t everything of course.  I also supplied proof of concept code for key vulnerabilities that were not easily reproducible through the application’s GUI.  My client shared with me that H charged more than $15,000 for their work.  In this case, my work was pro-bono, but If I were to charge the client next time, it would have cost them approximately $3000 including preparation and follow-up.  If I crunch the numbers (vulnerabilities per $) I figure that in this case I was about 20x more efficient than H.

I have considered whether luck played any role in this difference.  When it comes to finding vulnerabilities I can’t deny that luck does play a role.  But luck without skill will yield absolutely nothing useful.  With a 20x difference in value, and the fact that H must have surely used a top notch analyst, with top notch tools, it still doesn’t add up well for H.

While I believe I successfully answered the question whether it’s better to use an IT security freelancer such as myself versus a “Big Name” in security, there is a big question that remains:

Do you want a shiny rubber stamp? Or do you need real security?