This week I challenged a client (and myself) to a test. The client went out to get a vulnerability assessment of their SaaS web application from a North American firm who is recognized as one of the top IT security companies in the field. Let’s call them “H”. I was sympathetic when my client explained that the reason they picked H. It was precisely because H was widely recognized and it would be easier to “sell” the result of the assessment to their downstream customers. In other word H would provide a superior rubber stamp.
This bothered me a bit. So I offered the client the following challenge: If I do a second vulnerability assessment on the same web application will I find more vulnerabilities than H can?
Long story short. I found more vulnerabilities.
H found
– 1 High risk vulnerability
– 3 Medium risk vulnerabilities
– 1 Low risk vulnerability
– 5 Total
I found:
– 4 High risk vulnerabilities
– 5 Medium risk vulnerabilities
– 8 Low risk vulnerabilities
– 17 Total
Quantity isn’t everything of course. I also supplied proof of concept code for key vulnerabilities that were not easily reproducible through the application’s GUI. My client shared with me that H charged more than $15,000 for their work. In this case, my work was pro-bono, but If I were to charge the client next time, it would have cost them approximately $3000 including preparation and follow-up. If I crunch the numbers (vulnerabilities per $) I figure that in this case I was about 20x more efficient than H.
I have considered whether luck played any role in this difference. When it comes to finding vulnerabilities I can’t deny that luck does play a role. But luck without skill will yield absolutely nothing useful. With a 20x difference in value, and the fact that H must have surely used a top notch analyst, with top notch tools, it still doesn’t add up well for H.
While I believe I successfully answered the question whether it’s better to use an IT security freelancer such as myself versus a “Big Name” in security, there is a big question that remains:
Do you want a shiny rubber stamp? Or do you need real security?