Bugmusment 2017

The Paper itself:

• http://www.belfercenter.org/publication/taking-stock-estimating-vulnerability-rediscovery

Commentary:

Note that the paper in the selected area would be TS/SCI for both us and China. :)

(To be honest, I don't think it even does this)

• https://twitter.com/daveaitel/status/888031853273329664
• https://twitter.com/daveaitel/status/888046303892168704
• https://twitter.com/daveaitel/status/888045737006825472
• https://twitter.com/daveaitel/status/888042663164903427

Cannot be true.

Ok, so I can see how it went. After the Rand paper on 0day collisions came out, existing paper writers in the process of trying to point out how evil it was the Government knew about 0day were a bit up a creek without a paddle or even a boat of any kind.

Because here's the thing: The Rand paper's data agrees with every vulnerability researcher's "gut feeling" on 0day collision. You won't take a 5% over a year number to a penetration testing company and have them say, "NO WAY THAT IS MUCH TOO LOW!"

But if you were to take a 20% number to them, they would probably think something was wrong with your data. Which is exactly what I thought.

So I went to the data! Because UNLIKE the Rand paper, you can check out their GitHub, which is how all science should work. The only problem is, when you dig into the data, it does not say what the paper says it does!

Here is the data! https://github.com/mase-gh/Vulnerability-Rediscovery

From what I can tell, the Chromium data is from fuzzers, which naturally collide a lot. Especially when in most cases I can click on the rediscovery is from the exact same fuzzer, just hitting the same bug over and over in slightly different ways. The Android data I examined manually had almost all collisions from various libstagefright media parsing bugs, which are from fuzzers. A few seemed to be errors. In some cases, a CVE covers more than one bug, which makes it LOOK like they are collisions when they are not. This is a CVE issue more than anything else, but it skews the results significantly.

Ok, so to sum up:
The data I've looked at manually does not look like it supports the paper. This kind of research is hard specifically because manual analysis of this level of data is time consuming and requires subject matter experts.

It would be worth going in depth into the leaked exploits from ShadowBrokers etc. to see if they support any of the figures used in any of the papers on these subjects. I mean, it's hard not to note that Bruce Scheier has access to the Snowden files. Maybe there are some statistics about exploits in there that the rest of us haven't seen and he's trying to hint at?

This was the paragraph in the paper that worried me the most. There is NO ABILITY TO SCIENTIFICALLY HAVE ANY LEVEL OF PRECISION AS CLAIMED HERE.