Rebecca Slayton can write and has a cool name

This is a really great paper you can read here. Highly recommend it for its much more in-depth analysis of how offense is probably expensive. At the end, she goes into a cost/benefit of both offense and defense of Stuxnet - which in my opinion is the weakest part of the paper. You can't say vulnerabilities cost 0 dollars if they came from internal sources. And you can't just "average" the two possibilities you have and come up with a guess for how much something was.

I mean, the whole paper suffers a bit from that: If you're not intimately familiar with putting together offensive programs, there are many many moving pieces you don't account for in your spreadsheet. That's something you learn when running any business really. On the other hand, she's not on Twitter, so maybe she DOES have experience in fedworld and just doesn't want to go into depth?

Also, there's no discussion of opportunity costs. And a delay of three months on releasing a product, equally true for web applications and nuclear bombs, can be infinitely expensive.

But aside from that, this is the kind of work the policy world needs to start doing. Five stars, would read again.

I mean, the simpler way of saying this is the NSA mantra, which is that whoever knows the network better, controls it. And defenders obviously have a home-field advantage... :)

Quotas as a Cyber Norm

Originally I chose this picture as a way of illustrating perspective around different problems we have. But now I want a giant scorpion pet! So win-win!

Part of the security community's issue with the VEP is that it addresses a tiny problem that Microsoft has blown all out of proportion for no reason, and distracts attention from the really major and direct policy problems in cyber namely:

• Backdoored supply chains
• National Security Letters (and their cousin the All Writs Act as the FBI interprets it)
• Trojaned Cryptographic Standards

Vulnerabilities have many natural limits - like giant scorpions needing oxygen. If nothing else, it costs money to find them and test and exploit them, even assuming they are infinite in supply, which I assure you they are not. Likewise, vulnerabilities can be mitigated by a company with a good software development practice - there is a way for them to handle that kind of risk. A backdoored cryptographic standard or supply chain attack cannot be mitigated, other than by investing a lot of money in tamper proof bags, which is probably an unreasonable thing to ask Cisco to do. 

Deep down, forcing the bureaucracy to prioritize on actions that have no "cost" to them but high risk for an American company makes a lot more sense than something like the VEP, which imposes a prioritization calculus on something that is already high cost to the government.

Essentially what I'm asking for here is this: Limit the number of times a year we intercept a package from a vendor for backdooring. Maybe even publish that number? We could do this only for certain countries, perhaps? There are so many options, and all of them are better than "We do whatever we want to the supply chain and let the US companies bear those risks."

Likewise, do we have a policy on asking US companies to send backdoored updates to customers? Is it "Whenever it's important as we secretly define it?"

Imagine if China said, "Look, we backdoor two Huawei routers a year for intelligence purposes." Is that an OK norm to set?

A partial retraction of the Belfer paper

https://www.lawfareblog.com/what-you-see-what-you-get-revisions-our-paper-estimating-vulnerability-rediscovery

"Did not ever look at the data - hope the other two data sources are ok?"

As a scientist, I had the same reaction to the Belfer VEP paper that a climate scientist would when confronting a new "science" paper telling me the Earth was going into an ice age. So you can imagine my "surprise" when it got partially retracted a few days after being published in response to me having a few spare hours the afternoon the day after it was originally released.

Look, on the face of it, this paper's remaining numbers cannot be right. There are going to be obvious confounding variables and glaring statistical flaws in any version of this document that claims 15% of all bugs collide between two independent bug finders within the conditions this paper uses. They haven't released the new paper and data yet, so we will have to wait to find out what they are. But if you're in the mood for a Michael Bay movie-style trailer, I will say this: The answer is fuzzers, fuzzers, and more fuzzers. A better title for this paper would have been "Modern fuzzers are very complex and have a weird effect on bug tracking systems and also we have some unsubstantiated opinions on the VEP".

The only way to study these sorts of things is to get truly intimate with the data. This requires YEARS OF WORK reading reports daily about vulnerabilities and also experience writing and using exploits. Nobody in policy-land with a political science or international relations background wants to hear that. It sounds like something a total jerk would say. I get that and for sure Ari Schwartz gets that. But also I get that this is not a simple field where we can label a few things with a pocket-sized methodology and expect real data, the way this paper tried to.

An example of how not being connected to the bugs goes horribly wrong has been published on this blog previously, when a different Befler "researcher" (Mailyn Fiddler) had to correct her post on Stuxnet and bug collisions TWICE IN A ROW because she didn't have any of the understanding of the bugs themselves necessary to know when her thesis was obviously wrong.

As in the case of the current paper, she eventually claimed her "conclusion didn't change" just because the data changed drastically. That's a telling statement that divides evidence-based policy creation from ideological nonsense. Just as an ideological point of reference, Bruce Schneier, one of the current paper's authors, also was one of the people to work with the Guardian on the Snowden Archive.

The perfect paper on bug collision would probably find that the issue is multi-dimensional, and hardly a linear question of a "percentage". And any real effort to determine how this policy works against our real adversaries would be TOP SECRET CODEWORD at a minimum.

Something is very wrong with the Belfer bug rediscovery paper

This is what the paper says about its Chromium data:

Chrome: The Chrome dataset is scraped from bugs collected for Chromium, an open source software project whose code constitutes most the Chrome browser. 20 On top of Chromium, Google adds a few additional features, such as a PDF viewer, but there is substantial overlap, so we treat this as essentially identical to Chrome.21 Chrome presented a similar problem to Firefox, so to record only vulnerabilities with a reasonable likelihood of public discovery, we limited our collection to bugs labeled as high or critical severity from the Chromium bug tracker. This portion of the dataset comprises 3,397 vulnerability records of which there are 468 records with duplicates. For Chrome, we coded a vulnerability record as a duplicate if it had been merged with another, where merges were noted in the comments associated with each vulnerability record, or marked as a Duplicate in the Status field.

The problem with this methodology is simply that "merges" do not indicate rediscovery in the database. The vast majority of the findings relied upon for the paper are false positives.

To look at this, I went through their spreadsheet and collected out those mentioned 468 records. Then I examined them on the Chromium bug tracking system. The vast majority of them were "self-duplicates" from the automated fuzzing and crash detection systems.

I'm a Unix hacker so I converted it to CSV and wrote a Python script to look at the data. Happy to share scripts/data.

Looking at just the ones that have a CVE or got a reward makes more sense. There's probably only 45 true positives in this data set (i.e. the ones with CVE numbers). That's 1.3% which agrees with the numbers from the much cleaner OpenSSL Bug Database (2.4%) from this paper.



---
Notes:
Example false positives in their data set:

• This one has a CVE, but doesn't appear to be a true positive other than people noticed things crashed in many different ways from one root cause.
• Here someone at Google manually found something that clusterfuzz also found.
• Here is another clear false positive. Here's another. Literally I just take any one of them, and then look at it.
• Interesting one from Skylined, but also a false positive I think.

Decoding Kasperksy

http://fortune.com/2017/07/16/kaspersky-russia-us/

Although various internet blowhards are hard at work asking for "More information to be released" regarding why the US is throwing Kaspersky under the bus, that's never going to happen. It's honestly easiest to get in the press by pretending to be in disbelief as to what the United States is doing in situations like this.

I say pretend because, it's really pretty clear what the US is saying. They are saying, through leaks and not-so-subtle hints, that Kasperksy was involved in Russian operations. It's not about "being close to the Kremlin" or historical ties between Eugene Kaspersky and the FSB or some kind of DDoS prevention software. Those are not actionable in the way this has been messaged at the highest levels. It's not some sort of nebulous "Russian Software" risk. It's about a line being crossed operationally.

The only question is whether you believe Eugene Kaspersky, who denies anything untoward, or the US Intelligence Community, which has used its strongest language and spokespeople as part of this effort and has no plans to release evidence.

And, in this particular case, the UK intel team (which has no doubt seen the evidence) is backing the US up, which is worth noting, and they are doing it in their customary subtle but unmistakable way, by saying at no point was Kaspersky software ever certified by their NCSC.

The question for security consultants, such as Immunity, is how do we advise our US-based clients - and looking at the evidence, you would have to advise them to stop using Kaspersky software. Perhaps your clients are better off with VenusTech?

I'm pretty sure this AV company is deceased!

Bugmusment 2017

The Paper itself:

• http://www.belfercenter.org/publication/taking-stock-estimating-vulnerability-rediscovery

Commentary:

Note that the paper in the selected area would be TS/SCI for both us and China. :)

(To be honest, I don't think it even does this)

• https://twitter.com/daveaitel/status/888031853273329664
• https://twitter.com/daveaitel/status/888046303892168704
• https://twitter.com/daveaitel/status/888045737006825472
• https://twitter.com/daveaitel/status/888042663164903427

Cannot be true.

Ok, so I can see how it went. After the Rand paper on 0day collisions came out, existing paper writers in the process of trying to point out how evil it was the Government knew about 0day were a bit up a creek without a paddle or even a boat of any kind.

Because here's the thing: The Rand paper's data agrees with every vulnerability researcher's "gut feeling" on 0day collision. You won't take a 5% over a year number to a penetration testing company and have them say, "NO WAY THAT IS MUCH TOO LOW!"

But if you were to take a 20% number to them, they would probably think something was wrong with your data. Which is exactly what I thought.

So I went to the data! Because UNLIKE the Rand paper, you can check out their GitHub, which is how all science should work. The only problem is, when you dig into the data, it does not say what the paper says it does!

Here is the data! https://github.com/mase-gh/Vulnerability-Rediscovery

From what I can tell, the Chromium data is from fuzzers, which naturally collide a lot. Especially when in most cases I can click on the rediscovery is from the exact same fuzzer, just hitting the same bug over and over in slightly different ways. The Android data I examined manually had almost all collisions from various libstagefright media parsing bugs, which are from fuzzers. A few seemed to be errors. In some cases, a CVE covers more than one bug, which makes it LOOK like they are collisions when they are not. This is a CVE issue more than anything else, but it skews the results significantly.

Ok, so to sum up:
The data I've looked at manually does not look like it supports the paper. This kind of research is hard specifically because manual analysis of this level of data is time consuming and requires subject matter experts.

It would be worth going in depth into the leaked exploits from ShadowBrokers etc. to see if they support any of the figures used in any of the papers on these subjects. I mean, it's hard not to note that Bruce Scheier has access to the Snowden files. Maybe there are some statistics about exploits in there that the rest of us haven't seen and he's trying to hint at?

This was the paragraph in the paper that worried me the most. There is NO ABILITY TO SCIENTIFICALLY HAVE ANY LEVEL OF PRECISION AS CLAIMED HERE.

An important note about 0days

https://twitter.com/ErrataRob/status/886942470113808384

For some reason the idea that patches == exploits and therefor any VEP-like program that releases patches is basically also trickling out exploits is hard to understand if you haven't done it.

Also, here's a very useful quick note from the head of Project Zero: