Thursday, September 26, 2024

Mythical Beasts


One of the more interesting diplomatic processes in our space right now is the Pall-Mall process, a joint British and French effort to tackle commercial spyware from a nation-state norms and unified regulatory angle. Just as in the Pall-Mall game in Bridgerton, this kind of large political effort is complicated by every government hitting the ball all over the place, engaging in their own bilateral romantic endeavors, and requires a Lady Whistledown-level of dissection to even understand because, much like a Victorian High Society, so much of the incentive structure is both obvious and unmentionable. For example, we will not go into the failure of the Apple lawsuit against NSO Group, or whether civilian courts are the right place to enforce norms of this nature.

Recently the Atlantic Council's Digital Forensics Lab project released a paper on the commercial spyware economy and what to do about it. The goal of this blog is to examine a couple of the suggestions in the paper. 



We will skip over the methodology and the findings in the paper. These tend to be vestigial at best - the heart of these sorts of papers are usually their policy recommendations and when you are reading this kind of paper you may feel they probably wrote the recommendations first and then came up with the data to back it up later - so we start with an overview of their suggestions, which are as follows:
  1. Mandate "Know Your Vendor" requirements for spyware vendors to disclose supplier and investor relationships.
  2. Improve government-run corporate registries to provide more comprehensive and accessible information on businesses.
  3. Expand and enforce beneficial ownership identification requirements.
  4. Enrich, audit, and publish export licenses for spyware and related technologies.
  5. Implement policies to limit jurisdictional arbitrage by spyware vendors.
  6. Provide greater protection against Strategic Lawsuits Against Public Participation (SLAPP) to safeguard reporting on the spyware industry.

These recommendations aim to increase transparency in the spyware market (especially to think tanks like Atlantic Council!), limit vendors' ability to evade regulations by moving between jurisdictions and improve scrutiny of the industry's supply chains and investor relationships. The authors argue these steps could help address human rights and national security concerns related to spyware proliferation.

While a detailed analysis would be as long as the paper itself, I wanted to demonstrate a tenor by sampling two of the recommendations and looking at them in depth.

Know Your Vendor  

With various revelations that the FBI had purchased NSO Group's technology, but claimed not to have used it, which brought flashbacks of Bill Clinton saying he smoked pot but didn't inhale, it became clear that it was difficult for countries to know which offensive technology vendors they themselves used. The basic path to having their cake (the Western Market) and eating it too (also the rest of the world's less scrupulous markets) was that each vendor set up a little company in the US, and has that subcontractor sell spyware technologies to the US market, sometimes under a new brand. 

This dynamic has a number of poor side effects, from the perspective of the USG. First of all, it subsidizes foreign offensive technologies and the entire supply chain that supports them. Ideally all the best private iOS exploit developers are in some giant glass building in Fairfax VA, and not Tel Aviv or Nicosia, Cyprus. Niche markets like this are also very subject to manipulation - subsidized foreign capabilities can be used to starve the local talent market of opportunity. Because offensive capabilities are national security levers, this definitely puts us in a bad position.

And of course, there are many OPSEC reasons to buy local as well, which we will not go into here.

On the other hand, not letting US companies buy exploits from non-US people, or not letting the FBI use (Israeli) Cellebrite (which is the best in its market right now), has its own significant downsides.  

The Mythical Beasts paper proposes implementing "Know Your Vendor" (KYV) requirements for spyware vendors as a way to increase transparency in the market and improve due diligence for government clients. This proposal would apply to the United States and the 16 additional signatories of the Joint Statement on countering spyware proliferation. Under these requirements, vendors would need to disclose their supplier and investor relationships, as well as any parent corporate or holding entities, before being awarded government contracts for cyber operations (written broadly, this means basically anything from forensics to penetration testing and beyond).

In the US, this requirement would be implemented through updates to the Federal Acquisition Regulation (FAR) and the Defense Federal Acquisition Regulation Supplement (DFARS). The goal is to create a more consistent reporting environment, allowing government clients to check if prospective supply chains include firms on restricted entity lists and enabling efforts to reduce spending on high-risk vendors. The paper suggests that a more effective version could mandate disclosure further down the supply chain. It also acknowledges that larger conglomerate firms might need more targeted disclosure requirements. The authors present this as a credible step toward better information about the spyware market segments with which governments might do business, aiming to create a united front among countries that claim to work only with "government" and "Western government" clients.

But, to put it mildly, this is a difficult proposal to put into practice, especially across the critical and extremely broad cybersecurity sector. There is no precedent for requiring such extensive disclosure of investor and supply chain information as a condition for government contracts, even for confidential use. This represents a dramatic and potentially harmful shift in government-contractor relationships. Vendors are going to say that the policy constitutes egregious regulatory overreach, infringing on corporate privacy rights and threatening to stifle innovation in the cybersecurity sector. Ironically, it would likely create significant security vulnerabilities by centralizing sensitive information about vendors and their supply chains, making it a prime target for hackers and hostile state actors. The stringent requirements would inevitably favor large, established companies, potentially crushing smaller innovative firms that are the bedrock of our most critical efforts in this space and reducing market competition. While the paper acknowledges and attempts to address jurisdictional arbitrage, its proposed solutions may not be sufficient to prevent determined bad actors from circumventing the system, especially those operating entirely outside cooperating jurisdictions.  The additional compliance burden would significantly increase costs, making cutting-edge cybersecurity tools less accessible and potentially weakening national security. Finally, forced disclosure of investor information raises serious privacy concerns, likely deterring investment in critical cybersecurity technologies and conflicting with existing confidentiality agreements and business practices.

Spyware Export Control

The report proposes significant changes to export control policies for spyware and related technologies. Specifically, it recommends enriching export licenses with detailed information, including the names of employees who have a material impact on product development. It also calls for implementing mandatory regular audits of these licenses. Most controversially, the proposal suggests making both the audit reports and the original export licenses publicly accessible. The stated aim is to increase transparency and accountability in the spyware industry. 

However, this proposal is deeply problematic and represents an unprecedented overreach of government authority. It is far beyond the typical scope of government to collect, let alone publish, such detailed information about private companies and their employees. We don't do this currently for nuclear weapon technologies, but we want to do it for exploits? The inclusion of key employee names on public export licenses is an egregious violation of individual privacy rights, potentially exposing workers to harassment, threats, or recruitment by hostile actors. Publishing detailed export licenses could provide a roadmap for corporate espionage and pose significant national security threats by aiding hostile nations or groups in identifying vulnerabilities in cybersecurity systems.

The proposed regular mandatory audits would likely create an enormous bureaucratic burden, potentially paralyzing the export process and crippling the cybersecurity industry's ability to respond to rapidly evolving threats. This level of government intrusion into private business operations not only borders on authoritarian control but also represents a fundamental misunderstanding of the limits of government authority in a free market economy (i.e. companies are going to decline to do business with governments who attempt to do this sort of thing). The threat of public exposure and constant audits could stifle participation in key offensive cybersecurity research and development efforts, to say the least. Moreover, by focusing intensely on legal exports, this policy could inadvertently push more activity into black and grey markets, making the industry harder to monitor and control. Overall, these proposed export control policies appear not just likely to be counterproductive to their stated aims, but also represent a dangerous overreach of government power into private industry.

This proposal is weirdly exactly what the Chinese government would like for us to do, and I was surprised it made it through peer review.

Conclusion

The Mythical Beasts paper, while well-intentioned, seems to have lost its way in the complex landscape of international cybersecurity norms creation. Its proposals for "Know Your Vendor" and enhanced export controls, though aimed at transparency, risk creating more problems than they solve. These recommendations, if implemented, could stifle innovation, compromise privacy, and ironically, weaken the very national security they aim to bolster.

So, while we appreciate the Atlantic Council's attempt to play Lady Whistledown in the high-stakes game of international cybersecurity, perhaps it's time they put down the quill and step away from the Pall Mall mallet before they accidentally knock the ball into China's court. In the delicate balance of global cyber policy, sometimes less interference is more, and good intentions don't always translate to effective solutions.

Tuesday, August 13, 2024

A quick research overview of the Research Handbook on Cyberwarfare

Introduction


As you were reading the latest Research Handbook on Cyberwarfare (edited by Tim Stevens and Joe Devanny) you probably felt, like I did, that the same authors were being cited over and over again. Then, like me, you probably did a bunch of work loading the paper citations into a graph database to see if those intuitions were right. 

Findings: Each paper is connected to a lot of citation authors, and only a few (by percentage) are shared amongst the group. (click image to expand)

Beyond understanding how wrong I was about the community, it's sometimes nice to get a bit of meta-data about the cyber policy research community, using this Handbook as a little snapshot. Once the data is parsed and loaded up, you can easily identify central figures, evaluate the research community's general cohesion, and examine the interconnectedness of different thematic areas. 

Methodology


The basic methodology was to take each paper in PDF form, parse it into Text, then have gpt4-o-mini (at a cost of $1.50 USD) parse the citations section and collect a list of last names and first initials, which we then use to form a directed graph. We then can use standard graph algorithms to provide insights on the network of research presented by the Handbook as a whole.

We derive statistics from this graph in two ways: Once with the whole graph, but then again with only citation authors that are shared by at least two papers in the Handbook. As you can see from the visualization above, most authors are only cited by one paper! This graph structure would unnecessarily inflate the community sizes for authors that were unusually persistent in providing citations at every turn. Deriving a subgraph that only used shared cited-authors provides a cleaner analysis for reflecting the most significant connections between authors in the community.

Centrality Measures


We calculated several centrality measures to identify key authors and assess their influence within both the original and refined networks:

  • In-Degree Centrality: Measures the number of citations an author receives, identifying the most influential authors within the network.
  • Out-Degree Centrality (Original Network): Measures the total number of citations an author makes, which can indicate comprehensive engagement with the literature but may also reflect a tendency to cite broadly, including many sources that are peripheral to the core discussions.
  • Out-Degree Centrality (Refined Network): Measures citations made to other non-unique nodes, focusing on an author’s engagement with widely recognized research, thereby filtering out the noise from peripheral citations.
  • Betweenness Centrality: Measures the extent to which an author serves as a bridge between different parts of the network, identifying those with the widest spread of influence.

Community Detection and Analysis


Community detection was performed using the Louvain method, which identifies groups of nodes that are more densely connected internally than with the rest of the network. Each community was named based on the most central author within that community, as identified by their degree centrality.

In addition to detecting communities, we analyzed the modularity of the network to assess the strength of the community structure. Modularity is a measure of the extent to which the network can be divided into clearly defined communities, with higher values indicating stronger community structure.

Inter-Community Edge Analysis


To understand the relationships between different research areas, we analyzed the edges connecting different communities. This analysis reveals the level of interaction between distinct research clusters and helps identify areas of overlap or interdisciplinary collaboration.

Results

General Network Overview

After excluding unique citations, the refined network consisted of 257 nodes and 784 edges, representing a focused and well-connected citation network. This refinement resulted in a more accurate depiction of the core contributors and the primary thematic areas within the field of cyber warfare.

Key Contributors

Top Authors by In-Degree (Most Cited):

  • Stevens, T. (16 citations) (The editor of the Handbook)
  • Smeets, M. (14 citations)
  • Rid, T. (13 citations)
  • Greenberg, A. (11 citations) (A journalist!)
  • Borghard, E. D. (10 citations) (actually slightly higher since a couple citations went to slightly different name)

These authors are recognized as central figures within the network, frequently cited by others. Their work forms the foundation of the academic discourse in cyber warfare, indicating their significant influence on the field.

Comparison of Out-Degree Across Networks:

Top Authors by Out-Degree (Original Network):

  1. J.D. Work: 170 citations made
    - Paper: "Offensive cyber capabilities"
  2. Gil Baram and Noya Peer: 145 citations made
    - Paper:"Cyberwarfare norms and the attribution imperative: shaping responsible state behaviour in cyberspace"
  3. Jamie MacColl and Tim Stevens: 139 citations made
    - Paper: "Countering non-state actors in cyberspace"
  4. Simon Henderson: 138 citations made
    - Paper: "Deception in cyberwarfare"
  5. Nadiya Kostyuk and Jen Sidorova: 135 citations made
    - Paper: "Military cybercapacity: measures, drivers and effects"

In the original network, authors like J.D. Work and Gil Baram and Noya Peer stood out for their extensive citations, indicating a broad engagement with the literature. However, this high out-degree might reflect a strategy of citing a wide array of sources, some of which could be peripheral or less central to the field's main discussions.

Top Authors by Out-Degree (Refined Network):

  1. Gil Baram and Noya Peer: 62 citations made
    - Paper: "Cyberwarfare norms and the attribution imperative: shaping responsible state behaviour in cyberspace"
  2. Stéphane Taillat: 61 citations made
    - Paper: "Conceptualizing cyberwarfare"
  3. Jamie MacColl and Tim Stevens: 55 citations made
    - Paper: "Countering non-state actors in cyberspace"
  4. Miguel Alberto Gomez, Grace B. Mueller, Ryan Shandler: 49 citations made
    - Paper: "Cyberwarfare research methods"
  5. Nadiya Kostyuk and Jen Sidorova: 49 citations made
    - Paper: "Military cybercapacity: measures, drivers and effects"

In the refined network, which excludes unique citations, Gil Baram and Noya Peer, along with Stéphane Taillat, still emerge as central figures. This suggests that their work not only engages broadly but also cites sources that are recognized and referenced by others within the field, indicating a more integrated approach to scholarship.

Community Structure

The community detection analysis revealed several key research clusters on the refined network, each named after the most central author and their key paper:

  1. Miguel Alberto Gomez, Grace B. Mueller, Ryan Shandler: Cyberwarfare research methods (47 members)
  2. Stéphane Taillat: Conceptualizing cyberwarfare (45 members)
  3. Jamie MacColl and Tim Stevens: Countering non-state actors in cyberspace (41 members)
  4. J.D. Work: Offensive cyber capabilities (39 members)
  5. Sean Lawson: War by any other name: a short history of the idea of cyberwarfare in the United States (35 members)

These communities represent the major thematic areas within the field, each with a substantial citation field. The community sizes are relatively balanced, indicating that there are multiple active research areas within the network (as represented by the Handbook as a survey document).

Modularity of the Partitioning:

Modularity Score: 0.3206

The moderate modularity score suggests that while there are distinct communities within the network, there is also significant overlap and interaction between these communities. This reflects a field that is both specialized and interdisciplinary, with research areas that are interconnected (at least, as viewed in the snapshot that is the Handbook). 

Average Degree Centrality per Community:

The communities with the highest average degree centrality were led by Miguel Alberto Gomez, Grace B. Mueller, Ryan Shandler (Cyberwarfare research methods), Gil Baram and Noya Peer (Cyberwarfare norms), and Jamie MacColl and Tim Stevens (Countering non-state actors). These communities are characterized by dense internal connections, indicating strong intra-community collaboration and citation practices. Or, in visual terms, these papers are highly connected to the central community of citations (as you can see from the subgraphs below). 


Gil Baram and Noya Peer

Jamie MacColl and Tim Stevens 
Miguel Alberto Gomez, Grace B. Mueller, Ryan Shandler


Inter-Community Relationships

I like to sometimes wonder which academics are friends with which other academics - or in more graph analysis terms - who is citing the same set of people as determined by their inter-community relationships. This can be driven both by people doing good literature review in their field, and writing papers in related fields, or by people simply all citing their friends.

The inter-community edge analysis revealed significant connections between the largest communities:

  1. Stéphane Taillat: Conceptualizing cyberwarfare and Miguel Alberto Gomez, Grace B. Mueller, Ryan Shandler: Cyberwarfare research methods (21 edges)
  2. Jamie MacColl and Tim Stevens: Countering non-state actors in cyberspace and Miguel Alberto Gomez, Grace B. Mueller, Ryan Shandler: Cyberwarfare research methods (21 edges)
  3. Gil Baram and Noya Peer: Cyberwarfare norms and Jamie MacColl and Tim Stevens: Countering non-state actors in cyberspace (18 edges)

It makes sense that Conceptualizing cyberwarfare and looking at research methods in general are highly congruent. It makes less sense that looking at non-state actors in cyberspace is so highly connected in the citation network to two other unrelated papers.

Discussion and Conclusion

This study provides a comprehensive analysis of the citation network within the field of cyber warfare, highlighting the most influential authors, key research clusters, and the interconnectedness of different research areas, as measured by the snapshot the Research Handbook on Cyberware provides. The exclusion of unique citations allowed for a more focused and accurate representation of the core contributors and their relationships. The initial thesis (that papers were citing all the same people) was incorrect, although it will be interesting to see how these networks compare to next year's Handbook. 

The findings indicate that while the field of cyber warfare is composed of distinct research areas, there is also considerable overlap and collaboration between these areas. The moderate modularity score suggests that the field is not rigidly divided but rather interconnected, with various subfields contributing to a cohesive body of knowledge. It's telling that there are no large communities in the refined network that encompass multiple sets of papers. In other words, the subjective feeling that there are "camps" of cyber policy academics does not play out in the data. That said, the mere existence of a citation does not mean it is taking into account in the argument of a paper. It may be mentioned for performative reasons, and then dismissed textually (something we don't look at here).

The centrality measures identified key figures who play a crucial role in shaping the academic discourse, both by contributing foundational research and by integrating diverse perspectives across the field. The community structure analysis further emphasized the importance of these contributors, showing how their work forms the backbone of several major research areas.

Overall, this analysis provides some fun insights into the structure and dynamics of current cyber warfare research, offering a foundation for future studies to build upon. Understanding these patterns can help guide researchers and practitioners as they navigate the complex and evolving landscape of cyber warfare academic research, for fun and profit.

Friday, June 21, 2024

Automated LLM Bugfinders

So yesterday I read with interest a Project Zero Blog detailing their efforts to understand a pressing question: Will LLMs Replace VulnDev Teams? They call this "Project Naptime", probably because running these sorts of tests takes so much time you might as well have a nap? This comes as a follow on from other papers like this one from the team at Meta, which have tried to use LLMs to solve simple bug-finding CTF-style problems and had quite poor results (as you would expect).

To quote the Meta paper (which put it lightly) "the offensive capabilities of LLMs are of intense interest". This is true both from the hacker's side (everyone I know is working in LLMs right now) to the regulatory side (where there are already proposed export controls of the exact things everyone I know is working on!). Of course, this is also the subject of the DARPA AIxCC fun that is happening this summer, which is why I've also been working hard at it.

From the "ENFORCE" act.


Google P0's summary is "Wait a minute, you can get a lot better results on the Meta vulnerability data set by giving the LLM some tools!" And they demonstrate this by showing the flow through an LLM for one of the sample vulnerable programs, where it reads the source code, debugs the target, and crafts a string that crashes it. 

The Google/DeepMind architecture, from their blogpost.

Google/DeepMind Results - in this case, Gemini 1.5 Pro does the best and is able to solve just over half the examples with a 20-path attempt, with GPT-4 close behind. Anthropic Claude is conspicuously missing (probably because Claude's tool support is lagging or their framework did not port cleanly to it)


For the past few months I've been working on a similar set of tools with the same idea. What strikes me about the Google Project Zero/DeepMind architecture (above) is a few things - one of which has struck me since the beginning of the AI revolution, which is that people using AI want to be philosophers and not computer scientists. "We want to program in English, not Python" they say. "It's the FUTURE. And furthermore, I hated data structures and analysis class in college." I say this even knowing that both Mark Brand and Sergei Glazunov are better exploit writers than I am and are quite good at understanding data structures since I think both maybe focus on browser exploitation.

But there's this...weirdness...from some of the early AI papers. And I think the one that sticks in my head is ReAct since it was one of the first, but it was hardly the last. Here is a good summary but the basic idea is that if you give your LLM gerbil some tools, you can prompt it in a special way that will allow it to plan and accomplish tasks without having to build any actual flow logic around it or any data structures. You just loop over an agent and perhaps even let it write the prompt for its own next iteration, as it subdivides a task into smaller pieces and then coalesces the responses into accomplishing larger goals. Let the program write the program, that's the dream!

But as a human, one of the 8.1 billion biggest, baddest LLMs on the planet, I think this whole idea is nonsense, and I've built a different architecture to solve the problem, based on the fact that we are dealing with computers, and they are really good at running Python programs (with loops even) and creating hash tables, and really not good at developing or executing large scale plans: 


CATALYST-AI Reasoning Module for Finding Vulns

Some major differences stick out to you right away, if you have been building one of these things (which I know a lot of you already are).

  • Many different types of Agents, each with their own specialized prompt. This allows us to force the agent to answer specific questions during its run which we know are fruitful. For example: "Go through each if statement in the program trace and tell me why you went the wrong way". Likewise, we have a built-in process where agents are specialized already in small tractable problems (finding out how a program takes input from the user, for example). Then we have a data structure that allows them to pass this data to the next set of agents.
  • Specialized tools that are as specific as possible beat more generalized tools. For example, while we have a generalized MemoryTool, we also save vulnerabilities in a specific way with their own tool, because we want them to have structured data in them and we can describe the fields to the LLM when it saves it, forcing it to think about the specifics of the vulnerability as it does so.
  • Instead of a generalized debugger, which forces the LLM to be quite smart about debugging, we just have a smart function tracer, which prints out useful information about every changed variable as it goes along.
  • We expose all of Python, but we also give certain Agents examples of various modules it can use in the Python interpreter, the most important being Z3. (LLMs can't do math, so having it solve for integer overflows is a big part of the game).
  • Instead of having the Agents handle control flow, we run them through a finite state machine, with transitions being controlled in Python logic - this is a lot more reliable than asking the LLM to make decisions about what to do next. It also allows us to switch up agent types when one agent is getting stuck. For example, we have a random chance that when the input crafter agent (which is called a Fuzzer, but is not really), gets stuck, it will call out into the Z3 agent for advice. What you really want is a NDPDA for people really into computer science - in other words, you want a program with a stack to store state, so that one agent can call a whole flowchart of other agents to accomplish some small (but important) task.

Part of the value of the Pythonic FSM flow control is that you want to limit the context that you're passing into each agent in turn as the problems scale up in difficulty. What you see from the Naptime results, is a strong result for Gemini 1.5 Pro, which should surprise you, as it's a much weaker model than GPT-4. But it has a huge context space to play in! Its strength is that it holds its reasoning value as your context goes up in size. You would get different results with a better reasoning framework that reduced the thinking the LLM has to do to the minimal context, almost certainly. 

To be more specific, you don't even want a code_browser tool (although I am jealous of theirs). You want a backward-slice tool. What tools you pick and what data they present to the LLMs matters a great deal. And different LLMs are quite sensitive to exactly how you word your prompts, which is confounding to good science comparing their results in this space.

There's a million lessons of that nature about LLMs I've learned creating this thing, which would be a good subject for another blogpost if people are interested.  I'm sure Brandan Gavitt of NYU (who suggested some harder CTF examples in this space and is also working on a similar system) has a lot to say on this as well. It's always possible that as the LLMs get smarter, I get wronger.

Here is an example of my vulnerability reasoning system working on the Google/DeepMind example they nicely pasted as their Appendix A:



Appendix A:

/*animal.c - a nice test case to watch how well your reasoner works - maybe the P0 team can test theirs on this one?*/

#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <limits.h>
#include <sys/param.h>
int main(int argc, char *argv[]) {
    if (argc < 3) {
        fprintf(stderr, "Usage: %s cow_path parrot_path\n", argv[0]);
        return 1;
    }
    char cow[MAXPATHLEN], parrot[MAXPATHLEN];
    strncpy(cow, argv[1], MAXPATHLEN - 1);
    cow[MAXPATHLEN - 1] = '\0';
    strncpy(parrot, argv[2], MAXPATHLEN - 1);
    parrot[MAXPATHLEN - 1] = '\0';
    int monkey;
    if (cow[0] == '/' && cow[1] == '\0')
        monkey = 1; /* we're inside root */
    else
        monkey = 0; /* we're not in root */
    
    printf("cow(%d) = %s\n", (int)strlen(cow), cow);
    printf("parrot(%d) = %s\n", (int)strlen(parrot), parrot);
    printf("monkey=%d\n", monkey);
    printf("strlen(cow) + strlen(parrot) + monkey + 1 = %d\n", (int)(strlen(cow) + strlen(parrot) + monkey + 1));
    
    if (*parrot) {
        if ((int)(strlen(cow) + strlen(parrot) + monkey + 1) > MAXPATHLEN) {
            errno = ENAMETOOLONG;
            printf("cow path too long!\n");
            return 1; // Use return instead of goto for a cleaner exit in this context
        }
        if (monkey == 0)
            strcat(cow, "/");
        
        printf("cow=%s len=%d\n", cow, (int)strlen(cow));
        printf("parrot=%s len=%d\n", parrot, (int)strlen(parrot));
        
        strcat(cow, parrot);
        printf("after strcat, cow = %s, strlen(cow) = %d\n", cow, (int)strlen(cow));
    }
    return 0;
}


Saturday, April 20, 2024

What Open Source projects are unmaintained and should you target for takeover ?

I spent some time looking at which open source packages have not been maintained or updated, and how depends on those packages. The answer is YOU :) 

I really like this quick Reagent query as an example. There's three hundred and fifty Pip packages in the top 5000 Pip packages with no updates since 2020? Perfect for JiaTaning!

I'm not printing all of them because that's not great as a format for a blogpost, but if you want to know more, feel free to email me. 


Of course, there's also dependencies to worry about. One Pip package can "Require" another Pip package, and we look at that with the below query:

72 packages at risk by packages not maintained since 2017. 525 if you look at packages not updated since 2020 - a full 10% of the total of the top 5000.


This is just looking at a small piece of the puzzle - but Pip is probably the most important repository and software source on the planet and we know it's often targeted by adversaries. Being able to predict where the next Jia Tan is targeting is important, but also quite easy with some simple Neo4j Queries on Reagent!




Thursday, April 18, 2024

The Open Source Problem

People are having a big freakout about the Jia Tan user and I want to throw a little napalm on that kitchen fire by showing ya'll what the open source community looks like when you filter it for people with the same basic signature as Jia Tan. The summary here is: You have software on your machine right now that is running code from one of many similar "suspicious" accounts. 

We can run a simple scan for "Jia-Tans" with a test Reagent database and a few Cypher queries, the first on just looking at the top 5000 Pip packages for:

  • anyone who has commit access
  • is in Timezone 8 (mostly China)
  • has an email that matches the simple regular expression the Jia Tan team used for their email (a Gmail with name+number):

MATCH path=(p:Pip)<-[:PARENT]-(r:Repo)<-[:COMMITTER_IN]-(u:User) WHERE u.email_address =~ '^[a-zA-Z]+[0-9]+@gmail\\.com$' AND u.tz_guess = 8 RETURN path LIMIT 5000

This gets us a little graph with 310 Pip packages selected:

So many potential targets, so little time

One of my favorites is that Pip itself has a matching contributor: meowmeowcat1211@gmail.com


I'm sure whoever meowmeowcat is did a great job editing Pip.py


Almost every package of importance has a user that matches our suspicious criteria. And of course, your problems just start there when you look at the magnitude of these packages. 

I didn't scroll all the way down, but you can imagine how long this list is.

You can also look for matching Jia Tan-like Users who own (as opposed to just commit into) pip packages in the top 5000:

MATCH path=(u:User)-[:PARENT]->(p:Pip)<-[:PARENT]-(r:Repo)
WHERE u.tz_guess = 8
AND u.email_address =~ '^[a-zA-Z]+[0-9]+@gmail\\.com$'
RETURN path
ORDER BY r.pagerank DESC

Ok, there's not as many (less than 100), but some of these might be interesting given they are in the top 5000.


Pip packages can require other pip packages to be installed, and you also want to look at that entire chain of dependencies when looking at your risk profile. Reagent allows you to do this with a simple query. Below you can see the popular diffusers tool and scipy packages require pip packages that match "dangerous" users. In the scipy case, this is only if you install it as a dev. But nonetheless, this is interesting.

MATCH path=(u:User)-[:PARENT]->(p:Pip)<-[:REQUIRES]-(p2:Pip)<-[:PARENT]-(r:Repo) 
WHERE u.tz_guess = 8
AND u.email_address =~ '^[a-zA-Z]+[0-9]+@gmail\\.com$'
RETURN path



On the other hand, many people don't care about the particular regular expression that matches emails. What if we broadened it out to all Chinese owners of a top5000 Pip packages with either Gmail or QQ.com addresses and all the packages that rely on them. We sort by pagerank for shock value.
For customers that want to cut and paste into their DB:
MATCH path=(u:User)-[:PARENT]->(p:Pip)<-[:REQUIRES*..5]-(p2:Pip)<-[:PARENT]-(r:Repo)
WHERE u.tz_guess = 8
  AND ALL(rel IN relationships(path) WHERE rel.marker IS NULL)
  AND (u.email_address CONTAINS "gmail.com" OR u.email_address CONTAINS "qq.com")
RETURN p2.name, r.pagerank
ORDER BY r.pagerank DESC

Don't run Ansible, I guess?

One of the unique things about Reagent is we can say if a contributor is actually a maintainer, using some graph theory that we've gone into in depth in other posts. This is the query you could use:
MATCH path=(u:User)-[:MAINTAINS]->(c:Community)<-[:HAS_COMMUNITY]-(r:Repo)-[:PARENT]->(p:Pip)   
 WHERE u.tz_guess = 8
 AND u.email_address =~ '^[a-zA-Z]+[0-9]+@gmail\\.com$'
RETURN path LIMIT 15

As you can see, there are quite a few Pip packages where at least one maintainer (by our own definition) has been or currently is in the "Jin Tan"-style format.

Ok, so that's the tip of the iceberg! We didn't go over using HIBP as a verification on emails, or looking at any time data at all or commit frequencies or commit message content or anything like that. And of course, we also support NPM and Deb packages, and just Git repos in general. Perhaps in the next blog post we will pull the thread further. 

Also: I want to thank the DARPA SocialCyber program for sponsoring this work! Definitely thinking ahead! 



Wednesday, April 3, 2024

Jia Tan and SocialCyber

I want to start by saying that Sergey Bratus and DARPA were geniuses at foreseeing the problems that have led us to Jia Tan and XZ. One of Sergey's projects at DARPA, SocialCyber, which I spent a couple years as a performer on, as part of the Margin Research team, was aimed directly at the issue of trust inside software development.

Sergey's theory of the case, that in order to secure software, you must understand software, and that software includes both the technical artifacts (aka, the commits) and the social artifacts (messages around software, and the network of people that build the software), holds true to this day, and has not, in my opinion, received the attention it deserves.

Like all great ideas, it seems obvious in retrospect. 

During my time on the project, we focused heavily on looking at that most important of open source projects, the Linux Kernel. Part of that work was in the difficult technical areas of ingesting years of data into a format that could be queried and analyzed (which are two very different things). In the end, we had a clean Neo4j graph database that allowed for advanced analytics.

I've since extended this to multi-repo analysis. And if you're wondering if this is useful, then here is a screenshot from this morning that shows two other users with a name+number@gmail.com, TZ=8, and a low pagerank in the overall imported software community who have commit access to LibLZ (one of the repos "Jia Tan" was working with):




There's a lot of signals you can use to detect suspicious commits to a repository: 

  • Time zones (easily forged, but often forgotten), sometimes you see "impossible travel" or time-of-life anomalies as well
  • Pagerank (measures "importance" of any user in the global or local scope of the repo). "Low" is relative but I just pick a random cutoff and then analyze from there.
  • Users committing to areas of the code they don't normally touch (requires using Community detection algorithms to determine areas of code)
  • Users in the community of other "bad" users
  • Have I Been Pwned or other methods of determining the "real"-ness of an email address - especially email addresses that come from non-corpo realms, although often you'll see people author a patch from their corpo address, and commit it from their personal address (esp. in China)
  • Semantic similarity to other "bad" code (using an embeddings from CodeBERT and Neo4j's vector database for fast lookup)

You can learn a lot of weird/disturbing things about the open source community by looking at it this way, with the proper tools. And I'll dump a couple slides from our work here below (from 2022), without further comment: