Mozilla said the 271 vulnerabilities discovered by Mythos had "virtually no false positives."

The infidelity was tangible when the CTO of Mozilla last month stated that the AI-based vulnerability detection revealed that “zero days are counted” and “defense finally have a chance to win decisively.” Finally, it seemed to be part of an all too familiar pattern: Select a handful of impressive AI-rich results, leave the small print that could draw a more differentiated picture, and let the hype train roll on.

In view of the scepticism, Mozilla gave a look behind the scenes of his use of Anthropic Mythos – an AI model for identifying software vulnerabilities – to track 271 Firefox vulnerabilities within two months. In a post, the Mozilla engineers said that the finally ready-to-use breakthrough they had achieved was primarily the result of two things: (1) the improvement of the models themselves and (2) Mozilla's development of a custom “cable tree” that Mythos supported in analyzing the Firefox source code.

‘No false alarms’

The engineers said that their previous work with AI-assisted vulnerability detection had been “unwanted crap”. Usually someone asks a model to analyze a code block. The model would then create plausibly readable error reports, often to an unprecedented extent. However, if human developers continue to investigate, they found that a large percentage of the details was hallucinated. People would then have to invest considerable work in order to process the vulnerabilities in an old-fashioned way.

Mozilla's work with myth is different, said Mozilla Distinguished Engineer Brian Grinstead in an interview. The biggest distinction factor was the use of an agent-hardness, a code piece that LLM to conduct it through a number of specific tasks. In order for such a cable harness to be useful, considerable resources are required to adapt it to the project-specific semantics, tools and processes for which it is to be used.

Grinstead described the cable harness created by his team as “the code that drives the LLM to achieve a goal. He gives instructions to the model (e.g. “Find a bug in this file”), provides tools to him (for example, allows him to read/write files and evaluate test cases) and then executes it in a loop until it is finished.” Using Mythos enabled access to the same tools and pipelines that use human Mozilla developers, including the special Firefox build they use to test.