Recent research from Off-by-1 Labs at 1Password found that 54% of LLM-generated patches failed to resolve the target vulnerability, added a new vulnerability, or both.
Additionally:
- 33%+ of AI-generated patches that fix a vulnerability are fragile.
- 20% of AI-generated patches change the underlying application logic when they successfully fix a vulnerability.
- Only 17% of AI-generated patches fix the underlying vulnerability in a robust manner.
According to a blog post by Keith Hoodlet, researchers produced 6,080 patches across six recently disclosed CVEs in open source code, using “two frontier, cyber-capable reasoning models.” Overall, Hoodlet says, “the average success rate for generating a patch that fully resolved the vulnerability (without materially changing application behavior) was just 26.0%.”
Application behavior changes seen in the patches included “reimplementing file-local parsers, changing “allow list” logic to “deny list” logic, and other similar changes.”
Through this research, 1Password aims to provide "the tooling and methodology necessary to improve vulnerability remediation outcomes at scale.” Thus, they have released the FLAWED (Fix Like Artifacts with Embedded Defects) research harness for measuring how well AI agents patch vulnerabilities, along with pre-built datasets and an in-depth research paper.
Learn more at 1Password.
