LLM 漏洞检测在真实代码上表现不佳；SWE-bench 需要 90% 的任务才能保证可靠性

报道来源 [2]

1. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] · 2026-07-17 21:05

   LLM vulnerability detection hits 100% recall, then collapses on real code Structural priors lift LLM vulnerability recall from 20% to 100% on synthetic benchmar

   LLM vulnerability detection hits 100% recall, then collapses on real code Structural priors lift LLM vulnerability recall from 20% to 100% on synthetic benchmarks, but real CVE data exposes a 51-point collapse. https://www. notatechguy.com/llm-vulnerabil ity-detection-hits-100-re…

   链接 notatechguy.com/llm-vulnerability-detecti…
2. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] · 2026-07-17 21:00

   SWE-bench needs 90% of tasks for reliable agent benchmark results A replay analysis of three LLM agent benchmarks finds the safe partial-run fraction ranges fro

   SWE-bench needs 90% of tasks for reliable agent benchmark results A replay analysis of three LLM agent benchmarks finds the safe partial-run fraction ranges from 15% to over 95%, with no universal shortcut. https://www. notatechguy.com/swe-bench-need s-90-of-tasks-for-reliable-ag…

   链接 notatechguy.com/swe-bench-needs-90-of-tas…