New framework SWE-IF evaluates LLMs on code instruction following

Researchers have introduced SWE-IF, a new evaluation framework designed to assess Large Language Models' (LLMs) ability to follow code instructions beyond just functional correctness. This framework includes a taxonomy of 30 verifiable code instructions and deterministic verifiers, aiming to capture the 'vibe check' that reflects human preference for clean, intent-preserving, and correct code. Evaluations of 31 LLMs revealed that instruction following is a key differentiator, with a composite score of functional correctness and instruction following correlating best with human preference. AI

IMPACT This new evaluation framework could lead to LLMs that generate more human-aligned and maintainable code, improving developer productivity.

RANK_REASON The cluster contains an academic paper introducing a new evaluation framework for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• LLMs
• Ming Zhong
• SWE-IF
• VeriCode

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →