Prompt layout drastically alters LLM code analysis, experiment finds

A recent experiment tested how prompt layout affects the responses of small language models, specifically Gemma 2-2B-it and Qwen2.5-Coder-1.5B-Instruct. The study found that simply moving the question before the code snippet could deterministically flip the model's verdict on code defects. The Qwen2.5-1.5B model consistently answered "YES" to all queries, scoring 50% accuracy, while the Gemma 2-2B model showed minimal improvement over this baseline. Notably, neither model abstained from answering even when instructed to use "UNSURE," indicating a lack of confidence signaling. AI

IMPACT Demonstrates that prompt engineering significantly impacts LLM performance, highlighting the need for careful prompt design and validation.

RANK_REASON The item details an experiment evaluating the performance and stability of small language models on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Gemma 2-2B-it
• llama.cpp
• Qwen2.5-Coder-1.5B-Instruct

• model release
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →