Researchers have developed a new method called RAW (Refusal Aversion in the Wild) to measure the impact of Large Language Model (LLM) refusals on user engagement. This method leverages the inherent stochasticity of LLMs to create a natural experiment using existing conversation logs, allowing for causal measurement without the need for constrained A/B testing. Applied to 15 deployed LLMs across LMSYS-Chat-1M and WildChat datasets, the study found that approximately half of the measured models experienced a 3 to 17 percentage point drop in subsequent user re-engagement following a refusal. The remaining models showed no discernible effect, with one case attributed to persistent users retrying refused requests. AI
IMPACT Provides a scalable method to quantify user-side costs of LLM refusals, enabling better product design and user experience.
RANK_REASON Academic paper introducing a new measurement method for deployed LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →