A new evaluation framework called HireWatch has been developed to assess how well Large Language Models (LLMs) adhere to U.S. employment laws when generating interview questions. The framework utilizes a dataset of 200 lawful and unlawful questions, derived from federal statutes and agency guidance, to test LLMs across various job contexts. Initial evaluations of 18 LLMs demonstrated that their compliance with these laws is fragile, with violations increasing significantly under pressure, although explicit instructions to follow the law can mitigate but not eliminate these issues. AI
IMPACT Highlights potential legal risks in AI-driven hiring tools, necessitating careful development and oversight.
RANK_REASON The cluster contains an academic paper introducing a new evaluation framework for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →