Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
A new analysis of AI social impact evaluations reveals significant gaps in current reporting practices. The study examined 186 first-party release reports and 248 third-party sources, finding that while independent evaluators offer more comprehensive assessments of bias and harmful content, first-party reports are often superficial and declining in detail. Developers deprioritize disclosures on data provenance, labor costs, and infrastructure unless mandated by compliance or product adoption, highlighting a critical need for policies that enforce developer transparency and bolster independent evaluation ecosystems. AI
IMPACT Highlights the need for standardized and transparent reporting on AI's societal risks, potentially influencing future governance and development practices.