Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation
Researchers have developed a new method for generating query-specific rubrics to evaluate long-form reports, addressing the challenge of creating detailed and scalable assessment tools. This pipeline trains rubric generators using human preferences and reinforcement learning, incorporating rewards for preference consistency, format validity, and LLM-based rubric evaluation. The learned rubrics demonstrated superior performance in distinguishing preferred reports and significantly improved the training of report generation systems within both single-agent and multi-agent frameworks. AI
IMPACT This research introduces a novel approach to improve the evaluation and generation of long-form AI-generated reports, potentially enhancing the quality and reliability of AI writing tools.