The RAGAS evaluation framework has undergone significant changes, renaming key fields like `ground_truth` to `reference` and `question` to `user_input`. These changes, along with a shift in data schema from row-oriented to columnar, can cause existing evaluation datasets to fail. Users need to migrate their datasets, ensuring contexts are lists of strings and all arrays maintain consistent lengths to avoid silent data corruption. The distinction between `reference_contexts` (gold chunks) and `retrieved_contexts` (system output) is also critical for accurate retrieval metric calculation. AI
IMPACT Developers using RAGAS for LLM evaluation must update their datasets and code to accommodate recent field renames and schema changes.
RANK_REASON The item describes a change in a specific software tool (RAGAS) and provides guidance on how to adapt to it, rather than a new release or significant industry event.
- ground truth
- JSON
- langsmith
- Pandas
- Ragas
- reference_contexts
- retrieved_contexts
- TestsetGenerator
- TruLens
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →