Researchers have introduced new metrics—Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR)—to better evaluate the abstractiveness of text summarization models. These metrics aim to quantify how much a generated summary deviates from simply copying source text, moving beyond traditional measures like ROUGE. Empirical validation on the XSum dataset using models like BART-large-cnn and Pegasus-xsum showed that these metrics can effectively distinguish between extractive and abstractive summarization approaches, with the Abstraction Ratio also flagging potential hallucinations. AI
IMPACT These new metrics could lead to more accurate evaluations of summarization models, driving improvements in their ability to generate concise and non-hallucinatory summaries.
RANK_REASON The cluster contains a research paper detailing new metrics for evaluating text summarization models.
- Abstraction Ratio
- arXiv
- BART-large-cnn
- DistilBart
- mT5-small
- Pegasus-xsum
- Praveenkumar Katwe
- Reference Abstraction
- Summary Abstraction
- XSum dataset
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
