FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
Researchers have developed FATHOMS-RAG, a new benchmark designed to evaluate the end-to-end performance of retrieval-augmented generation (RAG) systems. This framework assesses a RAG pipeline's ability to ingest, retrieve, and reason across various data modalities including text, tables, and images. The study found that closed-source RAG pipelines generally outperform open-source ones, particularly when dealing with complex multimodal and cross-document information. AI
IMPACT Introduces a new evaluation framework for multimodal RAG systems, potentially driving improvements in their accuracy and reducing hallucinations.