New benchmark UNIKIE-BENCH evaluates large multimodal models for document information extraction

Researchers have introduced UNIKIE-BENCH, a new benchmark designed to systematically evaluate the performance of Large Multimodal Models (LMMs) in extracting key information from visual documents. The benchmark features two tracks: one for constrained-category KIE with predefined schemas and another for open-category KIE. Experiments using 15 state-of-the-art LMMs highlighted significant performance drops when dealing with varied schemas, long-tail information, and complex layouts, indicating ongoing challenges in accuracy and reasoning for LMMs in this domain. AI

IMPACT Provides a standardized evaluation framework for LMMs in document information extraction, highlighting current limitations.

RANK_REASON This is a research paper introducing a new benchmark for evaluating LMMs.

Read on arXiv cs.CV →

• LMMs
• UNIKIE-BENCH
• Zhenghao Liu
• Large Multimodal Models

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →