An Attribute-Based Measure of Video Complexity
Researchers have introduced VideoABC, a new framework designed to measure the complexity of video-question pairs for video-LLMs. This non-parametric measure utilizes a vocabulary of video attributes, such as scene complexity and event speed, to estimate the probability of a video-LLM failing on a given input. VideoABC combines k-means and universal lattice quantizers to ensure accurate estimates and generalization, even with limited reference data. Experiments demonstrate its effectiveness in outperforming other methods while providing explainable insights into benchmark complexity. AI
IMPACT Provides a new metric for evaluating and understanding the performance limitations of video-LLMs.