OmniXRef framework enables multimodal video editing with diverse references

Researchers have introduced OmniXRef, a novel framework designed for multimodal video editing. This system allows users to combine various references, including text instructions, image appearances, and video movements, to precisely control video attributes. OmniXRef utilizes a unified input design with a reference-bank interface and a diffusion transformer model to process source videos and multimodal context from large language models. To overcome the limitations of existing training data, a new dataset, OmniXRef-Data, was constructed using synthesis pipelines and curriculum learning, and a benchmark, OmniXRef-Bench, was developed for systematic evaluation of video-based editing controls. AI

IMPACT This framework could enhance creative control in video production by enabling more intuitive and flexible editing processes.

RANK_REASON The item describes a new research paper detailing a novel framework and model for video editing. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• Connected Papers
• DagsHub
• Gotit.pub
• Hugging Face
• Litmaps
• multimodal large language model
• OmniXRef
• OmniXRef-Bench
• OmniXRef-Data
• OmniXRef-Model
• R2-RoPE
• ScienceCast
• scite Smart Citations

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →