New BAT system uses LLMs for spatial sound reasoning

Researchers have developed BAT, a system that combines a binaural acoustic scene analysis model with a large language model (LLM) to enable reasoning about spatial sounds. To facilitate this, they created a new dataset by synthesizing spatial audio from AudioSet and SoundSpaces 2.0, and developed SpatialSoundQA, a question-answering dataset for spatial sound perception. The system's acoustic front end, the Spatial Audio Spectrogram Transformer (Spatial-AST), demonstrates strong performance in sound event detection, localization, and distance estimation. When integrated with the LLaMA-2 7B model, BAT shows superior capabilities in both spatial sound perception and reasoning. AI

IMPACT This research demonstrates the potential for LLMs to enhance spatial audio perception and interpretation, opening new avenues for AI in understanding complex acoustic environments.

RANK_REASON The cluster describes a new research paper detailing a novel system and dataset for spatial sound reasoning using LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• AudioSet
• large-language models
• LLaMA-2 7B
• SoundSpaces 2.0
• Spatial-AST
• Spatial Audio Spectrogram Transformer
• SpatialSoundQA
• Zhisheng Zheng

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →