Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities
Researchers have developed a new training strategy for multimodal semantic segmentation that addresses the challenge of missing sensor modalities. This method learns to sample modality availability scenarios directly from a pretrained latent space, rather than relying on random dropout. By quantifying the impact of each scenario on the shared latent representation and using a kernel smoothing technique, the strategy refines scenario scores to create a probability distribution for fine-tuning. Experiments on remote sensing datasets demonstrated that this approach outperforms standard fine-tuning and LoRA-based adaptation. AI
IMPACT Enhances robustness of AI models in real-world scenarios with incomplete data, potentially improving performance in remote sensing and other multimodal applications.