Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challenges
Researchers are exploring privacy risks associated with large language models (LLMs) and their adaptations. One study focuses on detecting sensitive personal information in Japanese pre-training corpora, developing a classifier for special care-required personal information (SCPI) under Japan's APPI. Another paper investigates privacy vulnerabilities in multi-modal LLMs, highlighting how they can leak sensitive data from images and memory, and introduces a dataset for evaluation. A third study benchmarks the effectiveness of differential privacy (DP) in adapting LLMs, finding that data distribution shifts significantly impact privacy risks, with parameter-efficient fine-tuning methods like LoRA offering better protection for out-of-distribution data. AI
IMPACT These studies highlight critical privacy challenges in LLMs, informing developers on data handling, multi-modal risks, and effective privacy protection techniques during model adaptation.
- LoRA
- Large Language Models
- Differential Privacy
- Multi-modal Large Language Models
- Differential Privacy (DP)
- Act on the Protection of Personal Information (APPI)
- Japanese Pre-Training Corpora
- Multi-modal Large Language Models (MLLMs)
- MM-Privacy dataset
- Special Care-Required Personal Information (SCPI)