Research paper analyzes optimal multitask representations under information constraints

A new research paper analyzes the optimal pretraining objectives for multitask representations under an information budget. The study formalizes this as a multitask information bottleneck, using a mutual information constraint to model incomplete information retention. Experiments on CelebA and dSprites datasets demonstrate that compression can alter which pretraining objective performs best, and task-aware training can even reverse the preservation order of downstream tasks as the information rate increases. AI

IMPACT Provides theoretical insights into optimizing AI model representations for various downstream tasks under computational constraints.

RANK_REASON The cluster contains a research paper published on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX
• Celeba
• DagsHub
• dSprites
• Gotit.pub
• Hugging Face
• IArxiv
• information bottleneck
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →