Scaling Laws for Masked-Reconstruction Transformers on Single-Cell Transcriptomics
Researchers have investigated scaling laws for masked-reconstruction transformers applied to single-cell transcriptomics data. Their study, using data from the CELLxGENE Census, found that in data-rich environments, these models exhibit power-law scaling similar to language and vision transformers. However, in data-limited scenarios, scaling behavior was negligible, suggesting data availability is a key constraint. The findings imply that scaling laws are applicable to genomics when sufficient data is present and highlight the importance of the data-to-parameter ratio. AI
IMPACT Establishes that AI scaling principles apply to biological data, potentially guiding future AI model development for genomics.