New research highlights limitations in conformal novelty detection methods

Researchers have identified fundamental limitations in existing novelty detection methods that use conformal p-values and the Benjamini-Hochberg procedure. These methods, while offering distribution-free control over the global false discovery rate, fail to reliably detect novelty at the decision boundary. The study introduces the concept of boundary false discovery rate (bFDR) and demonstrates that a previously proposed support line (SL) procedure, effective in continuous independent frameworks, does not control bFDR in conformal settings. The paper proposes several modifications to the SL procedure to restore reliability at the decision boundary by controlling bFDR, including adaptive procedures for situations with many expected novelties and subsampled procedures for small calibration samples. AI

IMPACT This research could lead to more reliable AI systems that can better identify novel or unexpected data points, crucial for applications requiring robust decision-making.

RANK_REASON Academic paper detailing methodological improvements in statistical analysis. [lever_c_demoted from research: ic=1 ai=0.7]

Read on arXiv stat.ML →

• arXiv
• Zijun Gao

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →