Robustness to nuisance perturbations enables unsupervised evaluation of single-cell foundation models

In plain English

The benchmark asks whether a cell embedding stays locally consistent when small amounts of measurement noise or sampling variation are introduced.

What they found

• Models differed nearly twofold in local-neighborhood preservation after 5% of counts were discarded.
• Cluster stability correlated with established bioconservation metrics at Spearman rho 0.78.

The catch

• The record is a preprint and has not completed peer review.
• This summary is based on the abstract; methods and supplementary analyses were not independently rechecked.

Methods at a glance

Study type

Unsupervised robustness benchmark

Biological system

Single-cell transcriptomic datasets

Organism

Multiple datasets

Dataset

39 single-cell datasets

Sample size

Five foundation models, one PCA baseline, and 39 datasets

Model architecture

Model-agnostic nuisance-perturbation evaluation

Training objective

No additional training; evaluate embedding stability

Code

Not reported in the abstract

Data

Uses 39 datasets; identifiers were not listed in the abstract.

Tasks and metrics

Tasks: Neighborhood preservation; Cluster stability

Metrics: Local-neighborhood preservation; Spearman correlation with bioconservation metrics

Baselines: PCA; Standard downstream benchmarks

Principal findings

• A 5% count perturbation exposed nearly twofold differences among models ranked similarly by standard benchmarks.

What remains unresolved

External validation: Tested across 39 datasets

• The record is a preprint and has not completed peer review.
• This summary is based on the abstract; methods and supplementary analyses were not independently rechecked.