Embedding Anisotropy in Medical Image Foundation Models: Post Hoc Whitening Degradation and Spectral-Aware Standardization.

Authors

Abstract

Foundation models produce high-dimensional embedding representations whose geometric properties fundamentally determine downstream task effectiveness. Embedding anisotropy-the concentration of representations in a narrow cone-has been documented in natural language processing but not systematically studied in vision foundation models. We characterize anisotropy in embeddings from eight architecturally diverse models (ResNet50, DINOv2-S/L, BiomedCLIP, PLIP, SigLIP, UNI, and Virchow) across six MedMNIST benchmarks and two native-resolution clinical datasets (NCT-CRC-HE-100K and ISIC 2019; N > 110,000 images total). All models exhibited substantial anisotropy: effective dimensionality ranged from 0.7 to 7.1% of the nominal embedding dimension. Post hoc whitening (PCA/ZCA) reduced classification AUC in 34 of 36 model-dataset combinations and in every dataset and every model evaluated (mean ΔAUC =  - 0.028). Because the combinations are crossed rather than independent, clustering-aware analyses (by dataset, by model, and a crossed mixed-effects model, 95% CI - 0.062 to + 0.006) gave the same - 0.028 mean degradation-directionally consistent across every dataset and every model, though with an imprecise magnitude; the degradation worsened monotonically as regularization weakened and was robust to classifier choice (logistic regression and MLP), numerical precision (FP16 vs. FP32), and image resolution. A variance-retention sweep located an empirical operating region of 90-95% beyond which degradation re-emerged. Whitening also degraded retrieval (ΔmAP =  - 0.149) and few-shot learning (Δ5-shot =  - 0.224), indicating that anisotropic geometry encodes task-relevant structure across all three tasks. Notably, a near-isotropic model (PLIP, AvgCosSim ≈ 0.003) degraded comparably to the most anisotropic models, indicating that the degradation reflects the variance structure of the eigenspectrum rather than anisotropy itself. We propose spectral-aware standardization (SAS), a gentle spectral transform that improves classification (ΔAUC =  + 0.005, 23/36; validated leave-one-dataset-out and on two held-out pathology models, UNI and Virchow, 12/12) and, unlike whitening, does so without large retrieval or few-shot penalties-a Pareto improvement over whitening across the three tasks. The gains over simple z-score standardization are nonetheless marginal, and for distance-based use, plain standardization remains the recommended default.

Tags

Topics