Leveraging a foundation model zoo for cell similarity search in oncological microscopy across devices

Abstract

Background: Cellular imaging analysis using the traditional retrospectiveapproach is extremely time-consuming and labor-intensive. Although AI-basedsolutions are available, these approaches rely heavily on supervised learningtechniques that require high quality, large labeled datasets from the samemicroscope to be reliable. In addition, primary patient samples are oftenheterogeneous cell populations and need to be stained to distinguish thecellular subsets. The resulting imaging data is analyzed and labeled manuallyby experts. Therefore, a method to distinguish cell populations across imagingdevices without the need for staining and extensive manual labeling would helpimmensely to gain real-time insights into cell population dynamics. Thisespecially holds true for recognizing specific cell types and states in responseto treatments.Objective: We aim to develop an unsupervised approach using general visionfoundation models trained on diverse and extensive imaging datasets to extractrich visual features for cell-analysis across devices, including both stained andunstained live cells. Our method, Entropy-guided Weighted Combinational FAISS(EWC-FAISS), uses these models purely in an inference-only mode without taskspecificretraining on the cellular data. Combining the generated embeddings inan efficient and adaptive k-nearest neighbor search allows for automated, cross device identification of cell types and states, providing a strong basis for AIassistedcancer therapy.Methods: We utilized two publicly available datasets. The WBC dataset includes14,424 images of stained white blood cell samples from patients with acutemyeloid and lymphoid leukemia, as well as those without leukemic pathology.The LISC dataset comprises 257 images of white blood cell samples from healthyindividuals. We generated four in-house datasets utilizing the JIMT-1 breastcancer cell line, as well as Jurkat and K562 (leukemic cell lines). These datasetswere acquired using the Nanolive 3D Cell Explorer-fluo (CX-A) holotomographicmicroscope and the BioTek Lionheart FX automated brightfield microscope. Theimages from the in-house datasets were manually annotated using Roboflowsoftware. To generate the embeddings, we used and optimized a concatenatedcombination of SAM, DINO, ConvNeXT, SWIN, CLIP and ViTMAE. The combinedembeddings were used as input for the adaptive k-nearest neighbor search,building an approximate Hierarchical Navigable Small World FAISS index. Wecompared EWC-FAISS to fully fined-tuned ViT-Classifiers with DINO-, andSWIN-backbones, a ConvNeXT architecture, as well as to NMTune as alightweight domain-adaptation method with frozen backbone.Results: EWC-FAISS performed competitively with the baselines on the originaldatasets in terms of macro accuracy. Macro accuracy is the average of classspecificaccuracies, treating all classes equally by averaging their individualaccuracies. EWC-FAISS ranked second for the WBC dataset (macro accuracy:97.6 ± 0.2), first for cell state classification from Nanolive (macro accuracy: 90 ± 0),and performed comparably for cell type classification from Lionheart (macroaccuracy: 87 ± 0). For the transfer to out-of-distribution (OOD) datasets, whichthemodel had not seen during training, EWC-FAISS consistently outperformed theother baselines. For the LISC dataset, EWC-FAISS achieved a macro accuracy of78.5 ± 0.3, compared to DINO FT’s 17± 1, SWINFT’s 44 ± 14, ConvNeXT FT’s 45±9, and NMTune’s 52 ± 10. For the cell state classification from Lionheart, EWCFAISShad a macro accuracy of 86 ± 1, while DINO FT, SWIN FT, and ConvNeXT FTachieved 65 ± 11, 68 ± 16, and 81 ± 1, respectively, and NMTune 81 ± 7. For thetransfer of cell type classification from Nanolive, EWC-FAISS attained a macroaccuracy of 85 ± 0, compared to DINO FT’s 24.5 ± 0.9, SWIN FT’s 57 ± 6,ConvNeXT FT’s 54 ± 4, and NMTune’s 63 ± 4. Additionally, building EWC-FAISSafter embedding generation was significantly faster than training DINO FT (∼ 6minutes compared to > 10 hours). Lastly, EWC-FAISS performed comparably indistinguishing cancerous cell lines fromPeripheral BloodMononuclear Cells with amean accuracy of 80 ± 5, compared to CellMixer with a mean accuracy of 79.7.Conclusion: We present a novel approach to identify various cell lines andprimary cells based on their identity and state using images acquired acrossvarious imaging platforms which vary in resolution, magnification and imagequality. Despite these differences, we could show that our efficient, adaptive knearestneighbor search pipeline can be applied on a large image datasetcontaining different cell types and effectively differentiate between the cellsand their states such as live, apoptotic or necrotic. There are severalapplications, particularly in distinguishing various cell populations in patientsamples or monitoring therapy.