Deep learning for co-learning from spatial transcriptomics and histopathology images

Co-learning from spatial omics and histopathology images, with the aim of predicting molecular features from histopathology alone.

Two patient examples showing whole-slide histopathology images, attention heatmaps, predicted score heatmaps, and the highest-attention and highest-score image tiles.

Background

Spatially resolved sequencing is a recent development: only in the last few years has it become possible to measure several molecular modalities in single cells while keeping their position in the tissue. These measurements are expensive. In many cases, however, they come with histopathology images of the same tissue, which are cheaper and more abundant.

This pairing raises a practical question: can a model learn from both data types at once, and can the expensive spatial features then be predicted from histology alone? Relevant methods include variational autoencoders for data integration and self-supervised representation learning for multi-modal omics. One possible focus is the identification of spatially resolved disease hotspots.

Questions we are working on

  • Which computational strategies best integrate spatial transcriptomics with morphological features from histopathology images?
  • How can the representations of the two modalities be aligned in a common space, efficiently and faithfully?
  • Can selected molecular features be predicted from histopathology images alone?

Key literature

  1. Liang J, et al. Spatial biomarker discovery via interpretable semantic learning in histopathology. Cancer Cell (2026). [closest idea to this project]
  2. Yang KD, et al. Multi-domain translation between single-cell imaging and sequencing data using autoencoders. Nat Commun (2021).
  3. Bergenstråhle L, et al. Super-resolved spatial transcriptomics by deep data fusion. Nat Biotechnol (2022).
  4. Tanevski J, et al. Explainable multiview framework for dissecting spatial relationships from highly multiplexed data. Genome Biol (2022).
  5. Jaume G, et al. HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis. arXiv:2406.16192 (2024).
  6. Shao D, et al. Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learning. arXiv:2603.22198 (2026).