Given a spatial slide with cell type annotations, the goal is to identify regions with homogeneous cell type compositions. For example, with a slide containing two distinct cell types, we aim to identify regions such as:

Region 1: only cell type A

Region 2: composed of both cell type A & B

Region 3: only cell type B

Figure 1. A slide with two different cell types represented by circles and rectangles with three spatial domain regions: 1, 2, and 3. Region 1 contains only cell type A; Region 2 contains both cell types A and B; Region 3 contains only cell type B.

Methods

By incorporating the radius distance, we extend traditional hoodscanR to identify spatial cell type domain.

1. Construct Neighborhood Proportion Matrix:

• For each cell, identify a set of neighboring cells within a defined radius 𝝿.
• A – Simple mode: In the set of neighboring cells, calculate the proportion of different cell types.
• B – Weighted mode: In the set of neighboring cells, calculate the weighted proportion of different cell types. Intuitively, neighboring cells that are close to the current cell contribute with higher weight and smaller weight when neighboring cells are further away. See Appendix for a detailed formula. 

The result of this step is a proportion matrix where each row is a vector of cell type proportion represented in a neighborhood of a cell. 

2. Hierarchical Clustering:

• Create 30 pseudo-cells via k-means clustering of the proportion matrix.
• Cluster pseudo-cells using hierarchical clustering.
• Assign original cells to their corresponding pseudo-cell cluster.

Results

Application to Ovarian Cancer:

To demonstrate the utility of our spatial cell type domain detection, we examined spatial transcriptomics data from 10X Genomics Xenium Prime FFPE Human Ovarian Cancer slides, identifying 16 distinct spatial domains.

Figure 2. Spatial cell type domain detection in ovarian cancer slide.

Among them, our analysis revealed a spatial domain containing pericytes, tumor endothelial cells (TECs), and tumor associated fibroblasts (TAFs), suggesting a critical role in tumor growth, particularly in angiogenesis.

Figure 3. Proportion heatmap of spatial cell type domains. Domain (cluster) 15 contains three cell types, including tumor associated fibroblasts, pericytes, tumor associated endothelial cells. This corresponds to the angiogenesis process in cancer.

Insights into Pericyte – Tumor Endothelial Cells (TECs) – Tumor Associated Fibroblasts (TAFs) Interactions:

• Pericytes, crucial for vascular stability and angiogenesis, interact dynamically with TAFs and TECs. 
• TAFs, key stromal components, remodel the extracellular matrix, secrete growth factors, and modulate immunity, thereby influencing both TEC and pericyte behavior, impacting angiogenesis and vessel stability. 
• Specifically, TAFs secrete pro-angiogenic factors and influence pericyte recruitment. In ovarian cancer, where angiogenesis drives growth and metastasis, TECs form leaky vessels, a condition where pericytes can either worsen or improve.

Figure 4. A – The cell type co-occurrence of Pericytes, TAFs, and TECs in Domain 15 from Figure 3. B – The gene expression analysis of paracrine signaling genes.

Molecular Evidence of Interaction:

Gene expression analysis reveals co-expression of PDGFB, PDGFA, PDGFRA, and VEGFB in pericytes, TECs, and TAFs, suggesting active paracrine signaling. 

• TAFs act as a significant source of VEGF, PDGF, and FGF, influencing TEC proliferation and pericyte recruitment. 
• VEGFB and PDGFs (PDGFB/PDGFA) recruit pericytes to TECs, maintaining their cell-cell interaction. VEGFB contributes to angiogenesis and vascular permeability, while PDGFB/PDGFA and PDGFRA stabilize vessels. 
• This gene expression profile indicates a signaling loop among pericytes, TECs, and TAFs, contributing to a pro-angiogenic tumor microenvironment.

Appendix:

Weighted Proportion Formula:

• P(x, h_a | τ) denotes the probability of cell x residing within the local neighborhood h_a.
• d(x, x_i) signifies the spatial Euclidean distance between cell x and its neighboring cell x_i.
• τ stands as the hyperparameter, facilitating fine-tuned modulation of the impact of neighboring cells.
• h_a denotes the cell neighborhood a, defined by the cell-level annotations provided by users. For example, if cell types were provided, h_a means cell type a neighborhood.
• 1(.) is the indicator function.

References:

Kalluri, R. The biology and function of fibroblasts in cancer. Nat Rev Cancer 16, 582–598 (2016).

Sahai, E. Mechanisms of cancer cell invasion. Cancer Cell 15, 419–423 (2009).

Öhlund, D., Elyada, E. & Tuveson, D. A. Fibroblast heterogeneity in the tumor microenvironment. Science 352, 1357–1361 (2016).

Hanahan, D. & Weinberg, R. A. Hallmarks of cancer: the next generation. Cell 144, 646–674 (2011).

Armulik, A. et al. Pericytes regulate the blood-brain barrier. Nature 468, 557–561 (2010).Andrae, J., Gallini, R. & Betsholtz, C. Role of platelet-derived growth factors in physiology and medicine. Genes Dev22, 1276–1312 (2008)