The Segmentation Crisis: Overcoming Cell Boundary Failures in Spatial Biology

Imaging-based spatial transcriptomics platforms such as 10x Genomics Xenium, Atera, and NanoString CosMx can localize transcripts at cellular and subcellular scales while preserving tissue architecture. But determining where a transcript is located is only part of the challenge.

To analyze spatial RNA at the cellular level, transcripts must ultimately be assigned to individual cells or cellular regions. This requires cell segmentation: identifying nuclei and estimating the boundaries of the surrounding cells.

This step is deceptively difficult. Cells can be tightly packed, touch or overlap, have irregular morphologies, or lack clearly visible boundaries. As a result, segmentation errors can propagate directly into cell-by-gene matrices and influence downstream analyses such as differential expression, cell-type assignment, clustering, and spatial neighborhood analysis.

A segmentation mask is therefore not simply a visualization layer. It defines the biological units on which much of the subsequent analysis is performed. 

Three Major Sources of Segmentation Error

Segmentation errors generally fall into three broad categories: under-segmentation, over-segmentation, and incorrect transcript assignment (Figure 1).

Figure 1: Schematic illustration of the major challenges in cell segmentation. The figure was refined using Google Gemini based on the authors’ original drawing.

• Under-segmentation — multiple cells become one: Adjacent cells are incorrectly merged into a single object, producing mixed expression profiles. For example, transcripts from a T cell and a tumor cell could be combined into an artificial “hybrid” profile.
• Over-segmentation — one cell becomes several: A single biological cell is incorrectly divided into multiple objects, potentially producing fragmented profiles with low transcript counts or objects that lack a nucleus.
• Transcript misassignment — transcripts are assigned to the wrong cell: Even when the number of segmented objects appears reasonable, inaccurate boundaries can assign transcripts near cell borders to neighboring cells. This can reduce apparent cell-type purity and distort cell-level expression profiles.

These errors are not necessarily independent. A poorly positioned boundary can simultaneously cause apparent over-segmentation, under-segmentation, and transcript misassignment.

Why Segmentation Errors Matter Biologically

Segmentation errors are not merely cosmetic.

They can propagate into almost every downstream analysis that relies on a cell-by-gene matrix:

Segmentation error
↓
Incorrect transcript assignment
↓
Distorted cell expression profile
↓
Altered clustering / annotation
↓
Potentially misleading biological interpretation

Recent work by Mitchel et al. (2026) ², spanning multiple tissues and spatial transcriptomics platforms, found that segmentation errors can substantially confound downstream analyses of cellular state, including differential expression, neighboring-cell influence, and ligand–receptor interactions.

This is particularly important for analyses of:

• rare cell populations,
• cell–cell interactions,
• ligand–receptor signaling,
• differential expression,
• neighborhood composition, and
• spatially organized cell states.

A suspicious hybrid profile therefore calls for a segmentation quality check before any claim of a novel biological state. 

Why Nuclear Staining Alone Is Not Enough

Most of these errors trace back to one step: how the cell boundary is estimated from the image.

A common starting point is DAPI, which gives a strong signal for identifying nuclei. A nucleus marks the center of a cell, and the cell extends beyond it. A DAPI-based workflow therefore needs an additional strategy to estimate the surrounding cellular region, such as:

• membrane or cell-boundary staining,
• cytoplasmic/interior staining,
• nuclear expansion, or
• transcript-aware segmentation.

For example, Xenium workflows can combine nuclear staining with cell-boundary and cell-interior information. When direct cellular boundary information is limited, nuclear expansion can be used as an approximation of the cell territory. This is useful, but it remains an approximation. The appropriate expansion depends on tissue morphology, cell size, packing density, and the quality of the available imaging information. 

A nucleus mask is evidence of where the nucleus sits, and the cell boundary remains an estimate built on top of it. 

Choosing and Benchmarking Segmentation Methods

These strategies are implemented in a growing set of algorithms, and their relative performance depends on the dataset. Image-based approaches such as Cellpose and Segment Anything (SAM) use morphological information, while transcript-aware approaches such as Baysor and Segger incorporate the spatial distribution of transcripts. Performance varies substantially across tissues, assays, and imaging conditions, so the key question becomes which approach performs reliably for a particular tissue, assay, and biological question. 

Figure 2, adapted from Table 3 of Li et al. (2024) ¹, illustrates this dataset-dependent performance across multiple segmentation approaches and three spatial transcriptomics datasets.

Figure 2: Segmentation performance across different algorithms and datasets. Adapted from Table 3, Li et al. (2024) ¹.

The results highlight an important feature of segmentation benchmarking: performance is dataset dependent. Metrics such as adjusted Rand index (ARI), normalized mutual information (NMI), and silhouette coefficient quantify agreement or separation within a particular evaluation framework, but they do not directly establish whether an inferred boundary is biologically correct. Consequently, segmentation benchmarking should combine quantitative metrics with direct boundary measurements where ground truth is available, transcript-assignment accuracy, cell-size and transcript-count distributions, visual inspection, and downstream biological validation.

More importantly, segmentation should not be treated as a purely computational exercise. Tissue morphology, cell density, architecture, staining quality, and the spatial organization of different cell types can all influence segmentation performance.

Our previous blog on five common mistakes in spatial transcriptomics illustrated how additional tissue information, such as H&E morphology, can provide valuable context for interpreting segmentation, clustering, and cell annotation. The broader lesson is that researchers should evaluate not only the computational output, but also whether the inferred cellular organization makes biological sense.

Segmentation methods should therefore be evaluated on representative tissue regions rather than selected solely on published rankings.

Interactive Segmentation and Tissue Evaluation with SpatialX

Visual inspection provides an important complement to computational benchmarking. SpatialX provides an interactive environment for evaluating spatial transcriptomics data and refining cell boundaries, including both sequencing-based and imaging-based spatial technologies (Figure 3).

Figure 3: Interactive evaluation of cell segmentation and transcript assignment in SpatialX. The left panel shows segmentation incorporating H&E morphology, while the right panel shows segmentation without H&E integration.

A useful segmentation workflow should allow researchers to:

1. inspect the underlying tissue morphology;
2. generate or import segmentation boundaries;
3. visualize transcript assignments;
4. identify suspiciously merged or fragmented cells;
5. adjust segmentation parameters; and
6. compare the resulting cell-level expression profiles.

This visual validation step is critical because segmentation errors are often easiest to recognize when molecular information is viewed together with tissue morphology.

For imaging-based spatial transcriptomics, where transcripts are detected at molecular coordinates and subsequently assigned to cells, segmentation can have a direct impact on the resulting cell-level expression matrix.

For sequencing-based technologies such as Visium HD, the analytical workflow is different. Cell segmentation is not inherently required because measurements are generated on predefined spatial bins. Instead, researchers may work directly at the bin level or use approaches such as deconvolution to estimate cell-type or cellular signals.

Understanding these differences between spatial technologies is essential when interpreting what a “cell” represents in a spatial dataset.

See your own tissue in SpatialX.

Load your Xenium, CosMx, or Visium HD data, compare segmentation with and without H&E, inspect transcript assignments, and refine boundaries in one interactive workspace.

[Request a SpatialX demo →]

Conclusion

Cell segmentation is the bridge between molecular coordinates and cell-level spatial biology—but it is also one of the largest sources of uncertainty in imaging-based spatial analysis.

Under-segmentation can merge neighboring cells.

Over-segmentation can fragment individual cells.

Transcript misassignment can create artificial mixed expression profiles even when the underlying transcript localization is accurate.

The solution is not simply to choose a more sophisticated model. Reliable spatial analysis requires appropriate imaging, tissue-aware segmentation, transcript assignment, quantitative QC, and visual validation.

Modern approaches increasingly combine morphology with molecular information, while transcript-aware methods provide alternatives when image-based boundaries are insufficient.

The central principle is:

A cell boundary is an inference—and every biological conclusion derived from that boundary should be treated accordingly.

For high-resolution spatial biology, segmentation is not a preprocessing detail. It is part of the measurement itself.

References

1. Li, Y., Lac, L., Liu, Q. & Hu, P. ST-CellSeg: Cell segmentation for imaging-based spatial transcriptomics using multi-scale manifold learning. PLoS Comput. Biol. 20, e1012254 (2024).

2. Mitchel, J., Gao, T., Petukhov, V., Cole, E. & Kharchenko, P. V. Impact and correction of segmentation errors in spatial transcriptomics. Nat. Genet. 58, 434–444 (2026).