Document Type

Thesis

Degree Name

Master of Applied Computing

Department

Physics and Computer Science

Program Name/Specialization

Applied Computing

Faculty/School

Faculty of Science

First Advisor

Emad Amin Mohammed

Advisor Role

supervisor

Second Advisor

Azam Asilian Bidgoli

Advisor Role

co-supervisor

Abstract

Medical image segmentation is a critical component of clinical decision-making, yet deep learning based segmentation models face persistent challenges. These models typically require large volumes of densely annotated data, exhibit strong sensitivity to domain shifts across scanners and patient populations, and lack explicit anatomical priors. Furthermore, despite continual architectural advances beyond fully convolutional networks, most segmentation models remain opaque, where they lack the granularity required for component-level analysis. As a result, when segmentation models fail, it is difficult to identify which internal representations are responsible, limiting developers’ ability to diagnose errors, improve robustness, or establish trust in clinical settings. Existing explainable AI techniques, many of which were designed for image classification, offer limited fidelity for dense pixel-wise prediction tasks and often fail to capture the internal mechanisms driving segmentation decisions. This thesis addresses these challenges through two complementary frameworks that target both the learning dynamics and the interpretability of medical image segmentation models.

In the first part, we propose Joint Retrieval-Augmented Segmentation (J-RAS), a retrieval-guided framework that reduces dependence on large annotated datasets and improves robustness under domain shift. Unlike prior retrieval-based approaches that treat retrieval as a static, passive module, J-RAS jointly optimizes a retrieval model and a segmentation model through an alternating contrastive and supervised learning procedure. This mutual adaptation transforms retrieval into an active collaborator that learns to emphasize segmentation-relevant anatomical cues. We demonstrate that J-RAS is architecture-agnostic and achieves strong cross-dataset generalization, enabling retrieval from one dataset while performing segmentation on another without retraining.

In the second part, we focus explicitly on the interpretability of segmentation models, motivated by the lack of component-level transparency in existing architectures. Rather than treating segmentation networks as black boxes, we propose a mechanistic interpretability framework that factorizes internal activations into interpretable latent. We train four independent SegFormer models, each specialized to a distinct data distribution of healthy brains, adult glioma, pediatric glioma, and Sub-Saharan African glioma and decompose their intermediate representations using Sparse Autoencoders (SAEs). This decomposition yields sparse, disentangled latent features corresponding to recurring anatomical and pathological activation patterns, enabling direct inspection and manipulation of individual components. To bridge raw latent activations with human understanding, we introduce an automated interpretation pipeline that converts SAE features into human-readable anatomical descriptions through activations projection, and geometry-based spatial metrics. To study representation transfer and alignment across datasets, we match latent features using the Hungarian algorithm, identifying shared and dataset-specific latents. Through controlled feature steering and swapping experiments, we establish causal links between internal representations and segmentation performance. Crucially, this enables direct failure diagnosis: when segmentation errors occur, we can identify which internal latent components are under-activated or misaligned. We observe a relationship between the amplification of selected sparse features and Dice score on failure cases, with targeted interventions rescuing up to 88% of failures. These results demonstrate that segmentation failures are not merely opaque errors, but can be traced to specific internal representational deficits and corrected through principled, component-level intervention.

Together, this thesis demonstrates that addressing the limitations of medical image segmentation requires not only improved data-efficient learning strategies, but also interpretable and transparent models that support component-level analysis, failure diagnosis, and principled intervention.

Convocation Year

2026

Convocation Season

Spring

Download

Share

COinS