On trustworthy AI

tl;dr: what it means to have a trustworthy model

Human intelligence is fundamentally characterized by metacognition, our ability to reflect on our own thinking. This capacity drives our pursuit of Artificial Intelligence (AI), where the externalization of our cognitive processes is a form of metacognition. Yet, such AI systems remain bounded by their assumptions and training data, and questioning those assumptions remains a human contribution. Therefore, the role of AI is not to replace human intelligence but to augment it, automating familiar tasks and freeing human cognition for higher-order challenges. For such augmentation to be trustworthy, AI must exhibit a similar awareness of its own limitations, much as human collaborators are trusted because they signal when they are uncertain. In high-stakes settings, where unwarranted confidence carries irreversible consequences, this means reliable uncertainty quantification. Computer Vision (CV) recently unlocked state-of-the-art performance across different domains using Deep Learning (DL), i.e., the convergence of large-scale data, computational power and mathematical foundations. This pragmatic turn toward raw computational power has created a troubling gap where theoretical understanding struggles to keep pace with practice, introducing significant risks by producing models that can be confidently wrong.

Defining trustworthiness

An ML model cannot be expected to independently derive the complex and nuanced rules of human morality from historical training data alone, nor can we guarantee it will follow those rules without some way of recognizing where its own competence runs out. For example, an AI risk assessment tool used for criminal sentencing could incorrectly flag defendants from over-policed neighborhoods as high-risk, unfairly influencing a judge to impose a longer prison sentence. Or a hospital using AI purely to optimize patient survival rates could deny a bed to an elderly person in favor of a younger one with marginally better odds, a decision devoid of human values like compassion and fairness.

These examples share a common failure. The system acts with full confidence in exactly the situations where a human would pause. Sometimes that is because the underlying data reflects structural inequities, and sometimes it’s because the decision calls for ethical judgment that no dataset can provide. Either way, a trustworthy system would signal that doubt instead of handing back a definitive answer.

This is the capability at the heart of the framework I’m proposing, which rests on two kinds of soundness. Moral soundness is about whether a system respects the values, rights, and expectations of the people it affects. Structural soundness is about whether it performs reliably, accurately, and predictably under the conditions it actually encounters. A single capability, what I call uncertainty awareness, a model’s capacity to recognize and communicate the limits of its own knowledge, turns out to be a partial but unifying lever on both. Of course, the uncertainty a model reports must be well-calibrated, and when the model defers there must be a more reliable fallback to catch the decision, such as human oversight or a safe default. The remaining dimensions sit largely beyond what any technical fix can reach, and call for institutional, legal, or societal solutions instead.

Structural soundness:

• Reliability: A system that defers to a human expert when uncertain contributes to more consistent and dependable outcomes.
• Robustness: By flagging inputs that look atypical of its training data, the system can abstain instead of predicting on them, which keeps its behavior more dependable under varied conditions.
• Resilience: Recognizing uncertainty can trigger safe fallback mechanisms, helping the system recover gracefully from failures.
• Accuracy: By abstaining when confidence is low, the system raises the accuracy of the predictions it does make.
• Verifiability: Stated uncertainty can be checked against observed outcomes, which makes the system’s confidence empirically testable.
• Efficiency: Uncertainty can guide how resources are allocated, informing when to request human input, how much computation to spend, and what new data to gather, though obtaining it carries its own cost.

Moral soundness:

• Transparency: By signaling when it is uncertain, the model makes the confidence behind its decisions more visible to users and operators.
• Honesty: The system avoids overclaiming, admitting its own uncertainty rather than returning a confident but baseless answer.
• Safety: Identifying high-uncertainty cases lets the system defer to human oversight or enter a safe mode, reducing the risk of harm.
• Explainability: A measure of certainty complements the model’s explanation by indicating how much weight to place on it.
• Fairness: By surfacing inputs from under-represented regions of the training data, uncertainty can flag where default predictions are least supported, directing review or data collection toward groups that would otherwise be silently underserved.

Trustworthy decision making

Adopting an AI system means living with two sources of uncertainty that never fully go away. The first is that our observations of the world are imperfect, so the data a system learns from may not faithfully reflect the conditions it later has to operate in. The second is that our models are necessarily simplified, limited by finite capacity, the architectures we choose, and mathematical assumptions that may not hold once the system meets reality.

To deal with both, I lay out a framework that walks through the steps an idealized ML pipeline should clear before a trained model is trusted to make decisions. It has four stages. The first two are operational. Acquisition is how the system gets its data, whether by sensing, retrieval, or synthesis from physical, archival, or virtual sources. Inference is the step of reasoning from that data toward a hypothesis or prediction. Both are well-worn territory, and the contribution here is the two verification stages that follow. Alignment and identifiability ask whether the system actually understands enough to be trusted.

Alignment is about the match between the data and the model. It asks two things. Do the observations conform to the distribution the model assumes, and is the model’s parameterization rich enough for the data it is actually seeing? On the data side it surfaces distribution shift, anomalies, and adversarial inputs. On the model side it surfaces capacity or structural mismatch. When misalignment shows up, two corrective strategies follow, each acting on the opposite side of this relationship. Retargeting is the data-driven move, redirecting acquisition back toward the assumed distribution when observations fall out of conformance, while holding the model fixed. Respecification is the model-driven move, restructuring the parameterization to accommodate the observed data while holding the data fixed, acknowledging that the current model lacks the capacity or structural fit it needs.

Identifiability carries the same data-versus-model split over to a different question, namely whether enough information exists to determine the quantities of interest. On the data side, the question is whether the observations carry enough information for an accurate prediction, regardless of how many of them you gather. Acquisition is traditionally treated as fixed, which makes the associated uncertainty look irreducible. This framework treats acquisition as a design choice instead, so you can enhance it, intervening to address confounding, improving instrumentation to reduce measurement error, or using richer protocols that resolve states that previously looked indistinguishable. On the model side, the concern is complementary. Even when the data does carry the needed information, a model can fail to extract it in regions it has not adequately learned, giving unreliable predictions on inputs unlike its training data. The remedy there is to refine the model, updating its parameters on new, informative datapoints and, where needed, adapting the training procedure itself, so it can resolve states it previously could not.

World models and active agents

Research attention is increasingly turning toward agentic systems that act over extended horizons, equipped with world models, learned representations of environment dynamics that let an agent simulate, plan, and adapt. Earlier chapters of this thesis developed methods for building such representations, but those contributions largely address static settings, which raises a natural question: how can the uncertainty quantification tools discussed here support more trustworthy active agents? The framework connects naturally to this frontier, since its corrective strategies map directly onto the decisions an agent must make. A world model has to maintain alignment between its learned dynamics and the true environment, recognizing when its internal simulation drifts away from reality, and it has to assess identifiability, distinguishing between multiple plausible world states and acknowledging when fundamental ambiguity remains. When either alignment or identifiability fails, the agent can choose the appropriate corrective strategy, though respecification, altering the fundamental structure of the model, remains a human design decision and so underscores the continued need for human oversight.

Traditional approaches also tend to treat aleatoric uncertainty as a single, irreducible lump. The framework here splits it in two: uncertainty that is genuinely inherent to the world, and uncertainty that is really an artifact of how the data was acquired (see the figure below). CT imaging is a good example. A scan might first produce an ambiguous image of a lesion, yet changing the reconstruction kernel or adjusting the scan parameters can bring it into focus. The same principle is what governs identifiability constraints in probabilistic image segmentation, which earlier chapters take up in detail. In cases like this the uncertainty is being driven by the acquisition process, which means it is reducible through active exploration, what I call enhancement. So aleatoric acquisition uncertainty is only irreducible if you hold the data source fixed. The same reasoning carries over to alignment as well. An image might look anomalous at first, its brightness, contrast, or color statistics drifting away from the training distribution, but by adjusting the camera’s exposure, white balance, or gain, an agent can pull those statistics back into the in-distribution regime.

Closing thoughts

The question remains: how can we build systems capable of this level of reasoning to make sound decisions? The solution lies in treating machine learning probabilistically rather than as a straightforward, deterministic input-output mapping. Using probability theory, we can reason under uncertainty and leverage this uncertainty to drive alignment and identifiability-related decisions.