Fraunhofer Institute for Cognitive Systems IKSFraunhofer IKS helps you systematically identify and quantify uncertainty in AI-based systems and integrate it into your safety and risk management (uncertainty estimation). Through workshops, customer-specific analyses, and co-engineering, we work together to develop uncertainty models, monitoring solutions, and evidence strategies.
© iStock.com/borchee
Uncertainty Estimation for (High-Risk) AI Systems:
Making Uncertainties Manageable
Contents
From Uncertainty to Risk: Why You Should Be Aware of Your Understanding Uncertainties Matters
Artificial intelligence (AI) and machine learning (ML) are increasingly being used in safety-critical applications, such as in the mobility sector. It is precisely in such open and dynamic contexts that the complexity of software systems increases. And as complexity grows, so does uncertainty:
- Data is often incomplete or changes over time.
- Models make assumptions that do not always apply to all scenarios.
- System behavior in new or rare situations is difficult to predict.
Uncertainties evolve throughout the entire lifecycle, from development and deployment to operation in the field. They arise in data, models, scenarios, and system behavior.
If uncertainties are not identified and continuously monitored, critical risks remain hidden. This jeopardizes the safety of the systems, their certification, and ultimately business success.
That is why automotive standards such as ISO/PAS 8800, ISO 21448 (SOTIF), and regulations such as the EU AI Act require structured risk and evidence management –that is, a systematic approach to dealing with uncertainty.
From Theory to Practice: What Are the Challenges of Uncertainty Estimation?
Risks arising from uncertainties must be mitigated. However, it is often unclear what constitutes compliance with standards. This complicates and delays the planning, approval, and certification of systems.
Furthermore, traditional testing methods and historical data are often insufficient to reliably estimate the behavior of AI-based functions in new or changed domains. Different stakeholders interpret requirements from standards and the EU AI Act differently. As a result, there is no common approach to handling uncertainty in practice.
Furthermore, many uncertainties remain unrecognized in everyday use (so-called “unknown unknowns”) and are therefore not considered in risk, safety, or operational decisions.
Uncertainties must therefore be systematically addressed, monitored, and — where possible — continuously reduced both during the design and development phase and at runtime (e.g., due to new situations, behaviors, or environments).
Many companies face similar questions in this regard:
- What uncertainties does my system have—and which of these are truly relevant in the context of existing standards?
- How do I foster a shared understanding among different stakeholders about where uncertainty arises and what assumptions we currently make implicitly?
- How do I put the regulatory requirements for managing uncertainty into practice, rather than just knowing them in the abstract?
- How much can I trust my AI-based system — and how do I justify this trust to internal stakeholders, auditors, and regulatory authorities?
Our Solution: Making Uncertainty in (High-Risk) AI Systems Visible and Manageable
Fraunhofer IKS helps companies make uncertainty in (high-risk) AI systems visible, quantifiable, and manageable—in accordance with the requirements of existing standards.
Our focus is on the gap between traditional safety approaches and AI-specific uncertainties (e.g., data and model uncertainty, ODD uncertainty). Many standards call for concrete approaches here but describe them only in broad terms. We contribute methodological expertise (e.g., probabilistic modeling, Bayesian methods, subjective logic, fuzzy logic, causal models) and combine it with our clients’ domain knowledge.
To this end, we offer a three-step process tailored to the typical questions our clients ask:
- Workshop: “Uncertainty Estimation in AI-Based Systems”
- Client-Specific Initial Analysis & High-Level Concept
- Co-Engineering: Development and Integration of Tailored Solutions
Workshop: “Uncertainty Estimation in AI-Based Systems”
In this workshop on uncertainty estimation, we will show you typical sources of uncertainty in AI-based systems and relate them to specific problems. The focus will be on the following questions:
- How do uncertainties arise in data, models, context, and operations?
- Contextualization within existing standards: Which standards require the management of uncertainty (risk management, data quality, monitoring)?
- What do exemplary solutions look like?
Outcome: By the end of the workshop, you will know where uncertainty arises in AI-based high-risk systems, why it is relevant in the context of existing standards, and what typical questions arise from it.
Customized Initial Analysis & High-Level Concept
The initial analysis focuses on your system. Together, we analyze your system architecture, data sources, models, and operational processes. This allows us to systematically identify relevant uncertainties and prioritize them based on risk and business impact. Specifically, we examine the following points:
- Analysis of architecture, data, models, ODD, and operational processes: What sources and types of uncertainty exist in your system?
- Prioritization by risk impact, including a gap analysis against existing standards: Which uncertainties are relevant from a standards perspective, and where are there currently gaps in evidence or processes?
- Development of a preliminary concept: Which issues should you address first, and which methods are suitable for them?
Result: Following our initial analysis, you will have a structured overview of the sources of uncertainty in your system, a prioritization based on risk and business impact, and a preliminary concept outlining which issues you should address, in what order, and using which methods.
Co-Engineering: Development and Integration of Customized Solutions
We work with you to develop customized uncertainty models and monitoring solutions for your systems. We support you in integrating these into your existing safety, testing, and operational processes, and collaborate with you to create safety cases and evidence strategies:
- Development and integration of uncertainty models and metrics into safety, testing, and operational processes.
- Support for safety cases, evidence strategies, and runtime monitoring. This also includes post-market monitoring as defined by the EU AI Act.
- Joint validation of the solutions.
Result: You receive proven uncertainty models, metrics, and monitoring solutions that are integrated into your existing safety, testing, and operational processes and can be used in compliance with standards.
How Our Uncertainty Estimation Solutions Can Help You
The complexity of AI systems and the requirements imposed by standards and regulations will continue to increase in the coming years. By implementing a structured, uncertainty-based validation process today, you can build capabilities that you can reuse in the long term for new features, ODD extensions, and future versions of standards:
- Compliance with standards and regulatory requirements: You know what standards and regulations require in terms of uncertainty. We help you translate these requirements into concrete analyses, artifacts (e.g., argumentation building blocks, evidence), and processes.
- Control during operation: Using appropriate metrics and monitoring concepts, you can also monitor and reduce uncertainty at runtime.
- Methodological expertise without extra effort: You benefit from our proven methodological toolkit without having to delve deeply into formal procedures yourself.
- Cross-industry knowledge transfer: You benefit from our in-depth knowledge in the field of automotive safety and learn how established concepts and best practices can also be meaningfully applied in less regulated industries.
Why Fraunhofer IKS Is the Right Partner for You
We combine many years of experience in safety engineering with specialized methodological expertise in AI and uncertainty modeling. This enables us to help you integrate AI-based components into existing safety and compliance frameworks.
We translate research into solutions that can be integrated into your existing processes. We primarily work on safety-critical, software-intensive systems in the mobility sector. We also apply our approaches to other domains, such as high-risk and quality applications. Please contact us directly.
Contact us — we’d be happy to assist you.