Safety Architecture for AI Components in Production

In the industry project, Fraunhofer IKS investigated an abstracted manufacturing scenario involving industrial robots. A simulated robotic arm was tasked with moving a workpiece (green cube) to a goal position (blue cube) while avoiding collisions with a human worker entering the workspace in different ways. 

The simplicity of this task makes this scenario ideal for analyzing the capabilities and limitations of AI in safety-critical applications. During the project, a trajectory planning function was trained using reinforcement learning, by learning the movement policy in a trial-and-error fashion. However, the training method does not allow for safety guarantees regarding the function's output. This is due to several AI risk factors which affect AI-based functions in all stages of their development lifecycles. Therefore, Fraunhofer IKS developed a systematic approach to identify risks and derive effective measures within the safety architecture. 

To identify and assess AI risks, established risk management practices according to ISO 31000 were utilized [1]. From a list of generic risk factors, Fraunhofer IKS successively analyzed their impact on the system under consideration. For the final list of risk factors, Fraunhofer IKS defined state-of-the-art mitigation measures that can reduce the level of risk at run-time. The AI Safety Envelope (AI-E) architecture served as a guide for integrating these measures into the system, which is based on prior work of Fraunhofer IKS [2].

The mitigation techniques were implemented as self-protecting, self-monitoring, or self-checking measures. These three techniques form the AI Safety Envelope, which wraps around the unreliable AI function and transforms it into a dependable subsystem. The AI-E architecture is an implementation of architectural mitigation for AI functions described in ISO/IEC TR 5469 [3]. 

Fraunhofer IKS conducted a series of simulation-based tests to quantitatively evaluate how effectively the AI-E architecture can mitigate the identified AI risk factors. Using the simulator Webots, 10,000 test scenarios were evaluated to measure both the rate of safely completed scenarios (safety rate) and the rate of successfully completed scenarios (success rate), to investigate the trade-off between safety and availability. 

The research team observed that each implemented mitigation measure improved the system's safety rate. In addition, the combination of all measures into a single envelope led to further enhancements in safety compared to each individual measure, indicating that the different measures address various AI risks and complement each other. 

Regarding the success rate, Fraunhofer IKS observed a trade-off between safety and availability. In some scenarios, the AI-E was overly cautious and intervened, even though the scenario could have been completed safely by the control AI function on its own. However, such an outcome is generally desirable to prevent hazardous situations and the resulting losses. Statistical analyses indicated that the reduction in availability leads to an acceptable loss in productivity. 

The successes of this project would not have been possible without the close collaboration between research and industry. The industry-driven use case allowed Fraunhofer IKS to apply its expertise to a manufacturing problem with high significance. For Hitachi, the project results provide a foundation for implementing AI-based production systems and applying AI functions in other safety-critical areas. 

Statement from Hitachi:

“Hitachi sincerely appreciates the significant results delivered through this collaboration with Fraunhofer IKS. The development and validation of the AI Safety Envelope architecture, demonstrated by substantial improvements in safety rates across extensive simulations, represent a meaningful advancement toward the safe and reliable integration of AI in autonomous control and other critical systems.

The systematic identification and mitigation of AI-related risks, aligned with international safety standards, provide a robust foundation for applying advanced AI functions not only in manufacturing, but also in a wide range of safety-critical domains. We look forward to leveraging these achievements as a basis for further innovation and practical implementation in the future.” 

Chief Researcher, Hitachi Ltd. R&D Group, Autonomous Control Research Department  
Satoshi Otsuka  

The primary benefit of the solution lie in the development of a safety architecture that enables the safe integration of AI-based control in industrial robots. The developed architecture enhances the reliability and safety of AI-driven systems, which is crucial for their application in safety-critical environments. 

The central problems addressed include: 

1. Safety Requirements: The developed safety architecture ensures the safe use of AI functions by systematically identifying and mitigating potential risks. 
2. Lack of Reliability in AI Functions: The implementation of the AI Safety Envelope increases the reliability of AI functions, enabling their use in safety-critical applications. 
3. Lack of Flexibility: Enabling the use of AI functions allows robots to dynamically respond to their environment, increasing adaptability to variabilities in the production process.

This success story demonstrates how innovative approaches in AI safety can usher in a new era in manufacturing. 

[1] ISO 31000:2018, “Risk management – Guidelines,” International Organization for Standardization, 2018.

[2] G. Weiss, P. Schleiss, D. Schneider, and M. Trapp, “Towards Integrating Undependable Self-Adaptive Systems in Safety-Critical Environments,” in Proceedings of the 13th International Conference on Software Engineering for Adaptive and Self-Managing Systems, May 2018, pp. 26–32.

[3] ISO/IEC TR 5469:2024, “Artificial intelligence – Functional safety and AI systems,” International Organization for Standardization, 2024.