Safe AI agents in industry: Reliably deploying LLM-based troubleshooting assistants

In industrial production, errors must be quickly identified and corrected. Especially in complex processes such as injection molding, troubleshooting is time-consuming and heavily dependent on the experiential knowledge of individual experts. LLM-based assistance systems promise to alleviate this burden: They can consolidate information, explain correlations, and derive recommendations for action.

At the same time, new risks arise. One key reason is that LLM-based systems can generate incorrect, inconsistent, or difficult-to-verify answers. In production-related applications, this can lead to incorrect machine operation, quality issues, production downtime, and thus additional costs.

It is therefore crucial to determine whether such risks can be systematically identified, assessed, and technically mitigated. This is also important in light of the increasing regulatory requirements for traceable AI applications.

The goal of the joint project between Fraunhofer IKS and Hitachi was to develop a methodological approach for deploying LLM-based agent systems reliably, verifiably, and safely in industrial applications. The specific application focused on two questions:

• How should a troubleshooting assistant handle unsafe or potentially dangerous situations?
• How can safety-critical recommendations be prevented?

Until now, many companies have lacked a systematic engineering approach for safeguarding agent systems and addressing these questions.

In collaboration with Hitachi, Fraunhofer IKS developed a methodological approach and measures to systematically analyze and effectively mitigate risks posed by LLM-based agents. To this end, the risk analysis was carried out as System-Theoretical Process Analysis (STPA). STPA is a systems-theoretical method for hazard and risk analysis that Fraunhofer IKS has tailored specifically to agent-based AI systems. This extension enables the structured identification of potential hazardous situations, unsafe recommendations for action, and LLM-specific causes of errors, such as hallucinations or incorrect context interpretations. Subsequently, technical mitigation measures and constraints were derived, such as structured input and output formats, consistency and range checks, and safe default responses.

Fraunhofer IKS and Hitachi validated the methodology using an LLM-based multi-agent troubleshooting assistant for injection molding production. The assistant system helps users analyze typical production errors and determine appropriate corrective actions. To do so, it processes information on machine states, historical data, and technical domain knowledge.

This use case demonstrates how LLM-based AI agents can make existing knowledge more accessible and assist with troubleshooting. The safety methodology ensures that recommendations are checked against defined safety limits and restricted if necessary. This reduces the effort required for troubleshooting, saving both time and costs.

In scenario-based test runs, errors and malfunctions were deliberately introduced into the system to test how the Troubleshooting Assistant behaves in critical situations (fault injection). Without countermeasures, the rate of safety-critical violations was over 40 percent. Using the developed methodology, all safety-critical rule violations were detected and prevented; the rate dropped to 0 percent.

This provides companies with a robust foundation for deploying LLM-based AI agents in a controlled and risk-aware manner.

The Fraunhofer IKS approach helps to

• identify and mitigate risks associated with the use of AI agents at an early stage,
• prevent operational errors, quality issues, and production downtime,
• ensure that AI-supported recommendations are transparent and verifiable.

As a result, companies can design their processes more efficiently and better meet regulatory requirements. This is particularly relevant for industrial applications with high standards for quality, safety, and compliance.

The methodology can be applied to other LLM-based assistance and agent systems, as it can be adapted to the respective application context. Fraunhofer IKS supports companies in integrating AI agents step by step into safety- or quality-critical processes.

Would you like to use LLM-based AI agents efficiently in your production processes without creating new safety or quality risks? Fraunhofer IKS supports you in systematically analyzing risks and developing appropriate safeguards. Contact us via the contact form.

Go to contact form