I had a presentation about using AI to manage risk and compliance (one of my favorite topics) at the ISACA conference in New York on August 19. Given the topic, it was not surprising that the integrity and veracity of the tools were one of the main concerns that people expressed in the post-talk discussions. Since we cannot really afford wrong information, how do we address AI hallucination?
What is hallucination?
In general, hallucination is a case when an AI generates false, sometimes even nonsensical, information and presents it with a high level of confidence. This could include blatantly false statements, fabricated evidence, and references, etc. In more extreme cases, the AI continues to insist on the false information even after repeated prompt challenges.
We all have experienced it with popular generic AI systems, but this becomes an even more serious issue if it is presented by an expert AI system like AI GRC.
Also, let’s be clear that humans sometimes exhibit similar behaviour - intentionally or not. While I will not discuss the intent aspect of the AI hallucination, we should recognize that, while it could be significantly reduced, it cannot be eliminated. The goal is really to make it significantly lower than that of a human.
Why does AI hallucinate?
Similar to a human, AI tends to hallucinate for various reasons:
- Insufficient training data/lack of consensus on the subject. Even if the model does not have sufficient information, it will still try to accomplish the task (give us an answer) by filling in the gaps.
- Heavily biased/inaccurate training data. This is the traditional “Garbage in - garbage out” problem
- Overfitting. The model that has learned its training data too thoroughly can struggle to generalize to new or unseen information, leading to irrelevant or incorrect responses
- Misinterpretation/lack of real-world grounding. The AI uses a model of reality; it is not OUR reality. This makes it difficult for it to differentiate between fiction and reality.
The “Quad”
What can we do about it? At Trustero, we refer to our “hallucination reduction” strategy as “The Quad”:
- Contextualization. This could be achieved in several ways. If one has plenty of specialized data, at the correct level of detail, they can create a Retrieval Augmented Generation (RAG) to help the model “focus” on the correct information. In some cases, this could also reduce biases in the underlying model.
When the necessary information is not sufficient for RAG, it could be indexed, categorized, and used by the model as part of forming the answer. The most common way is “uploading” the relevant files and asking the model to use them. A special case of this approach is utilizing a web search. The model could generate the search and utilize the answers from the search engine. While this is very powerful, it should be used with caution because of the amount of disinformation on the Internet. A prudent approach would be to limit the search to well-established, trustworthy sources.
- Prompting. When the prompt is ambiguous or vague, the AI interpretation may cause hallucination. Therefore, we should be as accurate and as exhaustive as possible. Additionally, the structure of the prompt is very important - first, we have to establish the context, then explain the task, and provide an explanation/reference to any information that could be useful in performing the task. There are additional prompt engineering techniques (like establishing a “reward” or “punishment”) that could shape the request. There are also tools that could generate appropriate prompts based on the desired outcome.
- Guardrails. Those could be part of the prompt, established in advance, or achieved by adjusting the model parameters. The goal is to restrict the AI within certain parameters. For example, as part of the prompt, we can instruct the AI to admit that it does not know something unless it has high confidence in the answer. We could also restrict the AI from using certain data in order to narrow the focus and avoid irrelevant information.
- Reasoning. Explaining the reason the AI came up with the answer, along with references to the data it used in the analysis, increases the likelihood of it correcting itself. More importantly, it allows the human to verify the logic, evidence, and conclusion. This is extremely important to establish trust in the AI system - not so dissimilar to how we establish trust with humans.
Conclusion
We cannot eliminate hallucination completely; however, we can take steps to significantly reduce it and make it insignificant. Using “The Quad” framework - Contextualization, Prompting, Guardrails, and Reasoning - could be very effective, especially when we have a human in the loop. Therefore, the combination of AI and a human could be quite effective.