KPMG implements Contextual AI Risk monitoring with Palqee Prisma
A substantial ROI and business value was delivered, enabling the roll-out of a scalable and efficient solution that optimizes resources and improves decision-making.
Purpose:
KPMG’s Lighthouse team have developed and successfully deployed a generative AI system which retrieves relevant information from various types of documents, quickly providing insights and extracting information with the goal to deliver operational efficiency for its internal teams and customers. A lack of oversight on system performance across clients and projects over time made it difficult for KPMG to trust the AI’s outputs.
They needed assurance on whether the AI is performing as intended and be able to identify what factors are compromising performance to mitigate risks. KPMG was aware the AI was hallucinating in some instances, but they didn’t have insight into what was influencing likelihood of hallucination and inaccurate responses. Palqee Prisma was implemented for the holistic monitoring of KPMG's AI system for the course of three months.
Unlike traditional AI observability and explainability methods, Palqee Prisma adds a context-based analysis layer that allows AI/ML engineers to transform any requirement or attribute into an observable metric, providing full visibility over potential risks in real-time. The main target was to evaluate KPMG’s AI performance including consistency, accuracy and hallucination. Beyond context-based performance evaluation, Prisma can also monitor other areas of AI Trustworthiness such as relevance and bias.
Key Takeaways:
The implementation of Prisma for KPMG uncovered critical insights and risks. Below are the main points and conclusions after three months of monitoring:
Improved Risk Visibility: Prisma identified previously undetected risks in the AI system used by KPMG, including a 13.89% chance of hallucination, a 6% influence from “tone of input” on the output of the model, and a 5% influence from “clarify of input” on the output of the model as well. The average accuracy fell significantly, at 37.12% in certain environments and 55.78% in specific projects, indicating contextual limitations of the AI. Prisma’s monitoring capabilities provided enhanced visibility into these areas, allowing for real-time mitigation strategies to be implemented.
Effective Risk Mitigation: By leveraging Prisma's AI monitoring capabilities, KPMG was able to establish clear risk mitigation strategies that aligned with their governance, compliance, and technical needs. This helped reduce exposure to critical risks while improving system reliability. E.g. targeted training, better content curation, AI inference stack adjustments.
Enhanced AI Explainability: Prisma successfully integrated explainability features that allowed KPMG's AI systems to generate more transparent and interpretable outputs, identifying interactions where loss of performance was due to the tone of the input or its complexity. This ensured that stakeholders could better understand AI-driven decisions, fostering greater trust and enabling compliance with regulatory standards.
Business Value and ROI: The results demonstrated tangible business value for KPMG, including improved decision-making processes, faster implementation of mitigation actions, reduced operational risks, and a clear potential for a competitive advantage in the AI landscape.
Conclusion:
Prisma uncovered several key technical risks, including hallucination risk, contextual influence on output, inaccurate responses, and environment-specific performance gaps. Continuous monitoring and adjustments are necessary for maintaining consistent performance levels.
The reliability and compliance of KPMG’s AI system has been increased as a result of the implementation. Detailed reports, enforceable AI use policies, and real-time performance monitoring were key to ensuring operational stability and trustworthiness.
A substantial ROI and business value was delivered, enabling the roll-out of a scalable and efficient solution that optimizes resources and improves decision-making across Latin America. It also provided KPMG with a competitive edge through reliable and transparent AI deployment, reducing AI issues to KPMG and client´s AI developments.