What should your company know about explainable AI and its principles?

When speaking of AI, many people think of black box algorithms that take millions of input data points, work their magic, and deliver unexplainable results that users are supposed to trust. This kind of model is created directly from data, and not even its engineers can explain its outcome. Black box models, such as neural networks, possess superior skills when it comes to challenging prediction tasks. They produce results of remarkable accuracy, but no one can understand how algorithms arrived at their predictions. In contrast, with explainable white box AI, users can understand the rationale behind its decisions, making it increasingly popular in business settings. These models are not as technically impressive as black box algorithms. Still, their transparency is a tradeoff as it offers a higher level of reliability and is preferable in highly regulated industries.

Explainable AI (XAI) refers to a set of techniques, design principles, and processes that help developers/organizations add a layer of transparency to AI algorithms so that they can justify their predictions.XAI can describe AI models, their expected impact, and potential biases. With this technology, human experts can understand the resulting predictions and build trust and confidence in the results. When speaking of explainability, it all boils down to what you want to explain. There are two possibilities: 1. Explaining AI model’s pedigree: how the model was trained, which data was used, which types of bias are possible, and how they can be mitigated. 2. Explaining the overall model: this is also called “model interpretability.” There are two approaches to this technique: a. Proxy modeling: a more understandable model, such as a decision tree, is used as an approximation of a more cumbersome AI model. Even though this technique gives a simple overview of what to expect, it remains an approximation and can differ from real-life results. b. Design for interpretability: designing AI models in a way that forces simple easy-to-explain behavior. This technique can result in less powerful models as it eliminates some complicated tools from the developer’s toolkit.

The US National Institute of Standards and Technology (NIST) developed four explainable AI principles:

XAI can provide a detailed model-level explanation of why a particular decision was made. This explanation comes in a set of understandable rules. Following a simplified loan application example below, when applicants are denied a loan, they will receive a straightforward justification: everyone who is over 40 years old, saves less than $433 per month and applies for credit with a payback period of over 38 years will be denied a loan. The same goes for younger applicants who save less than $657 per month.

In some industries, an explanation is necessary for AI algorithms to be accepted. This can be either due to regulations and/or human factors. Think about brain tumor classification. No doctor will be comfortable preparing for a surgery solely based on “the algorithm said so.” And what about loan granting? Clients who got their application denied would want to understand why. Yes, there are more tolerant use cases where an explanation is not essential. For instance, predictive maintenance applications are not a matter of life or death, but even then, employees would feel more confident knowing why particular equipment might need preemptive repair. Senior management often understands the value of AI applications, but they also have their concerns. According to Gaurav Deshpande, VP of Marketing at TigerGraph, there is always a “but” in executives’ reasoning: “…but if you can’t explain how you arrived at the answer, I can’t use it. This is because of the risk of bias in the black box AI system that can lead to lawsuits and significant liability and risk to the company brand as well as the balance sheet.” The ideal XAI solution is the one that is reasonably accurate and can explain its results to practitioners, executives, and end-users. Incorporating explainable AI principles into intelligent software:

AI has many applications in healthcare. Various AI-powered medical solutions can save doctors’ time on repetitive tasks, allowing them to primarily focus on patient-facing care. Additionally, algorithms are good at diagnosing various health conditions as they can be trained to spot minor details that escape the human eye. However, when doctors cannot explain the outcome, they are hesitant to use this technology and act on its recommendations. One example comes from Duke University Hospital. A team of researchers installed a machine learning application called Sepsis Watch, which would send an alert when a patient was at risk of developing sepsis. The researchers discovered that doctors were skeptical of the algorithm and reluctant to act on its warnings because they did not understand it. This lack of trust is passed to patients who are hesitant to be examined by AI. Harvard Business Review published a study where participants were invited to take a free assessment of their stress level. 40% of the participants registered for the test when they knew a human doctor would do the evaluation. Only 26% signed up when an algorithm was performing the diagnosis. When it comes to diagnosing and treatments, the decisions made can be life changing. No surprise that doctors are desperate for transparency. Luckily, with explainable AI, this becomes a reality. For example, Keith Collins, CIO of SAS, mentioned his company is already developing such a technology. Here is what he said: “We’re presently working on a case where physicians use AI analytics to help detect cancerous lesions more accurately. The technology acts as the physician’s ‘virtual assistant,’ and it explains how each variable in an MRI image, for example, contributes to the technology identifying suspicious areas as probable for cancer while other suspicious areas are not.”

Finance is another heavily regulated industry where decisions need to be explained. It is vital that AI-powered solutions are auditable; otherwise, they will struggle to enter the market. AI can help assign credit scores, assess insurance claims, and optimize investment portfolios, among other applications. However, if the algorithms provide biased output, it can result in reputational loss and even lawsuits. Not long ago, Apple made headlines with its Apple Card product, which was inherently biased against women, lowering their credit limits. Apple’s co-founder, Steve Wozniak, confirmed this claim. He recalled that together with his wife, they have no separate bank accounts nor separate assets, and still, when applying for Apple Card, his granted limit was ten times higher than his wife’s. As a result of this unfortunate event, the company was investigated by the New York State Department of Financial Services. With explainable AI, one can avoid such scandalous situations by justifying the output. For example, loan granting is one use case that can benefit from XAI. The system would be able to justify its final recommendation and give clients a detailed explanation if their loan application was declined. This allows users to improve their credit profiles and reapply later.

Autonomous vehicles operate on vast amounts of data, requiring AI to analyze and make sense of it all. However, the system’s decisions need to be transparent for drivers, technologists, authorities, and insurance companies in case of any incidents. Also, it is crucial to understand how vehicles will behave in case of an emergency. Here is how Paul Appleby, former CEO of a data management software company Kinetica, voiced his concern: “If a self-driving car finds itself in a position where an accident is inevitable, what measures should it take? Prioritize the protection of the driver and put pedestrians in grave danger? Avoid pedestrians while putting the passengers’ safety at risk?” These are tough questions to answer, and people would disagree on how to handle such situations. But it is important to set guidelines that the algorithm can follow in such cases. This will help passengers decide whether they are comfortable traveling in a car designed to make certain decisions. Additionally, after an incident, the provided explanation will help developers improve the algorithm in the future.

AI has many applications in manufacturing, including predictive maintenance, inventory management, and logistics optimization. With its analytical capabilities, this technology can add to the “tribal knowledge” of human employees. But it is easier to adopt decisions when you understand the logic behind them. Heena Purohit, Senior Product Manager for IBM Watson IoT, explains how their AI-based maintenance product approaches explainable AI. The system offers human employees several options on how to repair a piece of equipment. Every option includes a percentage confidence interval. So, the user can still consult their “tribal knowledge” and expertise when making a choice. Also, each recommendation can project the knowledge graph output together with the input used in the training phase.

When your company is preparing to deploy responsible XAI solutions, the first thing would be to determine what exactly you need to explain and to whom. Some of the questions addressed during the planning stage can be:

Next, your company needs to decide on the degree of explainability. Not all AI-based tools require the same degree of interpretability. PwC identifies six application criticality components that will help you determine your desired XAI level:

Implementing traditional AI is full of challenges, let alone its explainable and responsible version. Despite the obstacles, it will bring relief to your employees, who will be more motivated to act upon the system’s recommendations when they understand the rationale behind it. Moreover, XAI will help you comply with your industry’s regulations and ethical considerations.