Creation of XAI

Dramatic success in machine learning has created an explosion of new Artificial Intelligence (AI) capabilities. Continued advances promise to produce autonomous systems that perceive, learn, decide, and act on their own. These systems offer tremendous benefits, but their effectiveness will be limited by the machine’s inability to explain its decisions and actions to human users. This issue is especially important for the United States Department of Defense (DoD), which faces challenges that require the development of more intelligent, autonomous, and reliable systems. Explainable AI will be essential for users to understand, appropriately trust, and effectively manage this emerging generation of artificially intelligent partners.
The problem of explainability is, to some extent, the result of AI’s success. In the early days of AI, the predominant reasoning methods were logical and symbolic. These early systems reasoned by performing some form of logical inference on (somewhat) human readable symbols. Early systems could generate a trace of their inference steps, which could then become the basis for explanation. As a result, there was significant work on how to make these systems explainable (Shortliffe & Buchanan, 1975; Swartout, Paris, & Moore, 1991; Johnson, 1994; Lacave & Díez, 2002; Van Lent, Fisher, & Mancuso, 2004).
Yet these early AI systems were ineffective; they proved too expensive to build and too brittle against the complexities of the real world. Success in AI came as researchers developed new machine learning techniques that could construct models of the world using their own internal representations (e.g., support vectors, random forests, probabilistic models, and neural networks). These new models were much more effective, but necessarily more opaque and less explainable.
2015 was an inflection point in the need for XAI. Data analytics and machine learning had just experienced a decade of rapid progress (Jordan & Mitchell, 2015). The deep learning revolution had just begun, following the breakthrough ImageNet demonstration in 2012 (Krizhevsky, Sutskever, & Hinton, 2012). The popular press was alive with animated speculation about Superintelligence (Bostrom, 2014) and the coming AI Apocalypse (Gibbs, 2017, Cellan-Jones, 2014, Marr, 2018). Everyone wanted to know how to understand, trust, and manage these mysterious, seemingly inscrutable, AI systems.
2015 also saw the emergence of initial ideas for providing explainability. Some researchers were exploring deep learning techniques, such as the use of deconvolutional networks to visualize the layers of convolutional networks (Zeiler & Fergus, 2014). Other researchers were pursuing techniques to learn more interpretable models, such as Bayesian Rule Lists (Letham, Rudin, McCormick, & Madigan, 2015). Others were developing model-agnostic techniques that could experiment with a machine learning model—as a black box—to infer an approximate, explainable model, such as LIME (Ribeiro, Singh, & Guestrin, 2016). Yet others were evaluating the psychological and human-computer interaction aspects of the explanation interface. (Kulesza, Burnett, Wong, & Stumpf, 2015)
DARPA spent a year surveying researchers, analyzing possible research strategies, and formulating the goals and structure of the program. In August 2016, DARPA released DARPA-BAA-16-53 to call for proposals.

XAI Program Goals

The stated goal of Explainable Artificial Intelligence (XAI) was to create a suite of new or modified machine learning techniques that produce explainable models that, when combined with effective explanation techniques, enable end users to understand, appropriately trust, and effectively manage the emerging generation of AI systems .
The target of XAI was an end user who depends on decisions or recommendations produced by an AI system, or actions taken by it, and therefore needs to understand the system’s rationale. For example, an intelligence analyst who receives recommendations from a big data analytics system needs to understand why it recommended certain activity for further investigation. Similarly, an operator who tasks an autonomous system needs to understand the system’s decision-making model to appropriately use it in future missions. The XAI concept was to provide users with explanations that enable them to understand the system’s overall strengths and weaknesses; convey an understanding of how it will behave in future/different situations; and perhaps permit users to correct the system’s mistakes.
The XAI program assumed an inherent tension between machine learning performance (e.g., predictive accuracy) and explainability, a concern that was consistent with the research results at the time. Often the highest performing methods (e.g., deep learning) were the least explainable and the most explainable (e.g., decision trees) were the least accurate. The program hoped to create a portfolio of new machine learning and explanation techniques to provide future practitioners with a wider range of design options covering the performance-explainability trade space. If an application required higher performance, the XAI portfolio would include more explainable, high performing, deep learning techniques. If an application required more explainability, XAI would include higher performing, interpretable models.

XAI Program Structure

The program was organized into three major technical areas (TAs), as illustrated in Figure 1: (1) the development of new XAI machine learning and explanation techniques for generating effective explanations; (2) understanding the psychology of explanation by summarizing, extending and applying psychological theories of explanation; and (3) evaluation of the new XAI techniques in two challenge problem areas: data analytics and autonomy.