Creation of XAI
Dramatic success in machine learning has created an explosion of new
Artificial Intelligence (AI) capabilities. Continued advances promise to
produce autonomous systems that perceive, learn, decide, and act on
their own. These systems offer tremendous benefits, but their
effectiveness will be limited by the machine’s inability to explain its
decisions and actions to human users. This issue is especially important
for the United States Department of Defense (DoD), which faces
challenges that require the development of more intelligent, autonomous,
and reliable systems. Explainable AI will be essential for users to
understand, appropriately trust, and effectively manage this emerging
generation of artificially intelligent partners.
The problem of explainability is, to some extent, the result of AI’s
success. In the early days of AI, the predominant reasoning methods were
logical and symbolic. These early systems reasoned by performing some
form of logical inference on (somewhat) human readable symbols. Early
systems could generate a trace of their inference steps, which could
then become the basis for explanation. As a result, there was
significant work on how to make these systems explainable (Shortliffe &
Buchanan, 1975; Swartout, Paris, & Moore, 1991; Johnson, 1994; Lacave
& Díez, 2002; Van Lent, Fisher, & Mancuso, 2004).
Yet these early AI systems were ineffective; they proved too expensive
to build and too brittle against the complexities of the real world.
Success in AI came as researchers developed new machine learning
techniques that could construct models of the world using their own
internal representations (e.g., support vectors, random forests,
probabilistic models, and neural networks). These new models were much
more effective, but necessarily more opaque and less explainable.
2015 was an inflection point in the need for XAI. Data analytics and
machine learning had just experienced a decade of rapid progress (Jordan
& Mitchell, 2015). The deep learning revolution had just begun,
following the breakthrough ImageNet demonstration in 2012 (Krizhevsky,
Sutskever, & Hinton, 2012). The popular press was alive with animated
speculation about Superintelligence (Bostrom, 2014) and the coming AI
Apocalypse (Gibbs, 2017, Cellan-Jones, 2014, Marr, 2018). Everyone
wanted to know how to understand, trust, and manage these mysterious,
seemingly inscrutable, AI systems.
2015 also saw the emergence of initial ideas for providing
explainability. Some researchers were exploring deep learning
techniques, such as the use of deconvolutional networks to visualize the
layers of convolutional networks (Zeiler & Fergus, 2014). Other
researchers were pursuing techniques to learn more interpretable models,
such as Bayesian Rule Lists (Letham, Rudin, McCormick, & Madigan,
2015). Others were developing model-agnostic techniques that could
experiment with a machine learning model—as a black box—to infer an
approximate, explainable model, such as LIME (Ribeiro, Singh, &
Guestrin, 2016). Yet others were evaluating the psychological and
human-computer interaction aspects of the explanation interface.
(Kulesza, Burnett, Wong, & Stumpf, 2015)
DARPA spent a year surveying researchers, analyzing possible research
strategies, and formulating the goals and structure of the program. In
August 2016, DARPA released DARPA-BAA-16-53 to call for proposals.
XAI Program Goals
The stated goal of Explainable Artificial Intelligence (XAI) was to create a suite of new or modified machine learning techniques
that produce explainable models that, when combined with effective
explanation techniques, enable end users to understand, appropriately
trust, and effectively manage the emerging generation of AI systems .
The target of XAI was an end user who depends on decisions or
recommendations produced by an AI system, or actions taken by it, and
therefore needs to understand the system’s rationale. For example, an
intelligence analyst who receives recommendations from a big data
analytics system needs to understand why it recommended certain activity
for further investigation. Similarly, an operator who tasks an
autonomous system needs to understand the system’s decision-making model
to appropriately use it in future missions. The XAI concept was to
provide users with explanations that enable them to understand the
system’s overall strengths and weaknesses; convey an understanding of
how it will behave in future/different situations; and perhaps permit
users to correct the system’s mistakes.
The XAI program assumed an inherent tension between machine learning
performance (e.g., predictive accuracy) and explainability, a concern
that was consistent with the research results at the time. Often the
highest performing methods (e.g., deep learning) were the least
explainable and the most explainable (e.g., decision trees) were the
least accurate. The program hoped to create a portfolio of new machine
learning and explanation techniques to provide future practitioners with
a wider range of design options covering the performance-explainability
trade space. If an application required higher performance, the XAI
portfolio would include more explainable, high performing, deep learning
techniques. If an application required more explainability, XAI would
include higher performing, interpretable models.
XAI Program Structure
The program was organized into three major technical areas (TAs), as
illustrated in Figure 1: (1) the development of new XAI machine learning
and explanation techniques for generating effective explanations; (2)
understanding the psychology of explanation by summarizing, extending
and applying psychological theories of explanation; and (3) evaluation
of the new XAI techniques in two challenge problem areas: data analytics
and autonomy.