Four Principles of Explainable Artificial IntelligenceWe introduce four principles for explainable artificial intelligence (AI) that comprise fundamental properties for explainable AI systems. We propose that explainable AI systems deliver accompanying evidence or reasons for outcomes and processes; provide explanations that are understandable to individual users; provide explanations that correctly reflect the system’s process for generating the output; and that a system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output. We have termed these four principles as explanation, meaningful, explanation accuracy, and knowledge limits, respectively.Through significant stakeholder engagement, these four principles were developed to encompass the multidisciplinary nature of explainable AI, including the fields of computer science, engineering, and psychology. Because one-size-fits-all explanations do not exist, different users will require different types of explanations. We present five categories of explanation and summarize theories of explainable AI. We give an overview of the algorithms in the field that cover the major classes of explainable algorithms. As a baseline comparison, we assess how well explanations provided by people follow our four principles. This assessment provides insights to the challenges of designing explainable AI systems.National Institute of Standards and TechnologyNIST_36a9a026-66b6-11e0-86fc-e93d7a64ea2aInformation Access Division Information Technology LaboratoryP. Jonathon PhillipsCarina A. HahnPeter C. FontanaAmy N. YatesKristen GreeneMark A. PrzybockiInformation Technology LaboratoryDavid A. BroniatowskiTrustworthy AI_84b32146-d7ae-11ec-8473-5e9a1883ea00To introduce four principles for explainable artificial intelligence_84b322ea-d7ae-11ec-8473-5e9a1883ea00IntelligenceExplainabilityTrustExplanationProvide evidence or reason(s) for outputs and/or processes._84b323f8-d7ae-11ec-8473-5e9a1883ea001A system delivers or contains accompanying evidence or reason(s) for outputs and/or processes. | The Explanation principle states that for a system to be considered explainable it supplies evidence, support, or reasoning related to an outcome from or a process of an AI system. By itself, the explanation principle is independent of whether the explanation is correct, informative, or intelligible. This principle does not impose any metric of quality on those explanations. These factors are components of the meaningful and explanation accuracy principles. Explanations in practice will vary, and should, according to the given system and scenario. This means there will be a large range of ways an explanation can be executed or embedded into a system. To accommodate a large range of applications we adopt a deliberately broad definition of an explanation._84b32506-d7ae-11ec-8473-5e9a1883ea001.1.1Evidence/Reasoning_531a941c-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_1[Unspecified]AccuracyImproved... human-produced explanations for their own judgments,decisions, and conclusions are largely unreliable. Humans as a comparison group for explainable AI can inform the development of benchmark metrics for explainable AI systems;and lead to a better understanding of the dynamics of human-machine collaboration.Human Baseline ~ Largely Unreliable[To be determined]MeaningProvide explanations that are understandable to the intended consumer(s)._84b325f6-d7ae-11ec-8473-5e9a1883ea002A system provides explanations that are understandable to the intended consumer(s). | A system fulfills the Meaningful principle if the intended recipient understands the system’s explanation(s). There are commonalities across explanations which can make them more meaningful [84]... Meeting the Meaningful principle will be accomplished by understanding the audience’s needs, level of expertise, and relevancy to the question or query at hand. We provide a more detailed discussion of these purposes in Section 3. Measuring the meaningful principle is an area of ongoing work (Section 7.1). The challenge is to develop measurement protocols that adapt to different audiences. Rather than viewing this as a burden, we argue that both the awareness and appreciation of an explanation’s context support the ability to measure the quality of AI explanations. Scoping these factors will therefore bound the possibilities for how to execute the explanation in a goal-oriented and meaningful way.MisbehaviorState why systems did behave a certain way._84b326f0-d7ae-11ec-8473-5e9a1883ea002.1For example, stating why the system did behave a certain way can be more understandable than describing why it did not behave a certain way [76].2.1.1Explanation_531a9728-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_2[Unspecified]ComprehensibilityImproved... some assessments from humans may be more accurate when left automatic and implicit, compared to requiring an explicit judgment or explanation. Human judgments and decision making can oftentimes operate as a closed-box, and interfering with this closed-box process can be deleterious to the accuracy of a decision.Human Baseline ~ Questionable[To be determined]AudiencesConsider the intended audiences._84b327ea-d7ae-11ec-8473-5e9a1883ea002.2Many factors contribute to what individual people will consider a “good” explanation [55, 84, 139]. Therefore, developers need to consider the intended audience [44]. Several factors influence what information people will find important, relevant, or useful._531a9a34-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_3[Unspecified][To be determined]Knowledge, Experiences & DifferencesConsider prior knowledge and experiences and the overall psychological differences between people._84b328da-d7ae-11ec-8473-5e9a1883ea002.2.1These include a person’s prior knowledge and experiences and the overall psychological differences between people [18, 64, 90]. Moreover, what they consider meaningful will change over time as they gain experience with a task or system [18]. Different groups of people will also have different desires from a system’s explanations [13, 44, 50]._531a9b88-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_4[Unspecified][To be determined]Roles & RelationshipsDefine groups according to their role or relationship to the systems._84b329d4-d7ae-11ec-8473-5e9a1883ea002.2.2AI System DevelopersFor example: developers of a system are likely to have different desires from an explanation compared to an end-user.AI System UsersGroups may be defined broadly according to their role or relationship to the system._531a9ed0-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_5[Unspecified][To be determined]GoalsConsider the purposes of explanations._84b32b00-d7ae-11ec-8473-5e9a1883ea002.3In addition to its audience, what is considered meaningful will vary according to the explanation’s purpose. Different scenarios and needs will drive what is important and useful in a given context._531aa010-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_6[Unspecified][To be determined]AccuracyCorrectly reflect the reason for generating the output and/or accurately reflect the processes._84b32c22-d7ae-11ec-8473-5e9a1883ea003An explanation correctly reflects the reason for generating the output and/or accurately reflects the system’s process. | Together, the Explanation and Meaningful principles only call for a system to produce explanations that are intelligible to the intended audience. These two principles do not require that an explanation correctly reflects a system’s process for generating its output. The Explanation Accuracy principle imposes veracity on a system’s explanations. Explanation accuracy is a distinct concept from decision accuracy. Decision accuracy refers to whether the system’s judgment is correct or incorrect. Regardless of the system’s decision accuracy, the corresponding explanation may or may not accurately describe how the system came to its conclusion or action.Performance MetricsDevelop performance metrics for explanation accuracy._84b32d30-d7ae-11ec-8473-5e9a1883ea003.1Researchers in AI have developed standard measures of algorithm and system accuracy [23, 29, 52, 96, 97, 99, 103, 114]. While these established decision accuracy metrics exist, researchers are in the process of developing performance metrics for explanation accuracy. In Section 7.2, we review current work on this subject.3.1.1Explanation_531aa15a-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_7[Unspecified]AccuracyImprovedBased on our definition of explanation accuracy, these findings do not support the idea that humans reliably meet this criteria. As is the case with algorithms, human decision accuracy and explanation accuracy are distinct. For numerous tasks, humans can be highly accurate but cannot verbalize their decision process.Human Baseline ~ Unreliable[To be determined]DetailsAccount for levels of detail._84b32eb6-d7ae-11ec-8473-5e9a1883ea003.2Professors of NeuroscienceA professor of neuroscience may explain a new finding with extensive and technical details to a colleague.Undergraduate StudentsThat same finding will likely be distilled and changed for presenting to an undergraduate student in order to present the pertinent and higher level details. That same professor may explain the finding very differently to their untrained friends and parents.Additionally, explanation accuracy needs to account for the level of detail in the explanation. For some audiences and/or purposes, simple explanations will suffice. The given reasoning might succinctly focus on the critical point(s) or provide a high level reasoning without extensive detail. These simple explanations could lack nuances that are necessary to completely characterize the algorithm’s process for generating its output. However, these nuances may only be meaningful to certain audiences, such as experts of the system. This is similar to how humans approach explaining complex topics._531aa2a4-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_8[Unspecified][To be determined]FlexibilityAccommodate flexibility in explanation accuracy metrics._84b32fc4-d7ae-11ec-8473-5e9a1883ea003.2.1Together, this highlights the point that explanation accuracy and meaningfulness interact. A detailed explanation may accurately reflect the system’s processing, but sacrifice how useful and accessible it is to certain audiences. Likewise, a brief, simple explanation may be highly understandable but would not fully characterize the system. Given these considerations, this principle allows for flexibility in explanation accuracy metrics._531aa3f8-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_9[Unspecified][To be determined]LimitationsOperate only under design conditions and with confidence in outputs._84b330a0-d7ae-11ec-8473-5e9a1883ea004Knowledge Limits ~ A system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output. | The previous principles implicitly assume that a system is operating within the scope of its design and knowledge boundaries. The Knowledge Limits principle states that systems identify cases in which they were not designed or approved to operate, or in cases for which their answers are not reliable. By identifying and declaring knowledge limits, this practice safeguards answers so that a judgment is not provided when it may be inappropriate to do so. This principle can increase trust in a system by preventing misleading, dangerous, or unjust outputs. There are two ways a system can reach or exceed its knowledge limits.RelevanceEnsure that operations and queries are limited to the relevant domains._84b33190-d7ae-11ec-8473-5e9a1883ea004.1In one way, the operation or query to the system can be outside its domain. For example, in a system built to classify bird species, a user may input an image of an apple. The system could return an answer to indicate that it could not find any birds in the input image; therefore, the system cannot provide an answer. This is both an answer and an explanation._531aa646-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_10[Unspecified][To be determined]ConfidenceEnsure that confidence in the most likely answer is sufficiently high._84b33276-d7ae-11ec-8473-5e9a1883ea004.2In a second way, the confidence of the most likely answer may be too low, depending on an internal confidence threshold. To revisit an example of the bird classification system, the input image of a bird may be too blurry to determine its species. In this case, the system may recognize that the image is of a bird but that the image is of low quality. An example output may be: “I found a bird in the image, but the image quality is too low to identify it.”4.2.1Knowledge Limits_531aa902-d7b7-11ec-9da9-b7181a83ea00PLACEHOLDER_11[Unspecified]AssessmentImproved... human shortcomings in assessing their knowledge limits aresimilar to those of producing explanations themselves. When asked explicitly to produce an explanation, these explanations can interfere with more automatic processes gained by expertise; they often do not accurately reflect the true cognitive processes. Likewise ... when people are asked to explicitly predict or estimate their ability level relative to others, they are often inaccurate. However, when asked to assess their confidence for a given decision vs. this explicit judgment, people can gauge their accuracy at levels above chance. This suggests people do have insight into their own knowledge limits, although this insight can be limited or weak in some cases.Human Baseline ~ Limited or Weak[To be determined]_531aaa7e-d7b7-11ec-9da9-b7181a83ea002021-09-302022-05-19https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8312.pdf_531aabe6-d7b7-11ec-9da9-b7181a83ea00OwenAmburOwen.Ambur@verizon.net