THE NATIONAL ARTIFICIAL INTELLIGENCE RESEARCH AND DEVELOPMENT STRATEGIC PLANThis National Artificial Intelligence R&D Strategic Plan establishes a set of objectives for Federallyfunded AI research, both research occurring within the government as well as Federally-funded research occurring outside of government, such as in academia. The ultimate goal of this research is to produce new AI knowledge and technologies that provide a range of positive benefits to society, while minimizing the negative impacts. To achieve this goal, this AI R&D Strategic Plan identifies the following priorities for Federally-funded AI research [documented as goals in this StratML rendition]Artificial intelligence (AI) is a transformative technology that holds promise for tremendous societal and economic benefit. AI has the potential to revolutionize how we live, work, learn, discover, and communicate. AI research can further our national priorities, including increased economic prosperity, improved educational opportunities and quality of life, and enhanced national and homeland security. Because of these potential benefits, the U.S. government has invested in AI research for many years. Yet, as with any significant technology in which the Federal government has interest, there are not only tremendous opportunities but also a number of considerations that must be taken into account in guiding the overall direction of Federally-funded R&D in AI.Subcommittee on Networking and Information Technology Research and DevelopmentNITRD_b3dd12be-e784-11df-a5d3-6a0c7a64ea2aThe Subcommittee on Networking and Information Technology Research and Development (NITRD) is a body under the Committee on Technology (CoT) of the National Science and Technology Council (NSTC). The NITRD Subcommittee coordinates multiagency research and development programs to help assure continued U.S. leadership in networking and information technology, satisfy the needs of the Federal Government for advanced networking and information technology, and accelerate development and deployment of advanced networking and information technology. It also implements relevant provisions of the High-Performance Computing Act of 1991 (P.L. 102-194), as amended by the Next Generation Internet Research Act of 1998 (P. L. 105-305), and the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education and Science (COMPETES) Act of 2007 (P.L. 110-69). For more information, see www.nitrd.gov.Bryan BiegelCo-Chair - Director, National Coordination Office for Networking and Information Technology Research and DevelopmentJames KuroseCo-Chair - Assistant Director, Computer and Information Science and Engineering, National Science FoundationNational Science and Technology CouncilThe National Science and Technology Council (NSTC) is the principal means by which the Executive Branch coordinates science and technology policy across the diverse entities that make up the Federal research and development (R&D) enterprise. One of the NSTC’s primary objectives is establishing clear national goals for Federal science and technology investments. The NSTC prepares R&D packages aimed at accomplishing multiple national goals. The NSTC’s work is organized under five committees: Environment, Natural Resources, and Sustainability; Homeland and National Security; Science, Technology, Engineering, and Mathematics (STEM) Education; Science; and Technology. Each of these committees oversees subcommittees and working groups that are focused on different aspects of science and technology. More information is available at www.whitehouse.gov/ostp/nstc.John P. HoldrenChair - Assistant to the President for Science and Technology and Director, Office of Science and Technology PolicyAfua BruceExecutive Director, Office of Science and Technology PolicySubcommittee on Machine Learning and Artificial IntelligenceOn May 3, 2016,the Administration announced the formation of a new NSTC Subcommittee on Machine Learning and Artificial intelligence, to help coordinate Federal activity in AI. This Subcommittee, on June 15, 2016, directed the Subcommittee on Networking and Information Technology Research and Development (NITRD) to create a National Artificial Intelligence Research and Development Strategic Plan. A NITRD Task Force on Artificial Intelligence was then formed to define the Federal strategic priorities for AI R&D, with particular attention on areas that industry is unlikely to address.Ed FeltenCo-Chair - Deputy U.S. Chief Technology Officer, Office of Science and Technology PolicyMichael GarrisCo-Chair - Senior Scientist, National Institute of Standards and Technology, U.S. Department of CommerceOffice of Science and Technology PolicyThe Office of Science and Technology Policy (OSTP) was established by the National Science and Technology Policy, Organization, and Priorities Act of 1976. The mission of OSTP is threefold; first, to provide the President and his senior staff with accurate, relevant, and timely scientific and technical advice on all matters of consequence; second, to ensure that the policies of the Executive Branch are informed by sound science; and third, to ensure that the scientific and technical work of the Executive Branch is properly coordinated so as to provide the greatest benefit to society. The Director of OSTP also serves as Assistant to the President for Science and Technology and manages the NSTC. More information is available at www.whitehouse.gov/ostp.NITRD Task Force on Artificial IntelligenceThis document was developed through the contributions of the members and staff of the NITRD Task Force on Artificial Intelligence. A special thanks and appreciation to additional contributors who helped write, edit, and review the document: Chaitan Baru (NSF), Eric Daimler (Presidential Innovation Fellow), Ronald Ferguson (DoD), Nancy Forbes (NITRD), Eric Harder (DHS), Erin Kenneally (DHS), Dai Kim (DoD), Tatiana Korelsky (NSF), David Kuehn (DOT), Terence Langendoen (NSF), Peter Lyster (NITRD), KC Morris (NIST), Hector Munoz-Avila (NSF), Thomas Rindflesch (NIH), Craig Schlenoff (NIST), Donald Sofge (NRL), and Sylvia Spengler (NSF).Lynne ParkerCo-Chair - Division Director, Information and Intelligent Systems, National Science FoundationJason MathenyCo-Chair - Director, Intelligence Advanced Research Projects ActivityMilton CornNational Institutes of HealthNikunj OzaNational Aeronautics and Space AdministrationWilliam FordNational Institute of JusticeRobinson PinoDepartment of EnergyMichael GarrisNational Institute of Standards and TechnologyGregory ShannonOffice of Science and Technology PolicySteven KnoxNational Security AgencyScott TousleyDepartment of Homeland SecurityJohn LaunchburyDefense Advanced Research Projects AgencyFaisal D'SouzaTechnical Coordinator, National Coordination Office for Networking and Information Technology Research and DevelopmentRichard LindermanOffice of the Secretary of DefenseManufacturersTechnological advances can lead to a new industrial revolution in manufacturing, including the entire engineering product life cycle. Increased used of robotics could enable manufacturing to move back onshore. AI can accelerate production capabilities through more reliable demand forecasting, increased flexibility in operations and the supply chain, and better prediction of the impacts of change to manufacturing operations. AI can create smarter, faster, cheaper, and more environmentally-friendly production processes that can increase worker productivity, improve product quality, lower costs, and improve worker health and safety. Machine learning algorithms can improve the scheduling of manufacturing processes and reduce inventory requirements. ConsumersConsumers can benefit from access to what is now commercial-grade 3-D printing.ShippersPrivate-sector manufacturers and shippers can use AI to improve supply-chain management through adaptive scheduling and routing. Supply Chain ManagersSupply chains can become more robust to disruption by automatically adjusting to anticipated effects of weather, traffic, and unforeseen events.Financial SectorIndustry and government can use AI to provide early detection of unusual financial risk at multiple scales. Safety controls can ensure that the automation in financial systems reduces opportunities for malicious behavior, such as market manipulation, fraud, and anomalous trading. They can additionally increase efficiency and reduce volatility and trading costs, all while preventing systemic failures such as pricing bubbles and undervaluing of credit risk.GovernmentsTransportation SectorAI can augment all modes of transportation to materially impact safety for all types of travel. It can be used in structural health monitoring and infrastructure asset management, providing increased trust from the public and reducing the costs of repairs and reconstruction. AI can be used in passenger and freight vehicles to improve safety by increasing situational awareness, and to provide drivers and other travelers with real-time route information. AI applications can also improve network-level mobility and reduce overall system energy use and transportation-related emissions.Agricultural SectorAI systems can create approaches to sustainable agriculture that are smarter about the production, processing, storage, distribution, and consumption of agricultural products. AI and robotics can gather site-specific and timely data about crops, apply needed inputs (e.g., water, chemicals, fertilizers) only when and where they are needed, and fill urgent gaps in the agricultural labor force.Marketing SectorAI approaches can enable commercial entities to better match supply with demand, driving up revenue that funds ongoing private sector development. It can anticipate and identify consumer needs, enabling them to better find the products and services they want, at lower cost.Communications SectorAI technologies can maximize efficient use of bandwidth and automation of information storage and retrieval. AI can improve filtering, searching, language translation, and summarization of digital communications, positively affecting commerce and the way we live our lives.Science & Technology SectorsAI systems can assist scientists and engineers in reading publications and patents, refining theories to be more consistent with prior observations, generating testable hypotheses, performing experiments using robotic systems and simulations, and engineering new devices and software.ScientistsEngineersEducational SectorAI-enhanced learning schools can be universally available, with automated tutoring that gauges the development of the student. AI tutors can complement in-person teachers and focus education on advanced and/or remedial learning appropriate to the student. AI tools can foster life-long learning and the acquisition of new skills for all members of society.Health Care SectorAI can support bioinformatics systems that identify genetic risks from large-scale genomic studies (e.g., genome-wide association studies, sequencing studies), and predict the safety and efficacy of new pharmaceuticals. AI techniques can allow assessments across multidimensional data to study public health issues and to provide decision support systems for medical diagnoses and prescribe treatments. AI technologies are required for the customization of drugs for the individual; the result can be increased medical efficacy, patient comfort, and less waste.Legal SectorThe analysis of law case history by machines can become widespread. The increased sophistication of these processes can allow for a richer level of analysis for assisting the discovery process. Legal discovery tools can identify and summarize relevant evidence; these systems may even formulate legal arguments with increasing sophistication.Personal Services SectorAI software can make use of knowledge from multiple sources to provide more accurate information for a multitude of uses. Natural language systems can provide intuitive interfaces to technological systems in real-world, noisy environments. Personalized tools can enable automated assistance with individual and group scheduling. Text can be automatically summarized from multiple search outcomes, enhanced across multiple media. AI can enable real-time spoken multi-lingual translation.Law Enforcement OfficialsLaw enforcement and security officials can help create a safer society through the use of pattern detection to detect anomalous behavior in individual actors, or to predict dangerous crowd behavior. Intelligent perception systems can protect critical infrastructure, such as airports and power plants.Security OfficialsInfrastructure ManagersSafety and prediction: Distributed sensor systems and pattern understanding of normal conditions can detect when the probability of major infrastructure disruptions increases significantly, whether triggered by natural or man-made causes. This anticipatory capability can help indicate where the problem will be, to adapt operations to forestall disruption as, or even before it happens.... a future world in which AI is safely used for significant benefit to all members of society._d0b9d134-9547-11e6-bea8-c06b742aff29To convey a clear set of R&D priorities that address strategic research goals, focus Federal investments on those areas in which industry is unlikely to invest, and address the need to expand and sustain the pipeline of AI R&D talent._d0b9d33c-9547-11e6-bea8-c06b742aff29ProsperityIncreased economic prosperity: New products and services can create new markets, and improve the quality and efficiency of existing goods and services across multiple industries. More efficient logistics and supply chains are being created through expert decision systems. Products can be transported more effectively through vision-based driver-assist and automated/robotic systems. Manufacturing can be improved through new methods for controlling fabrication processes and scheduling work flows.Social WellbeingImproved educational opportunity and quality of life: Lifelong learning can be possible through virtual tutors that develop customized learning plans to challenge and engage each person based on their interests, abilities, and educational needs. People can live healthier and more active lives, using personalized health information tailored and adapted for each individual. Smart homes and personal virtual assistants can save people time and reduce time lost in daily repetitive tasks.SecurityEnhanced national and homeland security: Machine learning agents can process large amounts of intelligence data and identify relevant patterns-of-life from adversaries with rapidly changing tactics. These agents can also provide protection to critical infrastructure and major economic sectors that are vulnerable to attack. Digital defense systems can significantly reduce battlefield risks and casualties.LearningThe AI field is now in the beginning stages of a possible third wave, which focuses on explanatory and general AI technologies. The goals of these approaches are to enhance learned models with an explanation and correction interface, to clarify the basis for and reliability of outputs, to operate with a high degree of transparency, and to move beyond narrow AI to capabilities that can generalize across broader task domains. If successful, engineers could create systems that construct explanatory models for classes of real world phenomena, engage in natural communication with people, learn and reason as they encounter new tasks and situations, and solve novel problems by generalizing from past experience. Explanatory models for these AI systems might be constructed automatically through advanced methods. These models could enable rapid learning in AI systems. They may supply "meaning" or “understanding” to the AI system, which could then enable the AI systems to achieve more general capabilities.InvestmentMake long-term investments in AI research. _d0b9d4a4-9547-11e6-bea8-c06b742aff29Strategy 1Prioritize investments in the next generation of AI that will drive discovery and insight and enable the United States to remain a world leader in AI.Knowledge DiscoveryAdvance data-focused methodologies for knowledge discovery_d0b9d8d2-9547-11e6-bea8-c06b742aff291.1As discussed in the Federal Big Data Research and Development Strategic Plan, many fundamental new tools and technologies are needed to achieve intelligent data understanding and knowledge discovery. Further progress is needed in the development of more advanced machine learning algorithms that can identify all the useful information hidden in big data. Many open research questions revolve around the creation and use of data, including its veracity and appropriateness for AI system training. The veracity of data is particularly challenging when dealing with vast amounts of data, making it difficult for humans to assess and extract knowledge from it. While much research has dealt with veracity through data quality assurance methods to perform data cleaning and knowledge discovery, further study is needed to improve the efficiency of data cleaning techniques, to create methods for discovering inconsistencies and anomalies in the data, and to develop approaches for incorporating human feedback. Researchers need to explore new methods to enable data and associated metadata to be mined simultaneously. Many AI applications are interdisciplinary in nature and make use of heterogeneous data. Further investigation of multi-modality machine learning is needed to enable knowledge discovery from a wide variety of different types of data (e.g., discrete, continuous, text, spatial, temporal, spatio-temporal, graphs). AI investigators must determine the amount of data needed for training and to properly address large-scale versus long-tail data needs. They must also determine how to identify and process rare events beyond purely statistical approaches; to work with knowledge sources (i.e., any type of information that explains the world, such as knowledge of the law of gravity or of social norms) as well as data sources, integrating models and ontologies in the learning process; and to obtain effective learning performance with little data when big data sources may not be available.PerceptionEnhance the perceptual capabilities of AI systems_d0b9da4e-9547-11e6-bea8-c06b742aff291.2Perception is an intelligent system's window into the world. Perception begins with (possibly distributed) sensor data, which comes in diverse modalities and forms, such as the status of the system itself or information about the environment. Sensor data are processed and fused, often along with a priori knowledge and models, to extract information relevant to the AI system's task such as geometric features, attributes, location, and velocity. Integrated data from perception forms situational awareness to provide AI systems with the comprehensive knowledge and a model of the state of the world necessary to plan and execute tasks effectively and safely. AI systems would greatly benefit from advancements in hardware and algorithms to enable more robust and reliable perception. Sensors must be able to capture data at longer distances, with higher resolution, and in real time. Perception systems need to be able to integrate data from a variety of sensors and other sources, including the computational cloud, to determine what the AI system is currently perceiving and to allow the prediction of future states. Detection, classification, identification, and recognition of objects remain challenging, especially under cluttered and dynamic conditions. In addition, perception of humans must be greatly improved by using an appropriate combination of sensors and algorithms, so that AI systems can work more effectively with people. A framework for calculating and propagating uncertainty throughout the perception process is needed to quantify the confidence level that the AI system has in its situational awareness and to improve accuracy.Capabilities & LimitationsUnderstand theoretical capabilities and limitations of AI_d0b9dbb6-9547-11e6-bea8-c06b742aff291.3While the ultimate goal for many AI algorithms is to address open challenges with human-like solutions, we do not have a good understanding of what the theoretical capabilities and limitations are for AI and the extent to which such human-like solutions are even possible with AI algorithms. Theoretical work is needed to better understand why AI techniques -- especially machine learning -- often work well in practice. While different disciplines (including mathematics, control sciences, and computer science) are studying this issue, the field currently lacks unified theoretical models or frameworks to understand AI system performance. Additional research is needed on computational solvability, which is an understanding of the classes of problems that AI algorithms are theoretically capable of solving, and likewise, those that they are not capable of solving. This understanding must be developed in the context of existing hardware, in order to see how the hardware affects the performance of these algorithms. Understanding which problems are theoretically unsolvable can lead researchers to develop approximate solutions to these problems, or even open up new lines of research on new hardware for AI systems. For example, when invented in the 1960s, Artificial Neural Networks (ANNs) could only be used to solve very simple problems. It only became feasible to use ANNs to solve complex problems after hardware improvements such as parallelization were made, and algorithms were adjusted to make use of the new hardware. Such developments were key factors in enabling today’s significant advances in deep learning.GeneralizationPursue research on general-purpose artificial intelligence_d0b9dd3c-9547-11e6-bea8-c06b742aff291.4AI approaches can be divided into "narrow AI" and "general AI." Narrow AI systems perform individual tasks in specialized, well-defined domains, such as speech recognition, image recognition, and translation. Several recent, highly-visible, narrow AI systems, including IBM Watson and DeepMind's AlphaGo, have achieved major feats. Indeed, these particular systems have been labeled "superhuman" because they have outperformed the best human players in Jeopardy and Go, respectively. But these systems exemplify narrow AI, since they can only be applied to the tasks for which they are specifically designed. Using these systems on a wider range of problems requires a significant re-engineering effort. In contrast, the long-term goal of general AI is to create systems that exhibit the flexibility and versatility of human intelligence in a broad range of cognitive domains, including learning, language, perception, reasoning, creativity, and planning. Broad learning capabilities would provide general AI systems the ability to transfer knowledge from one domain to another and to interactively learn from experience and from humans. General AI has been an ambition of researchers since the advent of AI, but current systems are still far from achieving this goal. The relationship between narrow and general AI is currently being explored; it is possible that lessons from one can be applied to improve the other and vice versa. While there is no general consensus, most AI researchers believe that general AI is still decades away, requiring a long-term, sustained research effort to achieve it.ScalabilityDevelop scalable AI systems_d0b9deae-9547-11e6-bea8-c06b742aff291.5Groups and networks of AI systems may be coordinated or autonomously collaborate to perform tasks not possible with a single AI system, and may also include humans working alongside or leading the team. The development and use of such multi-AI systems creates significant research challenges in planning, coordination, control, and scalability of such systems. Planning techniques for multi-AI systems must be fast enough to operate and adapt in real time to changes in the environment. They should adapt in a fluid manner to changes in available communications bandwidth or system degradation and faults. Many prior efforts have focused on centralized planning and coordination techniques; however, these approaches are subject to single points of failure, such as the loss of the planner, or loss of the communications link to the planner. Distributed planning and control techniques are harder to achieve algorithmically, and are often less efficient and incomplete, but potentially offer greater robustness to single points of failure. Future research must discover more efficient, robust, and scalable techniques for planning, control, and collaboration of teams of multiple AI systems and humans.AnthropomorphismFoster research on human-like AI_d0b9e020-9547-11e6-bea8-c06b742aff291.6Attaining human-like AI requires systems to explain themselves in ways that people can understand. This will result in a new generation of intelligent systems, such as intelligent tutoring systems and intelligent assistants that are effective in assisting people when performing their tasks. There is a significant gap, however, between the way current AI algorithms work and how people learn and perform tasks. People are capable of learning from just a few examples, or by receiving formal instruction and/or "hints" to performing tasks, or by observing other people performing those tasks. Medical schools take this approach, for example, when medical students learn by observing an established doctor performing a complex medical procedure. Even in high-performance tasks such as world-championship Go games, a master-level player would have played only a few thousand games to train him/herself. In contrast, it would take hundreds of years for a human to play the number of games needed to train AlphaGo. More foundational research on new approaches for achieving human-like AI would bring these systems closer to this goal. RobotsDevelop more capable and reliable robots_d0b9e1c4-9547-11e6-bea8-c06b742aff291.7Significant advances in robotic technologies over the last decade are leading to potential impacts in a multiplicity of applications, including manufacturing, logistics, medicine, healthcare, defense and national security, agriculture, and consumer products. While robots were historically envisioned for static industrial environments, recent advances involve close collaborations between robots and humans. Robotics technologies are now showing promise in their ability to complement, augment, enhance, or emulate human physical capabilities or human intelligence. However, scientists need to make these robotic systems more capable, reliable, and easy-to-use. Researchers need to better understand robotic perception to extract information from a variety of sensors to provide robots with real-time situational awareness. Progress is needed in cognition and reasoning to allow robots to better understand and interact with the physical world. An improved ability to adapt and learn will allow robots to generalize their skills, perform self-assessment of their current performance, and learn a repertoire of physical movements from human teachers. Mobility and manipulation are areas for further investigation so that robots can move across rugged and uncertain terrain and handle a variety of objects dexterously. Robots need to learn to team together in a seamless fashion and collaborate with humans in a way that is trustworthy and predictable.HardwareAdvance hardware for improved AI_d0b9e340-9547-11e6-bea8-c06b742aff291.8While AI research is most commonly associated with advances in software, the performance of AI systems has been heavily dependent on the hardware upon which it runs. The current renaissance in deep machine learning is directly tied to progress in GPU-based hardware technology and its improved memory, input/output, clock speeds, parallelism, and energy efficiency. Developing hardware optimized for AI algorithms will enable even higher levels of performance than GPUs. One example is "neuromorphic" processors that are loosely inspired by the organization of the brain and,76 in some cases, optimized for the operation of neural networks. Hardware advances can also improve the performance of AI methods that are highly data-intensive. Further study of methods to turn on and off data pipelines in controlled ways throughout a distributed system is called for. Continued research is also needed to allow machine learning algorithms to efficiently learn from high-velocity data, including distributed machine learning algorithms that simultaneously learn from multiple data pipelines. More advanced machine learning-based feedback methods will allow AI systems to intelligently sample or prioritize data from large-scale simulations, experimental instruments, and distributed sensor systems, such as Smart Buildings and the Internet of Things (IoT). Such methods may require dynamic I/O decision-making, in which choices are made in real time to store data based on importance or significance, rather than simply storing data at fixed frequencies.Hardware AICreate AI for improved hardware_d0b9e4bc-9547-11e6-bea8-c06b742aff291.9While improved hardware can lead to more capable AI systems, AI systems can also improve the performance of hardware. This reciprocity will lead to further advances in hardware performance, since physical limits on computing require novel approaches to hardware designs. AI-based methods could be especially important for improving the operation of high performance computing (HPC) systems. Such systems consume vast quantities of energy. AI is being used to predict HPC performance and resource usage, and to make online optimization decisions that increase efficiency; more advanced AI techniques could further enhance system performance. AI can also be used to create self-reconfigurable HPC systems that can handle system faults when they occur, without human intervention. Improved AI algorithms can increase the performance of multi-core systems by reducing data movements between processors and memory -- the primary impediment to exascale computing systems that operate 10 times faster than today's supercomputers. In practice, the configuration of executions in HPC systems are never the same, and different applications are executed concurrently, with the state of each different software code evolving independently in time. AI algorithms need to be designed to operate online and at scale for HPC systems. Human-AI CollaborationDevelop effective methods for human-AI collaboration. _d0b9e70a-9547-11e6-bea8-c06b742aff29Strategy 2Rather than replace humans, most AI systems will collaborate with humans to achieve optimal performance. Research is needed to create effective interactions between humans and AI systems. -- While completely autonomous AI systems will be important in some application domains (e.g., underwater or deep space exploration), many other application areas (e.g., disaster recovery and medical diagnostics) are most effectively addressed by a combination of humans and AI systems working together to achieve application goals. This collaborative interaction takes advantage of the complementary nature of humans and AI systems. While effective approaches for human-AI collaboration already exist, most of these are “point solutions” that only work in specific environments using specific platforms toward specific goals. Generating point solutions for every possible application instance does not scale; more work is thus needed to go beyond these point solutions toward more general methods of human-AI collaboration. The tradeoffs must be explored between designing general systems that work in all types of problems, requiring less human effort to build, and greater facility for switching between applications, versus building a large number of problem-specific systems that may work more effectively for each problem. Future applications will vary considerably in the functional role divisions between humans and AI systems, the nature of the interactions between humans and AI systems, the number of humans and other AI systems working together, and how humans and AI systems will communicate and share situational awareness. Functional role divisions between humans and AI systems typically fall into one of the following categories: 1. AI performs functions alongside the human: AI systems perform peripheral tasks that support the human decision maker. For example, AI can assist humans with working memory, short or long-term memory retrieval, and prediction tasks. 2. AI performs functions when the human encounters high cognitive overload: AI systems perform complex monitoring functions (such as ground proximity warning systems in aircraft), decision making, and automated medical diagnoses when humans need assistance. 3. AI performs functions in lieu of a human: AI systems perform tasks for which humans have very limited capabilities, such as for complex mathematical operations, control guidance for dynamic systems in contested operational environments, aspects of control for automated systems in harmful or toxic environments, and in situations where a system should respond very rapidly (e.g., in nuclear reactor control rooms). Achieving effective interactions between humans and AI systems requires additional R&D to ensure that the system design does not lead to excessive complexity, undertrust, or overtrust. The familiarity of humans with the AI systems can be increased through training and experience, to ensure that the human has a good understanding of the AI system’s capabilities and what the AI system can and cannot do. To address these concerns, certain human-centered automation principles should be used in the design and development of these systems: 1. Employ intuitive, user-friendly design of human-AI system interfaces, controls, and displays. 2. Keep the operator informed. Display critical information, states of the AI system, and changes to these states. 3. Keep the operator trained. Engage in recurrent training for general knowledge, skills, and abilities (KSAs), as well as training in algorithms and logic employed by AI systems and the expected failure modes of the system. 4. Make automation flexible. Deploying AI systems should be considered as a design option for operators who wish to decide whether they want to use them or not. Also important is the design and deployment of adaptive AI systems that can be used to support human operators during periods of excessive workload or fatigue. Many fundamental challenges arise for researchers when creating systems that work effectively with humans. Several of these important challenges are outlined in the following subsections.Human Awareness[Develop] new algorithms for human-aware AI_d0b9e8a4-9547-11e6-bea8-c06b742aff292.1Over the years, AI algorithms have become able to solve problems of increasing complexity. However, there is a gap between the capabilities of these algorithms and the usability of these systems by humans. Human-aware intelligent systems are needed that can interact intuitively with users and enable seamless machine-human collaborations. Intuitive interactions include shallow interactions, such as when a user discards an option recommended by the system; model-based approaches that take into account the users’ past actions; or even deep models of user intent that are based upon accurate human cognitive models. Interruption models must be developed that allow an intelligent system to interrupt the human only when necessary and appropriate. Intelligent systems should also have the ability to augment human cognition, knowing which information to retrieve when the user needs it, even when they have not prompted the system explicitly for that information. Future intelligent systems must be able to account for human social norms and act accordingly. Intelligent systems can more effectively work with humans if they possess some degree of emotional intelligence, so that they can recognize their users’ emotions and respond appropriately. An additional research goal is to go beyond interactions of one human and one machine, toward a "systems-of-systems", that is, teams composed of multiple machines interacting with multiple humans. Human-AI system interactions have a wide range of objectives. AI systems need the ability to represent a multitude of goals, actions that they can take to reach those goals, constraints on those actions, and other factors, as well as easily adapt to modifications in the goals. In addition, humans and AI systems must share common goals and have a mutual understanding of them and relevant aspects of their current states. Further investigation is needed to generalize these facets of human-AI systems to develop systems that require less human engineering.Human AugmentationDevelop AI techniques for human augmentation_d0b9ea5c-9547-11e6-bea8-c06b742aff292.2While much of the prior focus of AI research has been on algorithms that match or outperform people performing narrow tasks, additional work is needed to develop systems that augment human capabilities across many domains. Human augmentation research includes algorithms that work on a stationary device (such as a computer); wearable devices (such as smart glasses); implanted devices (such as brain interfaces); and in specific user environments (such as specially tailored operating rooms). For example, augmented human awareness could enable a medical assistant to point out a mistake in a medical procedure, based on data readings combined from multiple devices. Other systems could augment human cognition by helping the user recall past experiences applicable to the user’s current situation. Another type of collaboration between humans and AI systems involves active learning for intelligent data understanding. In active learning, input is sought from a domain expert and learning is only performed on data when the learning algorithm is uncertain. This is an important technique to reduce the amount of training data that needs to be generated in the first place, or the amount that needs to be learned. Active learning is also a key way to obtain domain expert input and increase trust in the learning algorithm. Active learning has so far only been used within supervised learning—further research is needed to incorporate active learning into unsupervised learning (e.g., clustering, anomaly detection) and reinforcement learning.84 Probabilistic networks allow domain knowledge to be included in the form of prior probability distributions. General ways of allowing machine learning algorithms to incorporate domain knowledge must be sought, whether in the form of mathematical models, text, or others.Visualization & UIsDevelop techniques for visualization and AI-human interfaces_d0b9f33a-9547-11e6-bea8-c06b742aff292.3Better visualization and user interfaces are additional areas that need much greater development to help humans understand large-volume modern datasets and information coming from a variety of sources. Visualization and user interfaces must clearly present increasingly complex data and information derived from them in a human-understandable way. Providing real-time results is important in safety-critical operations and may be achieved with increasing computational power and connected systems. In these types of situations, users need visualization and user interfaces that can quickly convey the correct information for real-time response. Human-AI collaboration can be applied in a wide variety of environments, and where there are constraints on communication. In some domains, human-AI communication latencies are low and communication is rapid and reliable. In other domains (e.g., NASA’s deployment of the rovers Spirit and Opportunity to Mars), remote communication between humans and the AI system has a very high latency (e.g., round trip times of 5-20 minutes between Earth and Mars), thus requiring the deployed platform(s) to operate largely autonomously, with only high-level strategic goals communicated to the platform. These communications requirements and constraints are important considerations for the R&D of user interfaces.Language ProcessingDevelop more effective language processing systems_d0b9f33b-9547-11e6-bea8-c06b742aff292.4Enabling people to interact with AI systems through spoken and written language has long been a goal of AI researchers. While significant advances have been made, considerable open research challenges must be addressed in language processing before humans can communicate as effectively with AI systems as they do with other humans. Much recent progress in language processing has been credited to the use of data-driven machine learning approaches, which have resulted in successful systems that, for example, successfully recognize fluent English speech in quiet surroundings in real time. These achievements, however, are only first steps toward reaching longer-term goals. Current systems cannot deal with real-world challenges such as speech in noisy surroundings, heavily accented speech, children’s speech, impaired speech, and speech for sign languages. The development of language processing systems capable of engaging in real-time dialogue with humans is also needed. Such systems will need to infer the goals and intentions of its human interlocutors, use the appropriate register, style and rhetoric for the situation, and employ repair strategies in case of dialogue misunderstandings. Further research is needed on developing systems that more easily generalize across different languages. Additionally, more study is required on acquiring useful structured domain knowledge in a form readily accessible by language processing systems. Language processing advances in many other areas are also needed to make interactions between humans and AI systems more natural and intuitive. Robust computational models must be built for patterns in both spoken and written language that provide evidence for emotional state, affect, and stance, and for determining the information that is implicit in speech and text. New language processing techniques are needed for grounding language in the environmental context for AI systems that operate in the physical world, such as in robotics. Finally, since the manner in which people communicate in online interactions can be quite different from voice interactions, models of languages used in these contexts must be perfected so that social AI systems can interact more effectively with people.Ethics, Legalities & SocietyUnderstand and address the ethical, legal, and societal implications of AI. _d0b9f6b4-9547-11e6-bea8-c06b742aff29Strategy 3We expect AI technologies to behave according to the formal and informal norms to which we hold our fellow humans. Research is needed to understand the ethical, legal, and social implications of AI, and to develop methods for designing AI systems that align with ethical, legal, and societal goals. -- When AI agents act autonomously, we expect them to behave according to the formal and informal norms to which we hold our fellow humans. As fundamental social ordering forces, law and ethics therefore both inform and adjudge the behavior of AI systems. The dominant research needs involve both understanding the ethical, legal, and social implications of AI, as well as developing methods for AI design that align with ethical, legal, and social principles. Privacy concerns must also be taken into account; further information on this issue can be found in the National Privacy Research Strategy. As with any technology, the acceptable uses of AI will be informed by the tenets of law and ethics; the challenge is how to apply those tenets to this new technology, particularly those involving autonomy, agency, and control. As illuminated in "Research Priorities for Robust and Beneficial Artificial Intelligence": "In order to build systems that robustly behave well, we of course need to decide what good behavior means in each application domain. This ethical dimension is tied intimately to questions of what engineering techniques are available, how reliable these techniques are, and what trade-offs are made -- all areas where computer science, machine learning, and broader AI expertise is valuable." Research in this area can benefit from multidisciplinary perspectives that involve experts from computer science, social and behavioral sciences, ethics, biomedical science, psychology, economics, law, and policy research. Further investigation is needed in areas both inside and outside of the NITRD-relevant IT domain (i.e., in information technology as well as the disciplines mentioned above) to inform the R&D and use of AI systems and their impacts on society. The following subsections explore key information technology research challenges in this area.Fairness, Transparency & AccountabilityImprove fairness, transparency, and accountability-by-design_d0b9f740-9547-11e6-bea8-c06b742aff293.1Many concerns have been voiced about the susceptibility of data-intensive AI algorithms to error and misuse, and the possible ramifications for gender, age, racial, or economic classes. The proper collection and use of data for AI systems, in this regard, represent an important challenge. Beyond purely data-related issues, however, larger questions arise about the design of AI to be inherently just, fair, transparent, and accountable. Researchers must learn how to design these systems so that their actions and decision-making are transparent and easily interpretable by humans, and thus can be examined for any bias they may contain, rather than just learning and repeating these biases. There are serious intellectual issues about how to represent and "encode" value and belief systems. Scientists must also study to what extent justice and fairness considerations can be designed into the system, and how to accomplish this within the bounds of current engineering techniques.EthicsBuild ethical AI_8ad7a2fc-954e-11e6-9ea9-559e742aff293.2Beyond fundamental assumptions of justice and fairness are other concerns about whether AI systems can exhibit behavior that abides by general ethical principles. How might advances in AI frame new "machine-relevant" questions in ethics, or what uses of AI might be considered unethical? Ethics is inherently a philosophical question while AI technology depends on, and is limited by, engineering. Within the limits of what is technologically feasible, therefore, researchers must strive to develop algorithms and architectures that are verifiably consistent with, or conform to, existing laws, social norms and ethics -- clearly a very challenging task. Ethical principles are typically stated with varying degrees of vagueness and are hard to translate into precise system and algorithm design. There are also complications when AI systems, particularly with new kinds of autonomous decision-making algorithms, face moral dilemmas based on independent and possibly conflicting value systems. Ethical issues vary according to culture, religion, and beliefs. However, acceptable ethics reference frameworks can be developed to guide AI system reasoning and decision-making, in order to explain and justify its conclusions and actions. A multi-disciplinary approach is needed to generate datasets for training that reflect an appropriate value system, including examples that indicate preferred behavior when presented with difficult moral issues or with conflicting values. These examples can include legal or ethical "corner cases", labeled by an outcome or judgment that is transparent to the user. AI needs adequate methods for values-based conflict resolution, where the system incorporates principles that can address the realities of complex situations where strict rules are impracticable.ArchitecturesDesign architectures for ethical AI_8ad7a5e0-954e-11e6-9ea9-559e742aff293.3Additional progress in fundamental research must be made to determine how to best design architectures for AI systems that incorporate ethical reasoning. A variety of approaches have been suggested, such as a two-tier monitor architecture that separates the operational AI from a monitor agent that is responsible for the ethical or legal assessment of any operational action.87 An alternative view is that safety engineering is preferred, in which a precise conceptual framework for the AI agent architecture is used to ensure that AI behavior is safe and not harmful to humans.88 A third method is to formulate an ethical architecture using set theoretic principles, combined with logical constraints on AI system behavior that restrict action to conform to ethical doctrine.89 As AI systems become more general, their architectures will likely include subsystems that can take on ethical issues at multiple levels of judgment, including: 90 rapid response pattern matching rules, deliberative reasoning for slower responses for describing and justifying actions, social signaling to indicate trustworthiness for the user, and social processes that operate over even longer time scales to enable the system to abide by cultural norms. Researchers will need to focus on how to best address the overall design of AI systems that align with ethical, legal, and societal goals.Safety & SecurityEnsure the safety and security of AI systems. _d0b9f741-9547-11e6-bea8-c06b742aff29Strategy 4Before AI systems are in widespread use, assurance is needed that the systems will operate safely and securely, in a controlled, well-defined, and well-understood manner. Further progress in research is needed to address this challenge of creating AI systems that are reliable, dependable, and trustworthy. -- Before an AI system is put into widespread use, assurance is needed that the system will operate safely and securely, in a controlled manner. Research is needed to address this challenge of creating AI systems that are reliable, dependable, and trustworthy. As with other complex systems, AI systems face important safety and security challenges due to: * Complex and uncertain environments: In many cases, AI systems are designed to operate in complex environments, with a large number of potential states that cannot be exhaustively examined or tested. A system may confront conditions that were never considered during its design. * Emergent behavior: For AI systems that learn after deployment, a system's behavior may be determined largely by periods of learning under unsupervised conditions. Under such conditions, it may be difficult to predict a system's behavior. * Goal misspecification: Due to the difficulty of translating human goals into computer instructions, the goals that are programmed for an AI system may not match the goals that were intended by the programmer. * Human-machine interactions: In many cases, the performance of an AI system is substantially affected by human interactions. In these cases, variation in human responses may affect the safety of the system. To address these issues and others, additional investments are needed to advance AI safety and security, including explainability and transparency, trust, verification and validation, security against attacks, and long-term AI safety and value-alignment.Explainability & TransparencyImprove explainability and transparency_d0b9f772-9547-11e6-bea8-c06b742aff294.1A key research challenge is increasing the "explainability" or "transparency" of AI. Many algorithms, including those based on deep learning, are opaque to users, with few existing mechanisms for explaining their results. This is especially problematic for domains such as healthcare, where doctors need explanations to justify a particular diagnosis or a course of treatment. AI techniques such as decision-tree induction provide built-in explanations but are generally less accurate. Thus, researchers must develop systems that are transparent, and intrinsically capable of explaining the reasons for their results to users.TrustBuild trust_8ad7a81a-954e-11e6-9ea9-559e742aff294.2To achieve trust, AI system designers need to create accurate, reliable systems with informative, user-friendly interfaces, while the operators must take the time for adequate training to understand system operation and limits of performance. Complex systems that are widely trusted by users, such as manual controls for vehicles, tend to be transparent (the system operates in a manner that is visible to the user), credible (the system’s outputs are accepted by the user), auditable (the system can be evaluated), reliable (the system acts as the user intended), and recoverable (the user can recover control when desired). A significant challenge to current and future AI systems remains the inconsistent quality of software production technology. As advances bring greater linkages between humans and AI systems, the challenge in the area of trust is to keep pace with changing and increasing capabilities, anticipate technological advances in adoption and long-term use, and establish governing principles and policies for the study of best practices for design, construction, and use, including proper operator training for safe operation.Verification & ValidationEnhance verification and validation_8ad7ae96-954e-11e6-9ea9-559e742aff294.3New methods are needed for verification and validation of AI systems. "Verification" establishes that a system meets formal specifications, while "validation" establishes that a system meets the user's operational needs. Safe AI systems may require new means of assessment (determining if the system is malfunctioning, perhaps when operating outside expected parameters), diagnosis (determining the causes for the malfunction), and repair (adjusting the system to address the malfunction). For systems operating autonomously over extended periods of time, system designers may not have considered every condition the system will encounter. Such systems may need to possess capabilities for self-assessment, self-diagnosis, and self-repair in order to be robust and reliable.AttacksSecure against attacks_8ad7b242-954e-11e6-9ea9-559e742aff294.4AI embedded in critical systems must be robust in order to handle accidents, but should also be secure to a wide range of intentional cyber attacks. Security engineering involves understanding the vulnerabilities of a system and the actions of actors who may be interested in attacking it. While cybersecurity R&D needs are addressed in greater detail in the NITRD Cybersecurity R&D Strategic Plan, some cybersecurity risks are specific to AI systems. For example, one key research area is "adversarial machine learning" that explores the degree to which AI systems can be compromised by "contaminating" training data, by modifying algorithms, or by making subtle changes to an object that prevent it from being correctly identified (e.g., prosthetics that spoof facial recognition systems). The implementation of AI in cybersecurity systems that require a high degree of autonomy is also an area for further study. One recent example of work in this area is DARPA's Cyber Grand Challenge that involved AI agents autonomously analyzing and countering cyber attacks.Values & SafetyAchieve long-term AI safety and value-alignment_8ad7b4a4-954e-11e6-9ea9-559e742aff294.5AI systems may eventually become capable of "recursive self-improvement," in which substantial software modifications are made by the software itself, rather than by human programmers. To ensure the safety of self-modifying systems, additional research is called for to develop: self-monitoring architectures that check systems for behavioral consistency with the original goals of human designers; confinement strategies for preventing the release of systems while they are being evaluated; value learning, in which the values, goals, or intentions of users can be inferred by a system; and value frameworks that are provably resistant to self-modification. Datasets & EnvironmentsDevelop shared public datasets and environments for AI training and testing. _d0b9f984-9547-11e6-bea8-c06b742aff29Strategy 5The depth, quality, and accuracy of training datasets and resources significantly affect AI performance. Researchers need to develop high quality datasets and environments and enable responsible access to high-quality datasets as well as to testing and training resources. -- The benefits of AI will continue to accrue, but only to the extent that training and testing resources for AI are developed and made available. The variety, depth, quality, and accuracy of training datasets and other resources significantly affects AI performance. Many different AI technologies require high-quality data for training and testing, as well as dynamic, interactive testbeds and simulation environments. More than just a technical question, this is a significant "public good" challenge, as progress would suffer if AI training and testing is limited to only a few entities that already hold valuable datasets and resources, yet we must simultaneously respect commercial and individual rights and interests in the data. Research is needed to develop high-quality datasets and environments for a wide variety of AI applications, and to enable responsible access to good datasets and testing and training resources. Additional open-source software libraries and toolkits are also needed to accelerate the advancement of AI R&D. The following subsections outline these key areas of importance.Interests & ApplicationsDevelop and make accessible a wide variety of datasets to meet the needs of a diverse spectrum of AI interests and applications_d0b9fb50-9547-11e6-bea8-c06b742aff295.1The integrity and availability of AI training and testing datasets is crucial to ensuring scientifically reliable results. The technical as well as the socio-technical infrastructure necessary to support reproducible research in the digital area has been recognized as an important challenge --and is essential to AI technologies as well. The lack of vetted and openly available datasets with identified provenance to enable reproducibility is a critical factor to confident advancement in AI. As in other data-intensive sciences, capturing data provenance is critical. Researchers must be able to reproduce results with the same as well as different datasets. Datasets must be representative of challenging real-world applications, and not just simplified versions. To make progress quickly, emphasis should be placed on making available already existing datasets held by government, those that can be developed with Federal funding, and, to the extent possible, those held by industry. The machine learning aspect of the AI challenge is often linked with "big data" analysis. Considering the wide variety of relevant datasets, it remains a growing challenge to have appropriate representation, access, and analysis of unstructured or semi-structured data. How can the data be represented -- in absolute as well as relative (context-dependent) terms? Current real-world databases can be highly susceptible to inconsistent, incomplete, and noisy data. Therefore, a number of data preprocessing techniques (e.g., data cleaning, integration, transformation, reduction, and representation) are important to establishing useful datasets for AI applications. How does the data preprocessing impact data quality, especially when additional analysis is performed? Encouraging the sharing of AI datasets -- especially for government-funded research -- would likely stimulate innovative AI approaches and solutions. However, technologies are needed to ensure safe sharing of data, since data owners take on risk when sharing their data with the research community. Dataset development and sharing must also follow applicable laws and regulations, and be carried out in an ethical manner. Risks can arise in various ways: inappropriate use of datasets, inaccurate or inappropriate disclosure, and limitations in data de-identification techniques to ensure privacy and confidentiality protections.Commercial & Public InterestsMake training and testing resources responsive to commercial and public interests_8ad7b8b4-954e-11e6-9ea9-559e742aff295.2With the continuing explosion of data, data sources, and information technology worldwide, both the number and size of datasets are increasing. The techniques and technologies to analyze data are not keeping up with the high volume of raw information sources. Data capture, curation, analysis, and visualization are all key research challenges, and the science needed to extract valuable knowledge from enormous amounts of data is lagging behind. While data repositories exist, they are often unable to deal with the scaling up of datasets, have limited data provenance information, and do not support semantically rich data searches. Dynamic, agile repositories are needed. One example of the kind of open/sharing infrastructure program that is needed to support the needs of AI research is the IMPACT program (Information Marketplace for Policy and Analysis of Cyber-risk & Trust) developed by the Department of Homeland Security (DHS). This program supports the global cyber security risk research effort by coordinating and developing real-world data and information sharing capabilities, including tools, models, and methodologies. IMPACT also supports empirical data sharing between the international cybersecurity R&D community, critical infrastructure providers, and their government supporters. AI R&D would benefit from comparable programs across all AI applications.Software Libraries & ToolkitsDevelop open-source software libraries and toolkits_8ad7bbac-954e-11e6-9ea9-559e742aff295.3The increased availability of open-source software libraries and toolkits provides access to cutting-edge AI technologies for any developer with an Internet connection. Resources such as the Weka toolkit, MALLET, and OpenNLP, among many others, have accelerated the development and application of AI. Development tools, including free or low-cost code repository and version control systems, as well as free or low-cost development languages (e.g., R, Octave, and Python) provide low barriers to using and extending these libraries. In addition, for those who may not want to integrate these libraries directly, any number of cloud-based machine learning services exist that can perform tasks such as image classification on demand through low-latency web protocols that require little or no programming for use. Finally, many of these web services also offer the use of specialized hardware, including GPU-based systems. It is reasonable to assume that specialized hardware for AI algorithms, including neuromorphic processors, will also become widely available through these services. Together, these resources provide an AI technology infrastructure that encourages marketplace innovation by allowing entrepreneurs to develop solutions that solve narrow domain problems without requiring expensive hardware or software, without requiring a high level of AI expertise, and permitting rapid scaling-up of systems on demand. For narrow AI domains, barriers to marketplace innovation are extremely low relative to many other technology areas. To help support a continued high level of innovation in this area, the U.S. government can boost efforts in the development, support, and use of open AI technologies. Particularly beneficial would be open resources that use standardized or open formats and open standards for representing semantic information, including domain ontologies when available. Government may also encourage greater adoption of open AI resources by accelerating the use of open AI technologies within the government itself, and thus help to maintain a low barrier to entry for innovators. Whenever possible, government should contribute algorithms and software to open source projects. Because government has specific concerns, such as a greater emphasis on data privacy and security, it may be necessary for the government to develop mechanisms to ease government adoption of AI systems. For example, it may be useful to create a task force that can perform a "horizon scan" across government agencies to find particular AI application areas within departments, and then determine specific concerns that would need to be addressed to permit adoption of such techniques by these agencies.Standards & BenchmarksMeasure and evaluate AI technologies through standards and benchmarks. _d0b9fd12-9547-11e6-bea8-c06b742aff29Strategy 6Essential to advancements in AI are standards, benchmarks, testbeds, and community engagement that guide and evaluate progress in AI. Additional research is needed to develop a broad spectrum of evaluative techniques. -- Standards, benchmarks, testbeds, and their adoption by the AI community are essential for guiding and promoting R&D of AI technologies. The following subsections outline areas where additional progress must be made. StandardsDevelop a broad spectrum of AI standards_d0b9ff1a-9547-11e6-bea8-c06b742aff296.1The development of standards must be hastened to keep pace with the rapidly evolving capabilities and expanding domains of AI applications. Standards provide requirements, specifications, guidelines, or characteristics that can be used consistently to ensure that AI technologies meet critical objectives for functionality and interoperability, and that they perform reliably and safely. Adoption of standards brings credibility to technology advancements and facilitates an expanded interoperable marketplace. One example of an AI-relevant standard that has been developed is P1872-2015 (Standard Ontologies for Robotics and Automation), developed by the Institute of Electrical and Electronics Engineers (IEEE). This standard provides a systematic way of representing knowledge and a common set of terms and definitions. These allow for unambiguous knowledge transfer among humans, robots, and other artificial systems, as well as provide a foundational basis for the application of AI technologies to robotics. Additional work in AI standards development is needed across all subdomains of AI. Standards are needed to address:Software EngineeringManage system complexity, sustainment, security, and monitor and control emergent behaviors_8ad7be36-954e-11e6-9ea9-559e742aff296.1.1PerformanceEnsure accuracy, reliability, robustness, accessibility, and scalability_8ad7c282-954e-11e6-9ea9-559e742aff296.1.2MetricsQuantify factors impacting performance and compliance to standards_8ad7c58e-954e-11e6-9ea9-559e742aff296.1.3SafetyEvaluate risk management and hazard analysis of systems, human computer interactions, control systems, and regulatory compliance_8ad7c822-954e-11e6-9ea9-559e742aff296.1.4UsabilityEnsure that interfaces and controls are effective, efficient, and intuitive_8ad7cca0-954e-11e6-9ea9-559e742aff296.1.5InteroperabilityDefine interchangeable components, data, and transaction models via standard and compatible interfaces_8ad7cfb6-954e-11e6-9ea9-559e742aff296.1.6SecurityAddress the confidentiality, integrity, and availability of information, as well as cybersecurity_8ad7d24a-954e-11e6-9ea9-559e742aff296.1.7PrivacyControl for the protection of information while being processed, when in transit, or being stored_8ad7d6d2-954e-11e6-9ea9-559e742aff296.1.8TraceabilityProvide a record of events (their implementation, testing, and completion), and for the curation of data_8ad7daba-954e-11e6-9ea9-559e742aff296.1.9DomainsDefine domain-specific standard lexicons and corresponding frameworks_8ad7dd58-954e-11e6-9ea9-559e742aff296.1.10BenchmarksEstablish AI technology benchmarks_8ad7e1d6-954e-11e6-9ea9-559e742aff296.2Benchmarks, made up of tests and evaluations, provide quantitative measures for developing standards and assessing compliance to standards. Benchmarks drive innovation by promoting advancements aimed at addressing strategically selected scenarios; they additionally provide objective data to track the evolution of AI science and technologies. To effectively evaluate AI technologies, relevant and effective testing methodologies and metrics must be developed and standardized. Standard testing methods will prescribe protocols and procedures for assessing, comparing, and managing the performance of AI technologies. Standard metrics are needed to define quantifiable measures in order to characterize AI technologies, including but not limited to: accuracy, complexity, trust and competency, risk and uncertainty; explainability; unintended bias; comparison to human performance; and economic impact. It is important to note that benchmarks are data driven. Strategy 5 discusses the importance of datasets for training and testing. As a successful example of AI-relevant benchmarks, the National Institute of Standards and Technology (NIST) has developed a comprehensive set of standard test methods and associated performance metrics to assess key capabilities of emergency response robots. The objective is to facilitate quantitative comparisons of different robot models by making use of statistically significant data on robot capabilities that was captured using the standard test methods. These comparisons can guide purchasing decisions and help developers to understand deployment capabilities. The resulting test methods are being standardized though the ASTM International Standards Committee on Homeland Security Applications for robotic operational equipment (referred to as standard E54.08.01). Versions of the test methods are used to challenge the research community through the RoboCup Rescue Robot League competitions, which emphasize autonomous capabilities. Another example is the IEEE Agile Robotics for Industrial Automation Competition (ARIAC), a joint effort between IEEE and NIST, which promotes robot agility by utilizing the latest advances in artificial intelligence and robot planning. A core focus of this competition is to test the agility of industrial robot systems, with the goal of enabling those on the shop floors to be more productive, more autonomous, and requiring less time from shop floor workers. While these efforts provide a strong foundation for driving AI benchmarking forward, they are limited by being domain-specific. Additional standards, testbeds, and benchmarks are needed across a broader range of domains to ensure that AI solutions are broadly applicable and widely adopted.TestbedsIncrease the availability of AI testbeds_8ad7e4ec-954e-11e6-9ea9-559e742aff296.3The importance of testbeds was stated in the Cyber Experimentation of the Future report: "Testbeds are essential so that researchers can use actual operational data to model and run experiments on realworld system[s] … and scenarios in good test environments." Having adequate testbeds is a need across all areas of AI. The government has massive amounts of mission-sensitive data unique to government, but much of this data cannot be distributed to the outside research community. Appropriate programs could be established for academic and industrial researchers to conduct research within secured and curated testbed environments established by specific agencies. AI models and experimental methods could be shared and validated by the research community by having access to these test environments, affording AI scientists, engineers, and students unique research opportunities not otherwise available.EngagementEngage the AI community in standards and benchmarks_8ad7e794-954e-11e6-9ea9-559e742aff296.4AI CommunityGovernment leadership and coordination is needed to drive standardization and encourage its widespread use in government, academia, and industry. The AI community -- made up of users, industry, academia, and government -- must be energized to participate in developing standards and benchmark programs. As each government agency engages the community in different ways based on their role and mission, community interactions can be leveraged through coordination in order to strengthen their impact. This coordination is needed to collectively gather user-driven requirements, anticipate developer-driven standards, and promote educational opportunities. User-driven requirements shape the objectives and design of challenge problems and enable technology evaluation. Having community benchmarks focuses R&D to define progress, close gaps, and drive innovative solutions for specific problems. These benchmarks must include methods for defining and assigning ground truth. The creation of benchmark simulation and analysis tools will also accelerate AI developments. The results of these benchmarks also help match the right technology to the user's need, forming objective criteria for standards compliance, qualified product lists, and potential source selection. Industry and academia are the primary sources for emerging AI technologies. Promoting and coordinating their participation in standards and benchmarking activities are critical. As solutions emerge, opportunities abound for anticipating developer- and user-driven standards through sharing common visions for technical architectures, developing reference implementations of emerging standards to show feasibility, and conducting pre-competitive testing to ensure high-quality and interoperable solutions, as well as to develop best practices for technology applications. One successful example of a high-impact, community-based, AI-relevant benchmark program is the Text Retrieval Conference (TREC), which was started by NIST in 1992 to provide the infrastructure necessary for large-scale evaluation of information retrieval methodologies. More than 250 groups have participated in TREC, including academic and commercial organizations both large and small. The standard, widely available, and carefully constructed set of data put forth by TREC has been credited with revitalizing research on information retrieval. A second example is the NIST periodic benchmark program in the area of machine vision applied to biometrics, particularly face recognition. This began with the Face Recognition Technology (FERET) evaluation in 1993, which provided a standard dataset of face photos designed to support face recognition algorithm development as well as an evaluation protocol. This effort has evolved over the years into the Face Recognition Vendor Test (FRVT), involving the distribution of datasets, hosting of challenge problems, and conducting of sequestered technology evaluations. This benchmark program has contributed greatly to the improvement of facial recognition technology. Both TREC and FRVT can serve as examples of effective AI-relevant community benchmarking activities, but similar efforts are needed in other areas of AI. It is important to note that developing and adopting standards, as well as participating in benchmark activities, comes with a cost. R&D organizations are incentivized when they see significant benefit. Updating acquisition processes across agencies to include specific requirements for AI standards in requests for proposals will encourage the community to further engage in standards development and adoption. Community-based benchmarks, such as TREC and FRVT, also lower barriers and strengthen incentives by providing types of training and testing data otherwise inaccessible, fostering healthy competition between technology developers to drive best-of-breed algorithms, and providing objective and comparative performance metrics for relevant source selections.AI R&D WorkforceUnderstand the national AI R&D workforce needs._d0ba00dc-9547-11e6-bea8-c06b742aff29Strategy 7Advances in AI will require a strong community of AI researchers. An improved understanding of current and future R&D workforce demands in AI is needed to help ensure that sufficient AI experts are available to address the strategic R&D areas outlined in this plan. -- Attaining the needed AI R&D advances outlined in this strategy will require a sufficient AI R&D workforce. Nations with the strongest presence in AI R&D will establish leading positions in the automation of the future. They will become the front-runners in competencies like algorithm creation and development; capability demonstration; and commercialization. Developing technical expertise will provide the basis for these advancements. While no official AI workforce data currently exist, numerous recent reports from the commercial and academic sectors are indicating an increased shortage of available experts in AI. AI experts are reportedly in short supply, with demand expected to continue to escalate. 66High tech companies are reportedly investing significant resources into recruiting faculty members and students with AI expertise. Universities and industries are reportedly in a battle to recruit and retain AI talent. Additional studies are needed to better understand the current and future national workforce needs for AI R&D. Data is needed to characterize the current state of the AI R&D workforce, including the needs of academia, government, and industry. Studies should explore the supply and demand forces in the AI workplace, to help predict future workforce needs. An understanding is needed of the projected AI R&D workforce pipeline. Considerations of educational pathways and potential retraining opportunities should be included. Diversity issues should also be explored, since studies have shown that a diverse information technology workforce can lead to improved outcomes. Once the current and future AI R&D workforce needs are better understood, then appropriate plans and actions can be considered to address any existing or anticipated workforce challenges._d0ba02a8-9547-11e6-bea8-c06b742aff29Implementation FrameworkDevelop an AI R&D implementation framework to identify S&T opportunities and support effective coordination of AI R&D investments, consistent with Strategies 1-6 of this plan._d0ba04c4-9547-11e6-bea8-c06b742aff29Recommendation 1Federal agencies should collaborate through NITRD to develop an R&D implementation framework that facilitates coordination and progress on the R&D challenges outlined in this plan. This will enable agencies to easily plan, coordinate, and collaborate in support of this strategic plan. The implementation framework should take into account the R&D priorities of each agency, based on their missions, capabilities, authorities, and budget. Based on the implementation framework, funding programs may need to be established for coordinated execution of the national research agenda for AI. To help implement this Strategic Plan, NITRD should consider forming an interagency working group focused on AI, in coordination with existing working groups._d0ba06ae-9547-11e6-bea8-c06b742aff29National LandscapeStudy the national landscape for creating and sustaining a healthy AI R&D workforce, consistent with Strategy 7 of this plan._d0ba087a-9547-11e6-bea8-c06b742aff29Recommendation 2A healthy and vibrant AI R&D workforce is important to addressing the R&D strategic challenges outlined in this report. While some reports have indicated a potential growing shortage of AI R&D experts, no official workforce data exists to characterize the current state of the AI R&D workforce, the projected workforce pipeline, and the supply and demand forces in the AI workforce. Given the role of the AI R&D workforce in addressing the strategic priorities identified in this plan, a better understanding is needed for attaining and/or maintaining a healthy AI R&D workforce. NITRD should study how best to characterize and define the current and future AI R&D workforce needs, developing additional studies or recommendations that can ensure a sufficient R&D workforce to address the AI needs of the Nation. As indicated by the outcome of the studies, appropriate Federal organizations should then take steps to ensure that a healthy national AI R&D workforce is created and maintained. _d0ba0a96-9547-11e6-bea8-c06b742aff292016-10-18https://obamawhitehouse.archives.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/national_ai_rd_strategic_plan.pdfOwenAmburOwen.Ambur@verizon.net