AI: Expectations and RealitiesHard problems in AI -- Machine learning: fragility, opacity, and dynamism... • How do we engineer systems to safely deliver AI to the mission? • How do we harmonize domain models with AI techniques? • How can humans best partner with AI-enhanced systems? • What are successful models for continuous delivery, continuous integration and continuous verification for AI? • What is next generation AI beyond symbolic and machine learning (waves 1 and 2 have shown their limits)?Information Innovation OfficeI2O_5ffb1ace-7235-11ea-87b3-9a8e0183ea00* Advantage in cyber operations * Artificial intelligence to the mission * Resilient, adaptable, and secure systems * Confidence in the information domainDARPADARPA Achievements: * MATERIAL SCIENCE *GPS * NIGHT VISION * PREDATOR AUTONOMOUS VEHICLE * NEUROPROSTHETIC LIMBS * STEALTH FIGHTER * ADVANCED SEMICONDUCTORS * SATURN V * AUTONOMOUS VEHICLES * PERSONALIZED ASSISTANT THAT LEARNSDr. William ScherlisDirector, Information Innovation OfficeI2O ProgramsCommunicating with Computers (CwC) ProgramComputers and Humans Exploring Software Security (CHESS) ProgramExplainable AI (XAI) ProgramLearning with Less Labeling (LwLL) ProgramMachine Common Sense (MCS) ProgramSynergistic Discovery and Design (SD2) Program3G AI_5ffb1c7c-7235-11ea-87b3-9a8e0183ea00To solve hard problems in AI_5ffb1d44-7235-11ea-87b3-9a8e0183ea00ResilienceResilient, adaptable, and secure systemsAdaptabilityOpennessOpen – Hardware/software decouplingProgrammabilityProgrammable – Configure to the missionSecuritySecure – Trust and securityAI CoreDevelop core AI_5ffb1e52-7235-11ea-87b3-9a8e0183ea001Machine Common Sense (MCS) ProgramContext: Specialized cognitive building blocks: perception, reasoning, action. Approach: • Hybrids methods • Machine learning + game theory + optimization • Machine learning + explicit reasoning • Infrastructure: Computing and data handling • Looking ahead: Self adaptation – learning to learn ^ Frame specialized AI using common sense reasoning.Situational UnderstandingEnable AI applications to understand new situations_5ffb1efc-7235-11ea-87b3-9a8e0183ea001.1_00ef6828-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_1[To be determined]ReasonablenessEnable AI applications to monitor the reasonableness of their actions_5ffb1fb0-7235-11ea-87b3-9a8e0183ea001.2_00ef6bac-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_2[To be determined]Learning TransferEnable AI applications to transfer learning to new domains_5ffb21c2-7235-11ea-87b3-9a8e0183ea001.3_00ef6de6-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_3[To be determined]CommunicationEnable AI applications to communicate more effectively with people_5ffb228a-7235-11ea-87b3-9a8e0183ea001.4People_00ef71d8-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_4[To be determined]ApplicationsAdvance mission applications of AI_5ffb233e-7235-11ea-87b3-9a8e0183ea002Knowledge-directed AI Reasoning Over Schemas (KAIROS) ProgramContext: • Emerging AI-enabled mission concepts • Adversaries are nimble and capable • Human-AI partnering remains difficult • Talent pool is a challenge ^ Approach: • Close partnering of operators and engineers • Start with advisory AI ReasoningCreate schema-based artificial intelligence capability to enable contextual and temporal reasoning about complex real-world events_5ffb23f2-7235-11ea-87b3-9a8e0183ea002.1Text | Speech | Images | Video > Media Analysis with Temporal Annotation > Schema Application & Temporal Reasoning > Temporal Knowledge Base > Predictive Analysis > User Interaction2.1.1aText_00ef75d4-72ad-11ea-9d53-b7df97babdf6_00ef7b88-72ad-11ea-9d53-b7df97babdf6_00ef816e-72ad-11ea-9d53-b7df97babdf6Speech_00ef7d72-72ad-11ea-9d53-b7df97babdf6_00ef8a1a-72ad-11ea-9d53-b7df97babdf6Images_00ef7ec6-72ad-11ea-9d53-b7df97babdf6_00ef9582-72ad-11ea-9d53-b7df97babdf6Video_00ef801a-72ad-11ea-9d53-b7df97babdf6_00ef9e10-72ad-11ea-9d53-b7df97babdf6Feeds into Analysis & AnnotationCapture text as input[To be determined]2.1.1bSpeech_00ef816e-72ad-11ea-9d53-b7df97babdf6_00ef82cc-72ad-11ea-9d53-b7df97babdf6_00ef75d4-72ad-11ea-9d53-b7df97babdf6Text_00ef8524-72ad-11ea-9d53-b7df97babdf6_00ef8a1a-72ad-11ea-9d53-b7df97babdf6Images_00ef8678-72ad-11ea-9d53-b7df97babdf6_00ef9582-72ad-11ea-9d53-b7df97babdf6Video_00ef87e0-72ad-11ea-9d53-b7df97babdf6_00ef9e10-72ad-11ea-9d53-b7df97babdf6Feeds into Analysis & AnnotationCapture speech as input[To be determined]2.1.1cImages_00ef8a1a-72ad-11ea-9d53-b7df97babdf6_00ef8f2e-72ad-11ea-9d53-b7df97babdf6_00ef75d4-72ad-11ea-9d53-b7df97babdf6Text_00ef908c-72ad-11ea-9d53-b7df97babdf6_00ef816e-72ad-11ea-9d53-b7df97babdf6Speech_00ef91ea-72ad-11ea-9d53-b7df97babdf6_00ef9582-72ad-11ea-9d53-b7df97babdf6Video_00ef941a-72ad-11ea-9d53-b7df97babdf6_00ef9e10-72ad-11ea-9d53-b7df97babdf6Feeds into Analysis & AnnotationCapture images as input[To be determined]2.1.1dVideo_00ef9582-72ad-11ea-9d53-b7df97babdf6_00ef96ea-72ad-11ea-9d53-b7df97babdf6_00ef75d4-72ad-11ea-9d53-b7df97babdf6Text_00ef9852-72ad-11ea-9d53-b7df97babdf6_00ef816e-72ad-11ea-9d53-b7df97babdf6Speech_00ef99ba-72ad-11ea-9d53-b7df97babdf6_00ef8a1a-72ad-11ea-9d53-b7df97babdf6Images_00ef9c9e-72ad-11ea-9d53-b7df97babdf6_00ef9e10-72ad-11ea-9d53-b7df97babdf6Feeds into Analysis & AnnotationCapture video as input[To be determined]2.1.2Analysis & Annotation_00ef9e10-72ad-11ea-9d53-b7df97babdf6_00ef9fe6-72ad-11ea-9d53-b7df97babdf6_00efa158-72ad-11ea-9d53-b7df97babdf6Feeds into Applications & ReasoningMedia Analysis with Temporal Annotation[To be determined]2.1.3Applications & Reasoning_00efa158-72ad-11ea-9d53-b7df97babdf6_00efa2d4-72ad-11ea-9d53-b7df97babdf6_00efa446-72ad-11ea-9d53-b7df97babdf6Feeds into Knowledge BaseSchema Application & Temporal Reasoning[To be determined]2.1.5Knowledge Base_00efa446-72ad-11ea-9d53-b7df97babdf6_00efa5c2-72ad-11ea-9d53-b7df97babdf6_00efa8ba-72ad-11ea-9d53-b7df97babdf6Supports Prediction_00efa73e-72ad-11ea-9d53-b7df97babdf6_f4a20370-7296-11ea-acdd-5f008fbabdf6Supports InteractionTemporal Knowledge Base[To be determined]2.1.6Prediction_00efa8ba-72ad-11ea-9d53-b7df97babdf6_00efaa54-72ad-11ea-9d53-b7df97babdf6_00efabda-72ad-11ea-9d53-b7df97babdf6Facilitates InteractionPredictive Analysis[To be determined]2.1.7Interaction_00efabda-72ad-11ea-9d53-b7df97babdf6_00efad6a-72ad-11ea-9d53-b7df97babdf6_00efa446-72ad-11ea-9d53-b7df97babdf6Feeds Back into the Knowledge Base_00efaefa-72ad-11ea-9d53-b7df97babdf6_00efb152-72ad-11ea-9d53-b7df97babdf6Feeds into the SchemasUser Interaction[To be determined]2.1.8Schemas_00efb152-72ad-11ea-9d53-b7df97babdf6_00efb2ec-72ad-11ea-9d53-b7df97babdf6_00efa8ba-72ad-11ea-9d53-b7df97babdf6Support Prediction_00efb47c-72ad-11ea-9d53-b7df97babdf6_00efa158-72ad-11ea-9d53-b7df97babdf6Support Applications & ReasoningDomain-Specific Top-Level Schemas[To be determined]Systems EngineeringEngineer systems with embedded AI_5ffb24b0-7235-11ea-87b3-9a8e0183ea003Explainable AI (XAI) ProgramContext: Software and systems engineering are made more challenging with AI. Approach: • Adapt key aspects of the engineering process • Integration frameworks, planning, and design • Process, tooling, and measurement • Assurance and evidence • Data, systems infrastructure, and configurations Understanding, Trust & ManagementEnable human users to understand, trust, and effectively manage the emerging generation of AI partners_5ffb2564-7235-11ea-87b3-9a8e0183ea003.1HumansAI PartnersExplain second wave AI_00efb616-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_5[To be determined]Causal ModelsLearn more structured, interpretable, causal models_5ffb2622-7235-11ea-87b3-9a8e0183ea003.1.1_00efb7ba-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_6[To be determined]FeaturesLearn more explainable features_5ffb26e0-7235-11ea-87b3-9a8e0183ea003.1.2_00efb954-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_7[To be determined]Black-Box ModelsInfer an explainable model from any model as a black-box_5ffb279e-7235-11ea-87b3-9a8e0183ea003.1.3_00efbbca-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_8[To be determined]Evaluation & AcceptanceDevelop continuous evaluation and acceptance_5ffb2870-7235-11ea-87b3-9a8e0183ea004Guaranteeing AI Robustness against Deception (GARD) ProgramContext: • Machine learning fragility, opacity, and dynamism • Adversaries empowered in new ways, including attacking conventional systems • Assurance influences all aspects of engineering and design, from the outset ^ Approach: • Integrate assurance planning • Manage evidence to support confident accreditation decisionsDeceptionEnable machine learning systems to be robust against adversary deception_5ffb292e-7235-11ea-87b3-9a8e0183ea004.1Design robust and resilient AI models_00efbd78-72ad-11ea-9d53-b7df97babdf6PLACEHOLDER_9[To be determined]_00efbeea-72ad-11ea-9d53-b7df97babdf62020-03-30https://download.1105media.com/Custom/Workshops/2020/AI/Bill_Scherlis.pdf_00efc070-72ad-11ea-9d53-b7df97babdf6OwenAmburOwen.Ambur@verizon.net