Competency-Aware Machine Learning (CAML)The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) [solicited] innovative research proposals in the area of competency-awareness machine learning, whereby an autonomous system can self-assess its task competency and strategy and express both in a human-understandable form. This competency-awareness capability contributes to the goal of transforming autonomous systems from tools into trusted, collaborative partners. The resulting competency-aware machine learning systems will enable machines to control their behaviors to match user expectations and allow human operators to quickly and accurately gain insight into a system’s competence in complex, time-critical, dynamic environments. The Competency-Aware Machine Learning (CAML) program will, in this way, improve the efficiency and effectiveness of human-machine teaming. Proposed research should investigate innovative approaches that enable revolutionary advances in science. DSO will exclude proposals that propose evolutionary improvement to the existing state of practice.CAML will significantly improve human-machine teaming capabilities and the task synergies expected of autonomous systems. By creating a fundamentally new machine learning approach, CAML will facilitate mission planning by choosing the appropriate available asset based on task requirements, determining the level of autonomy to be granted based on an asset-proposed strategy, and controlling behaviors to adapt for operating conditions.Defense Advanced Research Projects AgencyDARPA_46ad30a4-ad5d-11df-9c96-10167a64ea2aDr. Jiangying ZhouTechnical POC - Program Manager, DARPA/DSOMilitary & Aerospace ElectronicsArtificial intelligence (AI) experts at four U.S. companies are helping military researchers determine if autonomous machines are self-aware of their own competencies and limitations to carry out assigned tasks. Officials of the U.S. Defense Advanced Research Projects Agency (DARPA) have announced four contracts cumulatively worth $20.9 million for the Competency-Aware Machine Learning (CAML) project. Companies involved are SRI International in Menlo Park, Calif.; Raytheon BBN Technologies Corp. in Cambridge, Mass.; Teledyne Scientific & Imaging LLC in Camarillo, Calif.; and the BAE Systems Electronic Systems segment in Burlington, Mass... CAML is a four-year program divided into a three-year research first phase, and a one-year technology-demonstration second phase. https://www.militaryaerospace.com/computers/article/14068536/self-aware-artificial-intelligence-ai-machine-learningSRI InternationalSRI International won a $4.7 million CAML contract on 25 Sept. 2019Raytheon BBN Technologies Corp.Raytheon BBN won a $6 million CAML contract on 27 Sept. 2019Teledyne Scientific & Imaging LLCTeledyne Scientific & Imaging won a $5.4 million CAML contract on 10 Oct. 2019BAE Systems Electronic SystemsBAE Systems won a $4.9 million CAML contract on 10 Oct. 2019The efficiency and effectiveness of human-machine teaming is improved._1e275bf2-896f-11ea-8e3b-09991c83ea00To enable autonomous systems to self-assess their task competency and strategy and express both in a human-understandable form._1e275db4-896f-11ea-8e3b-09991c83ea00PartnershipIn order to transform machine learning systems from tools into partners, users need to trust their machine counterparts.TrustOne component to building trust is understanding machine competence (i.e., an accurate insight into the machine’s skills, experience, and reliability).EmergenceState-of-the-art machine learning systems operate in a complex state space, and continuously develop emergent behaviors based on their internal structures and learning experiences.FlexibilityWhile such systems can perform well when their behaviors are applied in contexts that similarly match their structures and learning experiences, they are unable to communicate their task strategies, the completeness of their training relative to a given task, the factors that may influence their actions, or their likelihood to succeed under specific conditions. CommunicationVerificationTherefore, current systems require users to rely on traditional verification and validation (V&V) methodologies to predict a machine system’s competency. The complexity of learning systems and operating environments has made such V&V increasingly unrealistic for end users.ValidationEconomy of ForceThis leads to a poor “economy of force” as human operators must mediate the system’s actions.Decision MakingThis deficiency is especially significant in the Department of Defense (DoD) domain where machine systems often deal with high-stake decisions, and must cope with highly dynamic, fast-changing conditions. DynamicsSelf AwarenessCAML addresses this challenge by enabling learning systems that are self-aware of competency, have knowledge of learned abilities and the conditions under which those abilities were learned, and have knowledge of resultant task strategies and the situations for which those strategies are applicable.Situational AwarnessLearningKnowledgeStrategyMachine LearningDevelop a competency-aware machine learning framework._1e275e9a-896f-11ea-8e3b-09991c83ea00The goal of CAML is to develop a competency-aware machine learning framework to support transitioning machine learning systems from tools into trusted partners. Achieving this goal requires the development of new elements in machine learning, including memory mechanisms, knowledge abstraction and representation, and behavior self-modeling.Self-Knowledge of ExperiencesDevelop mechanisms for learning systems to discover conditions encountered during operation, and maintain a memory of experiences._1e275f76-896f-11ea-8e3b-09991c83ea00TA1Mechanisms developed in TA1 should support modeling of user-defined experience elements as well as those that are self-discovered. The former are specified prior to learning. The latter includes meta-knowledge that is emergent from the discovery of the dependencies of task behaviors (see TA2). Proposers should discuss how their model would both incorporate the predefined experience elements and interact with the mechanisms developed in TA2 to dynamically incorporate the emergent meta-knowledge.Task StrategiesEnable machine learning systems to analyze their task behaviors, summarize them into generalized patterns (task strategies), and identify dependencies that control its task behavior._1e276070-896f-11ea-8e3b-09991c83ea00TA2TA2 will develop mechanisms to enable a machine learning system to analyze its task behaviors, summarize them into generalized patterns (i.e. task strategies), and identify the meta-knowledge elements that control task behavior. The objective is to establish descriptions of conditional task behaviors that provide concise insight into learning system behaviors and their internal/external dependencies. For example, proposers might use symbolic rules to express a learned task behavior and corresponding dependencies. In the case of a self-driving car, an example of such a rule could be “IF dirt road, THEN drive slowly.” Competency StatementsEstablish a competency-aware learning framework, communicated with machine-derived, human-understandable, competency statements._1e276174-896f-11ea-8e3b-09991c83ea00TA3TA3 will develop new machine learning systems that can assess their own competency for a given task and express that competency in a human-understandable form. TA3 seeks to achieve these goals by integrating machine analysis of its experiences (TA1) and task strategies (TA2) to determine competency, and providing machine-derived, human-understandable, descriptions of its task strategies and expected performance via competency statements. Capability DemonstrationsDemonstrate competency-aware machine learning systems on DoD platforms._1e27625a-896f-11ea-8e3b-09991c83ea00TA4TA4 in Phase II will focus on demonstrations of the developed competency-aware machine learning on DoD platforms. The objective is to evaluate and validate the utility of the competency-aware learning system in operations relevant to DoD missions2019-02-192020-04-28https://beta.sam.gov/api/prod/opps/v3/opportunities/resources/files/4594947783eae4193543bd13891d52ee/download?api_key=null&status=archived&token=OwenAmburOwen.Ambur@verizon.net