About MIRIThe field of AI has a reputation for overselling its progress. In the “AI winters” of the late 1970s and 1980s, researchers’ failures to make good on ambitious promises led to a collapse of funding and interest in AI. Although the field is now undergoing a renaissance, overconfidence is still a major fear; discussion of the possibility of human-equivalent general intelligence is still largely relegated to the science fiction shelf. ^^ At the same time, researchers largely agree that AI is likely to begin outperforming humans on most cognitive tasks in this century. Given how disruptive domain-general AI could be, we think it is prudent to begin a conversation about this now, and to investigate whether there are limited areas in which we can predict and shape this technology’s societal impact.Researchers at MIRI tend to be relatively agnostic about how the state of the art in AI will change over the coming decades, and how many years off smarter-than human AI systems are. However, we think some qualitative predictions are possible: ^^ * As perception, inference, and planning algorithms improve, AI systems will be trusted with increasingly complex and long-term decision-making. Small errors will then have larger consequences. ^ * Realistic goals and environments for general reasoning systems will be too complex for programmers to directly specify. AI systems will instead need to inductively learn correct goals and environmental models. ^ * Systems that end up with poor models of their environment can do significant harm. However, poor models limit how well a planning system can control its environment, which limits the expected harm. ^ * There are fewer obvious constraints on the harm a system with poorly specified goals might do. In particular, an autonomous system that learns about human goals, but is not correctly designed to align its own goals to its best model of human goals, could cause catastrophic harm in the absence of adequate checks. ^ * AI systems’ goals or world-models may be brittle, exhibiting exceptionally good behavior until some seemingly irrelevant environmental variable changes. This is again a larger concern for incorrect goals than for incorrect belief and inference, because incorrect goals don’t limit the capability of an otherwise high-intelligence system.Machine Intelligence Research InstituteMIRI_d769f7a8-e0c1-11ed-a8fb-5c122483ea00The Machine Intelligence Research Institute is a research nonprofit studying the mathematical underpinnings of intelligent behavior. MIRI TeamMIRI LeadersNate SoaresEXECUTIVE DIRECTOR ~ Nate Soares heads MIRI’s research program. He first joined MIRI in 2014 as a research fellow, quickly earning a strong reputation for his strategic insight and high productivity. Nate is the primary author of most of MIRI’s agent foundations technical agenda, including the overview document “Agent Foundations for Aligning Machine Intelligence with Human Interests” (2014) and “Corrigibility” (2015). Prior to MIRI, Nate worked as a software engineer at Google.Malo BourgonCHIEF OPERATING OFFICER ~ Malo Bourgon (email) oversees MIRI’s day-to-day operations and program activities. Before becoming COO, Malo worked for MIRI as a program management analyst, helping implement many of MIRI’s current systems, processes, and program activities. He also co-chairs the IEEE committee on the Safety and Beneficence of Artificial General Intelligence and Artificial Superintelligence. Malo joined MIRI in 2012 shortly after completing a master’s degree in engineering at the University of Guelph.MIRI Research StaffThe following is a non-exhaustive list of full-time research staff at MIRI.Eliezer YudkowskyEliezer Yudkowsky is a decision theorist who is widely cited for his writings on the long-term future of artificial intelligence. His views on the social and philosophical significance of AI have had a major impact on ongoing debates in the field, and as MIRI’s senior research fellow, his work in mathematical logic has heavily shaped MIRI’s research agenda. He is the author of the Cambridge Handbook of Artificial Intelligence chapter “The Ethics of Artificial Intelligence” with Nick Bostrom (2014), and has written a number of popular introductions to the science of human rationality.Benya FallensteinBenya Fallenstein works on basic theoretical questions raised by the challenge of aligning advanced AI systems with human goals. These include decision- and game-theoretic problems that arise when artificial agents reason about future versions of themselves or about other, similarly powerful agents in their environment. Since joining the research team in 2014, she has spent time developing models of logical uncertainty (uncertainty about which mathematical statements are true), self-reference in higher-order theorem-proving systems, and the specification of safe AI goals. Benya holds a bachelor’s in mathematics from the University of Vienna.Scott GarrabrantScott Garrabrant earned his PhD in mathematics from UCLA studying applications of theoretical computer science to enumerative combinatorics. His main research area is in logical uncertainty, and he is the primary author of “Logical Induction” (2016), a highly general method for assigning probabilities to logical sentences. He is also interested in other aspects of logical uncertainty and counterfactuals.Sam EisenstatSam Eisenstat works on questions relating to the foundations of reasoning and agency. He studied pure mathematics at the University of Waterloo, where he carried out research in mathematical logic. Before joining MIRI, he worked on automatic construction of deep learning models at Google. He currently works on logical uncertainty, and in particular is exploring analogies between current theories of logical uncertainty and Bayesian reasoning. He has also done work on decision theory and counterfactuals.Tsvi Benson-TilsenTsvi Benson-Tilsen works on the foundations of rational agency, including logical uncertainty, logical counterfactuals, and reflectively stable decision making, as well as other questions of AI alignment. Before joining MIRI as a full-time researcher, he collaborated on “Logical Induction”. Tsvi holds a BSc in Mathematics with honors from the University of Chicago, and is on leave from the UC Berkeley Group in Logic and the Methodology of Science PhD program. Tsvi joined MIRI in June 2017.Abram DemskiAbram Demski is currently completing a PhD in Computer Science at the University of Southern California. His research to date has focused on cognitive architectures and artificial general intelligence. He is interested in filling in the gaps that exist in formal theories of rationality, especially those concerned with what humans are doing when reasoning semi-formally about mathematics.Jesse LiptrapJesse Liptrap joined MIRI in 2017 after spending four years as a software engineer at Google, working on the Knowledge Graph. Previously he worked as a bioinformatician at UC Berkeley. He holds a BS in math from Caltech and a PhD in math from UC Santa Barbara, where he studied category-theoretic underpinnings of topological quantum computing.Katja GraceRESEARCHER, AI IMPACTS ~ Katja Grace contributes to AI Impacts, an independent research project focused on social and historical questions related to artificial intelligence outcomes. Her analyses include “Algorithmic Progress in Six Domains” (2013). She writes the blog Meteuphoric, and is sometimes a PhD student in logic, computation, and methodology at Carnegie Mellon University.MIRI Research AssociatesStuart ArmstrongPhD Mathematics, Oxford | Cofounder, Aligned AIRamana KumarPhD Computer Science, Cambridge | Research Scientist, Google DeepMindVanessa KosoyBSc Mathematics, Tel Aviv University | Algorithm Engineer, Epicycle TechnologiesMIRI Spotlighted Research SupportRob BensingerRESEARCH COMMUNICATIONS ~ Rob Bensinger helps communicate background information and updates about MIRI’s research activities and strategy. His research interests include value theory and the relationship between philosophy and psychology. Rob joined MIRI in 2013Colm Ó RiainGROWTH ~ Colm Ó Riain coordinates philanthropic strategy and hiring to support MIRI’s growth plans. After 15 years working in the video game industry at companies including Electronic Arts and Activision, he moved into philanthropy work at Zynga.org and Harmony Project before joining MIRI in 2016. He has a master’s degree in AI from the University of Rochester and a joint honours bachelor’s in Mathematics and Computer Science. Colm is also a professional violinist and composer.Alex VermeerPROCESS AND PROJECTS ~ Alex Vermeer improves the processes and systems within and surrounding MIRI’s research team and research programs. This includes increasing the quality and quantity of workshops and similar programs, implementing best practices within the research team, coordinating the technical publication and researcher recruiting pipelines, and other research support projects. Alex holds an engineering degree from the University of Guelph, and joined MIRI in 2012.MIRI Spotlighted AdvisorsNick BostromProfessor, University of Oxford | Director, Future of Humanity InstituteLuke MuehlhauserResearch Analyst, Open PhilanthropyVictoria KrakovnaCofounder, Future of Life Institute | Research Scientist, Google DeepMindJaan TallinnCofounder, Centre for the Study of Existential Risk | Cofounder, Future of Life Institute | Cofounder, SkypeMIRI BoardBlake BorgesonDIRECTOR | Cofounder, Recursion PharmaceuticalsAnna SalamonDIRECTOR | President, Center for Applied RationalityEliezer YudkowskyDIRECTOR | Senior Research Fellow, MIRIEdwin EvansCHAIR | Cofounder, Linc GlobalNate SoaresDIRECTOR | Executive Director, MIRIMIRI Financial ContributorsTop Contributors ~ Since its founding in 2000, the Machine Intelligence Research Institute has had 4,356 distinct donors. Our largest contributors, each of whom has given at least $500,000, are:Anonymous Ethereum Cryptocurrency InvestorAn anonymous Ethereum cryptocurrency investor in 2018 and 2021.Open Philanthropya joint initiative with the philanthropic foundation Good Ventures.Vitalik Buterinthe inventor and co-founder of Ethereum.The Thiel FoundationThe Thiel Foundation, a private foundation funded by Paypal co-founder and venture capitalist Peter Thiel.Anonymous Ethereum Cryptocurrency InvestorA different anonymous Ethereum cryptocurrency investor in 2017.Berkeley Existential Risk InitiativeJaan TallinnSkype and Kazaa developer and co-founder of the existential risk institutes FLI and CSER.Jed McCalebdeveloper of the Ripple and Stellar currency exchange systems.Loren Merrittx264 video codec library developer.Edwin EvansCofounder of Linc Global.MIRI Top ContributorsFor a list of donors who have given $5,000 or more and wanted to be acknowledged, see our Top Contributors page. https://intelligence.org/topcontributors/General-purpose AI systems are safer and more reliable when they are developed_d769f92e-e0c1-11ed-a8fb-5c122483ea00To develop formal tools for the clean design and analysis of general-purpose AI systems_d769fa28-e0c1-11ed-a8fb-5c122483ea00Artificial IntelligenceRobustnessSafetyResearchLongtermismDecision MakingImprove decision-making systems_d769fb72-e0c1-11ed-a8fb-5c122483ea00Stuart RussellStuart Russell, a MIRI research advisor and co-author of the leading textbook on artificial intelligence, argues in “The Long-Term Future of Artificial Intelligence” that we should integrate questions of robustness and safety into mainstream capabilities research:Our goal as a field is to make better decision-making systems. And that is the problem. […If] you’re going to build a superintelligent machine, you have to give it something that you want it to do. The danger is that you give it something that isn’t actually what you really want — because you’re not very good at expressing what you really want, or even knowing what you really want — until it’s too late and you see that you don’t like it. ^^ If you think about it just in terms of an optimization problem: The machine is solving an optimization problem for you, and you leave out some of the variables that you actually care about. Well, it’s in the nature of optimization problems that if the system gets to manipulate some variables that don’t form part of the objective function — so it’s free to play with those as much as it wants — often, in order to optimize the ones that it is supposed to optimize, it will set the other ones to extreme values.Robustness & SafetyIntegrate questions of AI robustness and safety into mainstream capabilities research_d769fc4e-e0c1-11ed-a8fb-5c122483ea001Strategic AlignmentDesign AI systems so that their actions are well-aligned with what human beings want_d769fd34-e0c1-11ed-a8fb-5c122483ea002My proposal is that we should stop doing AI in its simple definition of just improving the decision-making capabilities of systems. […] With civil engineering, we don’t call it “building bridges that don’t fall down” -- we just call it "building bridges." Of course we don’t want them to fall down. And we should think the same way about AI: of course AI systems should be designed so that their actions are well-aligned with what human beings want. But it’s a difficult unsolved problem that hasn’t been part of the research agenda up to now. ^^ We want to change the field so that it feels like civil engineering or like nuclear fusion. [… We] created a hydrogen bomb explosion — unlimited amounts of energy, more than we could possibly use. But it wasn’t in a socially beneficial form. And now it’s just what fusion researchers do — containment is what fusion research is. That’s the problem that they work on.LongtermismJump-start a paradigm of AI research that is conscious of the field’s long-term impact_d769fe24-e0c1-11ed-a8fb-5c122483ea003In line with Russell’s talk, MIRI’s work is aimed at helping jump-start a paradigm of AI research that is conscious of the field’s long-term impact. Our methodology is to break down the alignment problem into simpler and more precisely stated subproblems, develop basic mathematical theory for understanding these problems, and then make use of our newfound understanding in engineering applications.SubproblemsBreak down the alignment problem into simpler and more precisely stated subproblems_d769ff0a-e0c1-11ed-a8fb-5c122483ea003.1Mathematical TheoryDevelop basic mathematical theory for understanding these problems_d769fff0-e0c1-11ed-a8fb-5c122483ea003.2Engineering ApplicationsMake use of our newfound understanding in engineering applications_d76a00cc-e0c1-11ed-a8fb-5c122483ea003.32023-04-22https://intelligence.org/about/OwenAmburOwen.Ambur@verizon.net