M-12-14 -- MEMORANDUM TO THE HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES: -- Use of Evidence and Evaluation in the 2014 BudgetSince taking office, the President has emphasized the need to use evidence and rigorous evaluation in budget, management, and policy decisions to make government work effectively. This need has only grown in the current fiscal environment. Where evidence is strong, we should act on it. Where evidence is suggestive, we should consider it. Where evidence is weak, we should build the knowledge to support better decisions in the future.Agencies should demonstrate the use of evidence throughout their Fiscal Year (FY) 2014 budget submissions. Budget submissions also should include a separate section on agencies' most innovative uses of evidence and evaluation, addressing some or all of the issues below. Many potential strategies have little immediate cost, and the Budget is more likely to fund requests that demonstrate a commitment to developing and using evidence. The Budget also will allocate limited resources for initiatives to expand the use of evidence, including but not limited to approaches outlined below. Agencies may include these initiatives in their submission at the guidance level or with proposed addbacks.Office of Management & BudgetOMB_4b105fb0-7e6f-11e1-9e13-fa3bfa1999f6Jeffrey D. ZientActing DirectorU.S; Federal AgenciesChief Evaluation OfficersThese goals can be accomplished by different kinds of leaders, ranging from a chief evaluation officer who reports to the Secretary or Deputy Secretary to the head of an independent institute in the agency. An existing official could play the role, or a forceful new position could replace several less empowered ones. OMB invites agencies to propose in their budget submissions ways to strengthen the agency's evaluation capacity, within tight resource constraints.Departmental SecretariesDepartmental Deputy SecretariesCouncil of Economic AdvisersOMB and the Council of Economic Advisers will organize a series of topical discussions with senior policy officials and research experts in the agencies. The meeting agendas will focus on administrative and policy levers for driving an increasing share of Federal investments into evidence-based practices. We will plan summer meetings in order to help inform agencies' evaluation plans and budget submissions, and will also have follow-up meetings in the fall.Interagency Evaluation Working GroupOMB will reinvigorate the interagency evaluation working group established in 2010 with a series of meetings focused on issues commonly affecting evaluators, such as procurement rules, the Paperwork Reduction Act, and the integration of evidence in agencies' decision-making process.Performance Improvement CouncilThe Performance Improvement Council will convene research, performance management, and program officials to develop ways to improve performance measures, validate their correlation with outcome data from program impact evaluations, and use data analytics to support more cost-effective decision-making.Office of Science and Technology PolicyThe Office of Science and Technology Policy has created a "community of practice" for agency personnel involved in designing and managing incentive prizes and has organized a Science of Science Policy working group that is developing tools aimed at establishing a more scientific, empirical evidence basis for science and technology policymaking.Science of Science Policy Working GroupDan RosenbaumOMB Contact_32c99f54-0d0b-11e2-87e2-baf70ac9daa5_32c9a2ce-0d0b-11e2-87e2-baf70ac9daa5EvaluationsPropose new evaluations._32c9a3c8-0d0b-11e2-87e2-baf70ac9daa51Proposing new evaluations. As in 2011 and 2012, OMB invites agencies to propose new evaluations. Areas of potential focus may include the following:Low-Cost Evaluations[Conduct] low-cost evaluations using administrative data or new technology._32c9a486-0d0b-11e2-87e2-baf70ac9daa51.1Coalition for Evidence-based PolicyLow-cost evaluations using administrative data or new technology: As explained in the Coalition for Evidence-based Policy's recent brief: agencies can often use administrative data (such as data on wages, employment, emergency room visits or school attendance) to conduct rigorous evaluations, including evaluations that rely on random assignment, at low cost. Similarly, the private sector has used new software and online tools to dramatically reduce the time and cost of experimentation. Agencies should consider whether they can use such data or technology to support rigorous evaluations of their existing programs or new initiatives.Waivers & Performance Partnerships[Conduct] evaluations linked to waivers and performance partnerships_32c9a544-0d0b-11e2-87e2-baf70ac9daa51.2Distressed CommunitiesDisconnected YouthEvaluations linked to waivers and performance partnerships: One of the best ways to learn about a program is to test variations and subject them to evaluation, using some element of random assignment or a scientifically controlled design. OMB invites agencies to explain how they will use existing waiver authorities to evaluate different approaches to improving outcomes. Agencies should also consider seeking authority from Congress, through the FY 2014 budget process, to allow new waivers linked to evaluation or to establish cross-agency "performance partnerships" that enable blending of multiple funding streams to test better ways to align services and improve outcomes. Several agencies are seeking such authority in 2013 for initiatives supporting distressed communities and disconnected youth.Existing ProgramsExpand evaluation efforts within existing programs_32c9a602-0d0b-11e2-87e2-baf70ac9daa51.3CongressDepartment of LaborExpansion of evaluation efforts within existing programs: In addition to specifying evaluations to be performed with dedicated funding, agencies can also add a general policy and requirements favoring evaluation into existing grants, contracts, or waivers. These measures may require new legislation. For example, Congress recently approved the Department of Labor's request for a small cross-agency set-aside for evaluation activities.Systemic Measurement[Carry out] systemic measurement of costs and cost per outcome_32c9a6f2-0d0b-11e2-87e2-baf70ac9daa51.4Systemic measurement of costs and cost per outcome: Agencies are encouraged to include measurement of costs and costs per outcome as part of the routine reporting of funded programs to allow for useful comparison of cost-effectiveness across programs.PublicationRelease evaluations promptly through agency websites or alternative means_32c9a7c4-0d0b-11e2-87e2-baf70ac9daa51.5Agencies should release evaluations promptly through either their agency websites or alternative means. OMB particularly welcomes agency proposals to improve public access to, and understanding of, evidence about what works and what does not.Cost-EffectivenessUse comparative cost-effectiveness data to allocate resources_32c9a88c-0d0b-11e2-87e2-baf70ac9daa52Washington State Institute for Public Policy (WSIPP)Pew Charitable TrustEnvironmental Protection AgencyU.S. Department of AgricultureUsing comparative cost-effectiveness data to allocate resources. Through the Pew Charitable Trust's Results First initiative, a dozen States are currently adopting a model developed by the Washington State Institute for Public Policy (WSIPP) that ranks programs based on the evidence of their return on investment. Once evidence-based programs have been identified, such an analysis can improve agency resource allocation and inform public understanding. For example, the Environmental Protection Agency and the U.S. Department of Agriculture are working together to incorporate evidence about the cost-effectiveness of different pollution control strategies in the Chesapeake Bay restoration effort.ResearchIdentify areas where research provides strong evidence regarding the comparative cost-effectiveness of agency investments_32c9add2-0d0b-11e2-87e2-baf70ac9daa52.1OMB invites agencies to identify areas where research provides strong evidence regarding the comparative cost-effectiveness of agency investments. The research may pertain to the allocation of funding across agency programs (e.g., research showing that some funding streams have higher returns on investments) or within programs (e.g., research showing that some types of grantees or programmatic approaches have higher returns). Agencies should describe the body of research and then apply its results to support a proposed resource reallocation. OMB is more likely to support an existing resource allocation or a request for new resources supported in this way, and may feature the agency's reasoning in the 2014 Budget.Grant-MakingInfuse evidence into grant-making_32c9aed6-0d0b-11e2-87e2-baf70ac9daa53Infusing evidence into grant-making. Grant-making agencies should demonstrate that, between FY 2013 and FY 2014, they are increasing the use of evidence in formula and competitive programs. Agencies should consider the following approaches, among others:Formula GrantsEncourage use of evidence in formula grants_32c9afbc-0d0b-11e2-87e2-baf70ac9daa53.1Encouraging use of evidence in formula grants: OMB invites agencies to propose ways to increase the use of evidence-based practices within formula grant programs. For example, formula funds can be conditioned on the adoption of evidence-based practices, and high-quality technical assistance can be used to share and support implementation of evidence-based practices. Competitive programs can assign points to applicants based on their integration of such practices into formula streams.Evidence-Based GrantsImplement evidence-based grant programs_32c9b0a2-0d0b-11e2-87e2-baf70ac9daa53.2Department of EducationU.S. Agency for International DevelopmentDepartment of Health and Human ServicesEvidence-based grants: Several agencies -- ranging from the Department of Education to the U.S. Agency for International Development -- have implemented evidence-based grant programs that apply a tiered framework to assess the evidence supporting a proposed project and to determine appropriate funding levels. Under this approach, programs supported by stronger evidence, as established in a rigorous agency process, are eligible for more funding. All programs are expected to evaluate their results. Examples of tiered-evidence programs include the Department of Education's Investing in Innovation program and the Department of Health and Human Services' Teen Pregnancy Prevention and Home Visiting programs. Even without creating tiers, agencies can provide points or significant competitive preference to programs that the agency determines are backed by strong evidence, and can build the evidence base by embedding evaluation into programs. Because running evidence-based programs requires more resources, agencies may wish to combine multiple smaller programs into larger, evidence-based efforts.Pay for SuccessInvite grant applicants to use a "pay for success" approach_32c9b19c-0d0b-11e2-87e2-baf70ac9daa53.3Department of JusticeDepartment of LaborPhilanthropic EntitiesPrivate EntitiesState GovernmentsLocal GovernmentsFoundationsPay for Success: Taking the principle of acting on evidence one step further, the Departments of Justice and Labor will be inviting grant applicants to use a "pay for success" approach, under which philanthropic or private entities (the "investors") pay providers upfront and are only repaid by the government if certain outcomes are met. Payment amounts are based, in part, on the amount that the Federal, State, or local government saves. A pay-for-success approach is appropriate where: (i) improved prevention or other up-front services can produce better outcomes that lead to cost savings at the Federal, State, or local level; and (ii) foundations or others are willing to invest. To date, the Administration has focused its Pay for Success planning on programs financed with discretionary appropriations. OMB invites agencies to apply a pay-for success model for programs funded by either discretionary or mandatory appropriations. Agencies should also consider using the new authority under the America COMPETES legislation to support incentive prizes of up to $50 million. Like Pay for Success, well designed prizes and challenges can yield a very high return on the taxpayer dollar.EnforcementUse evidence to inform enforcement_32c9b296-0d0b-11e2-87e2-baf70ac9daa54Using evidence to inform enforcement. Rigorous evaluation of strategies for enforcing criminal, environmental, and workplace safety laws often reveals that some approaches are significantly better than others at securing legal compliance. OMB encourages agencies to indicate how their allocation or reallocation of resources among enforcement strategies is informed by such evidence._c4418c26-0d0b-11e2-8354-f2190bc9daa5Evaluation CapacityStrengthen agency evaluation capacity_32c9b4c6-0d0b-11e2-87e2-baf70ac9daa55Strengthening agency evaluation capacity. Agencies should have a high-level official who is responsible for program evaluation and can:Research AgendaDevelop and manage the agency's research agenda_32c9b5d4-0d0b-11e2-87e2-baf70ac9daa55.1StudiesConduct or oversee rigorous and objective studies_c4418f78-0d0b-11e2-8354-f2190bc9daa55.2InputProvide independent input to agency policymakers on resource allocation and to program leaders on program management_b3e62bdc-0d0d-11e2-b71e-04060bc9daa55.3Agency PolicymakersProgram LeadersStaff & ResearchersAttract and retain talented staff and researchers, including through flexible hiring authorities such as the Intergovernmental Personnel Act_b3e62f4c-0d0d-11e2-b71e-04060bc9daa55.4Program Performance MeasuresRefine program performance measures, in collaboration with program managers and the Performance Improvement Officer_b3e630a0-0d0d-11e2-b71e-04060bc9daa55.5Performance Improvement OfficersProgram Managers2012-05-182012-10-02http://www.whitehouse.gov/sites/default/files/omb/memoranda/2012/m-12-14.pdfOwenAmburOwen.Ambur@verizon.net