Toward Metrics for Re(imagining) Governance: The Promise and Challenge of Evaluating Innovations in How We GovernThis GovLab Working Paper is an attempt to describe the promise and challenge of evaluating new practices of problem solving in governance, specifically citizen-engagement interventions. The purpose is to inform those innovators who are eager to develop a deeper insight into what works for the key steps, opportunities and challenges involved when assessing impact. In addition, the paper aims to develop a foundation for further interdisciplinary research and debate. This paper summarizes the recent lessons learned in the field of evaluation and examines how new methods could help assess the impacts of participatory governance and emerging social technologies used in governance. This work is not meant to be a comprehensive review of current approaches to evaluating new governance initiatives but is intended to frame the issues involved, and suggest where work needs to be done to develop better assessments. We are also providing an annotated and open bibliography for further reading, inviting others to expand upon it. The Governance LabTGLAleise BarnettCo-authorDavid DemboCo-authorStefaan G. VerhulstCo-authorJohn D. and Catherine T. MacArthur FoundationThis paper was made possible with support from the John D. and Catherine T. MacArthur Foundation Innovators in GovernanceInnovators who are eager to develop a deeper insight into what works for the key steps, opportunities and challenges involved when assessing [the] impact [of] new practices of problem solving in governance.To describe the promise and challenge of evaluating new practices of problem solving in governance.EVIDENCE-BASED INTERVENTIONSMove from faith-based to evidence-based interventions.I. I. MOVING FROM FAITH-BASED TO EVIDENCE-BASED INTERVENTIONS -- Governance—how institutions analyze information and make decisions to solve collective problems—is broken. Around the world, we face increasingly complex challenges ranging from widespread poverty to serious ecological crises that threaten our planet’s future. Yet trust in traditional institutions of governance is at an all-time low. At the same time, we are living through the greatest era of disruptive innovation and rapid experimentation since the Industrial Revolution. Tremendous progress in information and communication technologies, including big data and social media, are empowering individuals to engage with one another—and with traditional institutions of governance—to tackle problems collectively. Groups of individuals with diverse social, intellectual, and professional backgrounds can now use technology to collaborate in new ways that can drive progress more rapidly and effectively than ever before. From local and federal governments to leading universities and Fortune 500 companies, institutions have an opportunity to reevaluate how they solve problems in the networked age.   While there is good reason to believe that breakthroughs may come from recent innovations such as communitybased problem solving, behavioral economic insights about human behavior, or predictive analytic experiments, there are limited studies measuring exactly how productive it is to use these kinds of new governance techniques. Without a deeper understanding of whether, when, why and to what extent an intervention has made an impact, any initiative we design will be sub-optimal and will produce less than the desired results. If we are going to accelerate the rate of experimentation in governance and create more agile institutions capable of piloting new techniques and getting rid of ineffectual programs, we need research that will enable us to move away from “faith-based” engagement initiatives toward “evidence-based” ones.METRICS & METHODSLearn from evaluation.IIEngineers Without Borders CanadaLEARNING FROM EVALUATION: METRICS AND METHODS -- Before we try to change how we solve problems and plan new initiatives we need to know what has worked already and what hasn’t. Over the past few decades there have been renewed calls to measure the impact of societal initiatives in order to allocate resources wisely and understand what to continue, scale up or cancel. With government and philanthropic funding shrinking, both government agencies and charities see outcome-based evaluation as an urgent need. There has also been an increased demand to understand with more precision and nuance whether programs are successful, how they are successful and what factors have led to that success. Even when an initiative demonstrates positive outcomes, it is still helpful to understand: * Whether there are different impacts for some populations and objectives than for others. * How much better off people are as a result of the initiative—only slightly, or significantly? * Whether a program works differently in different geographical regions. * Why a program has been effective and what factors have contributed to its success. It can also be helpful to understand what doesn’t work, despite the incentive to report only successful elements of a program in order to maintain funding. Engineers Without Borders Canada, for instance, produces an annual Failure Report which shares learnings about less effective innovations in a step to “create a culture of creativity and calculated risk taking.” With a clear understanding of both successes and failures, more effective and more innovative initiatives can be designed and better opportunities for intervention can be identified. Evaluation is a complex and challenging task that requires judgment to choose how, what and when to measure along with the criteria for an intervention to be deemed successful. Our review of the existing literature on evaluation has highlighted several issues, challenges and approaches.CONCEPTUAL FRAMEWORKDevise conceptual frameworks that make explicit the relationships among resources available to implement interventions, activities planned, and sought-after results.II.ATo determine an intervention’s impact, an evaluation must be based upon a concept or theory of change. In order to understand the theory of how and why an initiative will work, it is helpful to devise a conceptual framework, or logic model. The logic model makes explicit the relationships among resources available to implement an intervention, activities planned, and sought-after results. It also theorizes how the results, or outputs, of the initiative will lead to both short-term beneficial outcomes and longer-term, fundamental impact. The figure below depicts an example of a logic model for an initiative dedicated to reducing the incidence of diabetes. Part of the initative’s theory of change is that the morbidity of diabetes can be reduced by offering faith-based nutrition and exercise courses encouraging people at risk for diabetes to make healthier lifestyle choices. However, there are assumptions at each stage of the logic model: for example, the assumption that people will want to participate in the program or that participating in the program will lead to a change in knowledge or behavior. It is crucial to acknowledge the assumptions of the logic model and assess their accuracy, since flawed assumptions will lead to misleading conclusions. METRICS & INDICATORS Choose indicators [to enable] feedback for stakeholders to understand performance and how it is measured.II.BThe selection of indicators and metrics for assessment is necessarily a value-based decision, since ultimately we measure what we deem important. A metric can be thought of as a numerical unit of measurement, such as ROI (return on investment), while an indicator is a metric tied to one or more targets, such as GDP (gross domestic product), so that it can measure the degree to which specific goals are being met. Indicators must be chosen with care: they become the feedback mechanism for allowing internal and external stakeholders to understand performance and how it is measured. Indicators also become points of reference that serve as part of the rationale for making decisions, both in the short- and long-term. They should be chosen in the context of the logic model for an initiative. Indicators build on outputs, which are basic metrics of success in quantitative terms, such as number of trainings delivered or number of people participating in a program. Outputs themselves are basic indicators of a sort. But more high-level indicators are also needed to measure how well an initiative achieves the short-term outcomes and long-term impact it is intended to. Different stakeholders may have different criteria for determining whether a program is successful. While some parties may be more interested in knowing whether certain outputs have been delivered, others may measure success only by the amount of impact an initiative made on the outcomes of interest. By consulting a variety of stakeholders in designing an evaluation strategy, researchers can confirm their agreement on a causal framework and establish agreed-upon indicators to measure success. The application of newer, more participatory and bottom-up design strategies provides new opportunities to fine-tune and improve the metrics used, reflecting what matters most for the targeted population. METHODOLOGYDesign research to evaluate societal intervention programs.II.COnce indicators have been established, data can be collected in a number of different ways. These methods include, for instance, direct observation, questionnaires, document review, focus groups, interviews, and examining administrative data. The best method or methods may depend on the program itself. For example, a questionnaire-based strategy won’t work without access to the population participating in a program. There are traditionally three broad categories of research designs to evaluate societal intervention programs: * Experimental * Quasi-experimental * Non-experimentalExperimental DesignUse random assignment to compare results between the two groups to establish the effect of the program.II.C.iExperimental designs use random assignment; researchers randomly assign one group of people to be program participants and another group to be non-participants, and compare results between the two groups to establish the effect of the program. Experimental designs are often commonly referred to as randomized controlled trials, or RCTs, and when done correctly are often thought of as the most rigorous research design. For example, researchers may be interested in finding out whether participating in a microfinance program leads to higher savings on average. After obtaining a list of people who wish to participate in the program, they would randomly assign half of the people to the program group and the other half to the control group, who would not receive the program. The only difference between the control group and program group is access to the program.  This means that researchers would be able to confidently conclude that it is the program which causes higher savings for the program group rather than some other characteristic or phenomenon.Quasi-Experimental DesignUse a program group and comparison group which has not taken part in the program.II.C.iiQuasi-experimental designs may use a program group and comparison group which has not taken part in the program, or may simply consider one group, measuring the effects before and after that group receives the program. Quasi-experimental designs differ from experimental designs in that participants in the program and comparison group are NOT randomly assigned; rather a comparison group may be selected after a program group has been established. This technique is less rigorous because there is no certainty that the two groups are comparable across all characteristics; for example, those who choose to sign up for a microfinance program may be more motivated to save, and it may be this characteristic which ultimately explains their higher savings rather than the program.Non-Experimental TechniquesUse surveys or other non-experimental techniques to gain descriptive information.II.C.iiiFinally, non-experimental techniques, considered the least rigorous method for measuring program success, may involve a single survey delivered after a program in order to gain descriptive information. Non-experimental techniques cannot accurately pinpoint the causes of the outcomes they measure. For example, if researchers using this design survey participants in a job skills development program and find that many of them have gotten jobs, they would not be able to determine whether this was simply due to a rise in the overall employment rate rather than the new skills this group developed.Performance MeasurementUse ongoing monitoring to establish whether operational goals are being met and delivering the desired outputs.II.C.ivPerformance measurement is another assessment technique and can provide different insights than program evaluation. While outcome/impact evaluation measures the overall success of a program, performance measurement uses ongoing monitoring to establish whether operational goals are being met and delivering the desired outputs.Process EvaluationDetermine whether programs have been implemented according to their plans.II.C.vProcess evaluations are focused on determining whether a program has been implemented according to plan. Process evaluations consider one snapshot in time, while performance measurement is an ongoing monitoring process.NEW TRENDS & TOOLSUse more meticulous and comprehensive research methods to measure the impact, using new devices and tools to gather evidence.II.DOrganizations looking to create social change are increasingly using more meticulous and comprehensive research methods to measure the impact of their work, using new devices and tools to gather evidence. The Impact Reporting and Investment Standards (IRIS), developed by the Global Impact Investing Network, for instance, can be used to describe an organization’s performance from a social, environmental and financial perspective. IRIS allows organizations to understand their own performance. And by looking at different IRIS measures, stakeholders concerned about impact investing can estimate the aggregated impact of that sector. Another new device used for assessment in the development arena is a poverty scorecard, developed by Mark Schreiner, Director of Microfinance Risk anagement, L.L.C. Individually created for target countries, these instruments are made up of 10 simple and observable indicators comprised of questions such as whether a household has a straw roof. Since many indicators of poverty are highly correlated, the scorecards are succinct because the answer to one question can imply the answer to another question. For example, in all likelihood a household with a straw roof will not have indoor plumbing. In the poverty scorecard, indicators were selected to be objective, verifiable, quick to ask and answer, and liable to change over time so they can be used to track progress or negative trends. Each indicator was then assigned a point value, which can be used to calculate a poverty score. This is an easily implemented instrument which allows field workers to quickly assess poverty levels. Our review of the literature has also shown an increased call for qualitative methods rooted in ethnography. True ethnography focuses on understanding the perspective and culture of a group of people in order to understand their behavior. Quasi-ethnographic research methods intend to take into account people’s experiences and perspectives to add a contextual dimension to research and evaluation. Small organizations that don’t have the resources for traditional monitoring and evaluation may find it especially useful to solicit feedback and hear about the experiences of program participants. Ongoing feedback can also help organizations make changes and understand program accomplishments as they go. Finally, increased availability of real-time data along with enhanced analytical capabilities (often called big data) promises to radically change and improve how we assess outcomes and impact. When designed well, big data may allow practitioners to track progress and understand where existing interventions require adjustment much faster. But in the words of a recent UN Global Pulse report on the use of big data to assess development goals,“With the promise come questions about the analytical value and thus policy relevance of  this data—including concerns over the relevance of the data in developing country contexts, its representativeness, its reliability—as well as the overarching privacy issues of utilising personal data.”EVALUATIONEvaluate innovations in governance.IIIEVALUATING INNOVATIONS IN GOVERNANCE -- These basic considerations of evaluation—conceptual framework, metrics and indicators, and methodology—are critical in assessing any societal intervention. In addition, there are particular considerations in applying each of these to assessments of citizen engagement and data-sharing in governance. The following observations are presented to help establish best practices from the limited yet growing literature on assessing the impact of new governance initiatives.CONCEPTUAL FRAMEWORK III.AIn a recent paper, Fukuyama described the poor state of assessing governance worldwide, which he explains as resulting from the lack of any conceptual framework: [W]e cannot measure what we cannot adequately conceptualize, we have to start with the concept first.” A related challenge in identifying a logic model or conceptual framework for governance innovation is linked with the diversity of goals underlying citizen engagement and data-sharing in governance. Goals may be as broad as enhancing democracy or as narrow as helping a specific agency operate more efficiently. Abelson and Gauvan, for instance, have this to say about goals of public participation and democracy generally: “Democratic theory tells us that public participation is undertaken for different purposes and with different underlying goals. Tensions exist between views of participation as an essential element of successful democracy (and inherently desirable in its own right) and participation as a means for achieving something else, be it a specific decision outcome, a desire for more informed, accountable or legitimate decision making, or perhaps to delay or share the blame for a difficult decision. Lying somewhere between is the desire for public participation to contribute to a more educated and engaged citizenry.” They go on to discuss how different parties to an intervention may have differing goals: “[D]ifferent evaluation perspectives…may exist among interested parties. For example, sponsors and taxpayers tend to be interested in value for money. But sponsors and organizers of public participation should also be interested in questions of efficacy and effectiveness (if the purpose is summative evaluation) and whether the public participation method implemented was successful as measured against its goals (to address a formative evaluation purpose). Participants themselves are increasingly interested in whether their involvement makes a difference (i.e., policy impact) and, as taxpayers, they also want to see that their involvement was meaningful given that investments in public participation are typically made at the expense of direct service and program delivery. These differing perspectives are integrally linked to the different underlying goals for public participation.” Chess has identified three different approaches that are routinely taken to assess public participation: “1) user-based evaluation, which assumes that different participants will have different goals and that the evaluation must take these different goals into account; 2) theory-based evaluation which is driven by theories and models of public participation and applies normative criteria universally to any public participation effort; and 3) goal-free evaluation which is not constrained by any stated goals and is conducted in the absence of any theory.”User-Based EvaluationIII.A.iTheory-Based EvaluationIII.A.iiGoal-Free EvaluationIII.A.iiiMETRICS & INDICATORSDevelop a set of common standards for assessing citizen engagement initiatives.III.BCoalition for Evidence-based PracticeThe Coalition for Evidence-based Practice issued the report, “Which Study Designs Can Produce Rigorous Evidence of Program Effectiveness?” with an appendix that includes a number of case studies of interventions in government where RCTs have been effective.Institute of Development StudiesThe Institute of Development Studies examined various evaluations in the open/participatory government arena. One of the reports looks at assessments of transparency and accountability initiatives in the fields of Service Delivery, Budget Processes, Freedom of Information, Natural Resource Governance, and Aid Transparency.Center for Technology in GovernmentThe Center for Technology in Government at SUNY Albany has conducted a number of studies in related fields (see e.g., The Dynamics of Opening Government Data and Developing Public Value Metrics for Returns to Government ICT Investments). Taewoo Nam, also at SUNY Albany, reviewed a number of studies of evaluation specifically in the area of citizen sourcing using Web 2.0 technologies and discusses frameworks that researchers can use to evaluate the impacts of citizen sourcing.OpenDataResearch.orgExploring the Emerging Impacts of Open Data in Developing Countries (ODDC) is a recent project aimed at investigating ways in which open data improve governance, support citizens’ rights and promote more inclusive development in developing countries. They are attempting to share information on evaluation and assessment of open data initiatives. Their website presents a number of case studies evaluating the impacts of open data.World BankThe World Bank, through its Striking Poverty program, is engaged in a debate on evaluating citizen involvement not only in transparency and accountability, but also in interventions that have an impact on people’s lives.NestaNesta’s initative “Making Evidence Useful” includes a network of centres in the U.K. that will develop and evaluate assessment methods to identify what works.Project on Government OversightThe Project on Government Oversight highlights best practices in open government in the U.S., including a list of resources on open government initiatives.Selecting indicators and metrics of success for citizen engagement in governance is a value-laden process with abundant opportunity for misinterpretation. It is not clear whether a universal framework for success can be achieved, but some have suggested that there is a need to develop a set of common standards for assessing citizen engagement initiatives. Each initiative may be so different that the desired outcomes will vary widely depending on the program. However, it may be possible to develop thematic buckets within each subset of the participatory government space, defining a set of metrics for each area. Some researchers assessing open government have suggested that rigid distinctions between quantitative and qualitative assessments are no longer helpful. The SUNY Albany Center for Technology in Government suggests that measures of the return on investment for government interventions should be judged only on whether they are “valid and useful in the relevant context of measurement.” The Open Government initiative of the Obama administration provides a good example of the evolution in measurement and metrics needed in this field. To assess the success of this effort, the Openthegovernment.org organization developed agency evaluations using a scale of 1 to 3 to judge adherence to the directive. But that exercise is only able to determine whether agencies made their data available to the public. Missing from the directive itself, and from initial evaluation efforts, was any measurement of progress on the overall goal of the directive: to engage the public in governance. As these initiatives move forward, additional metrics are being developed to measure more substantive issues of participation. As the report on this study explained, “…directional policy without benchmarks or specific increments of improvement is akin to diagnosing fever without a thermometer.” Several other organizations have been carrying out evaluations/assessments of open participatory government interventions, some of which already have case studies that give an indication of the effectiveness of various methods of evaluation. They include the following examples:METHODOLOGYTest programs that then may be tried on a grander scale.III.CMichael GreenstoneCoalition for Evidence-Based PolicyThe Coalition for Evidence-Based Policy has a website devoted to “Top Tier Evidence” that promotes the use of randomized controlled studies. They rely on a recent National Academy of Sciences report that includes “Recommendation on Criteria for Establishing Strong Evidence of Effectiveness.” The Coalition suggests, though, that the possibility of selecting a randomized intervention and control groups is itself a significant problem. The agency contemplating the intervention must be able to make assignments to those who will receive the intervention as well as to those who won’t, or who will receive a different intervention.National Academy of SciencesAlthough costly and time-consuming, RCTs are considered helpful in rigorously establishing impact of relatively mature open government initiatives. There is a difference of opinion, however, as to the applicability of this approach to evaluating government. For this approach to work, experiments have to be set up ahead of time with a control group and a group that is part of the program. There are ethical issues involved, as providing government services to one part of the population and withholding them from another is tricky. One possibility that has been part of our federal “experiment” is to have individual states or localities test programs that then may be tried on a grander scale. Michael Greenstone has written about the need for these kinds of experiments. In “Toward a Culture of Persistent Regulatory Experimentation,” Greenstone points out some of the problems with randomized controlled trials and evaluating government programs in general. “Doing What Works,” the project of the Center for American Progress, also calls for experimenting at the state and local level and adapting what works to the national level... Quasi-experimental techniques can also be useful in evaluating outcomes while non-experimental methods can be a valuable starting point for new initiatives. Recent developments and new tools such as big data should also factor into how open government evaluations are designed. And finally, as discussed in Section II, qualitative assessments are gaining renewed attention as ways of assessing open and participatory government initiatives. There are recent attempts to make such assessments more rigorous and also to mix qualitative and quantitative assessments, as advocated by Khargram and Thomas. These two argue for a new “Platinum” standard that incorporates RCTs with qualitative evidence to be used in assessing public administration and third party governance.CONCLUSION & QUESTIONSIVCONCLUSION AND QUESTIONS MOVING FORWARD -- We are witnessing broad experimentation in how we solve public problems through increased public participation in governance and the use of new information and communications technologies. But what do we actually know about the impacts of these changes? Given the budgetary conditions of most governments we can expect more calls for “doing what works” both for fiscal reasons and to have substantive impacts on social problems. But how do we know what works? And for whom? And exactly how? When we do discover that interventions “work,” or work to some extent? How do we replicate these efforts? Equally important, how do we adjust or end interventions that don’t work, or don’t work as well as we think they should? The following four themes are important to incorporate as we move forward in evaluating interventions in citizen engagement in governance:ObjectivesProvide answers to the questions we really should seek to answer to achieve certain participatory objectives.IV.ADifferent horses for different courses. “Citizen engagement” comprises many different “means” to achieve many different “ends.” Depending on the context, citizens can play different roles: as providers of ideas and expertise (think of crowd sourcing, predictive analytics, grand challenges, prize-induced innovation, brainstorming, etc.); or as representatives of specific interests (in the context of participatory budgeting, citizen juries and deliberative polling). And the contexts of engagement may differ substantially—from post-conflict zones to gentrified city blocks. How do we provide answers to the questions we really should seek to answer: to achieve certain participatory objectives, what works, with whom, and under what conditions? ImpactsDetermine the impact on people’s lives.IV.BImproving people’s lives. Too often, the indicators used to measure citizen engagement are only meant to quantify the level of engagement—such as the number of people that participated or the volume of comments received. For citizen engagement to be meaningful and relevant, more effort is needed to answer the question: How do we start determining what the impact is on on people’s lives?—the ultimate benchmark of success.EngagementEngage citizens to determine what success should look like and what to measure.IV.CCitizensWe measure what we value. Indicators (particularly of the statistical kind) are sometimes presented as neutral or scientific tools of measurement. In fact, though, they are inextricably linked with values—i.e., we measure what we care about. Indicators are socio-political constructs. As such, we should consider whether and how to involve citizens in determining how citizen engagement is measured (possibly using new on-line mechanisms such as rating and feedback tools). How can we best engage with citizens to determine what success should look like and what to measure?EvaluationImprove evaluation of governance innovation through increased experimentation in methods and practice.IV.DExperimentation in how we measure. While much experimentation in citizen engagement is taking place, experimentation in how we measure citizen participation is limited. Recent studies have shown that a mix of methodologies is needed to evaluate interventions in government. Reliance on the traditional gold standard of RCTs is not appropriate in many instances, and at the same time, relying solely on anecdotal evidence is no longer sufficient. But advances in both these areas hold out promise. In order to distill best practices and lessons learned, the field would benefit from increased experimentation using a variety of methods to understand the value that open government brings to people and the difference it makes in their lives. In particular, the use of big data to provide real time feedback that allows us to witness the impact of change to policy “as it happens” offers many opportunities for faster experimentation. How can we improve evaluation of governance innovation through increased experimentation in methods and practice, including the use of big data?2013-04-182013-09-23http://thegovlab.org/wp-content/uploads/2013/04/GovLabMetrics.pdfOwenAmburOwen.Ambur@verizon.net