Putting Together the Performance Pieces: A Practical Guide for Federal AgenciesThe goal of this project was to explore the extent to which departments are moving ... activities [required by the GPRA Modernization Act of 2010] beyond compliance exercises led by the PIO's office -- and in so doing, building a performance culture grounded in continuous improvement. We also sought to understand, for those agencies that were more successful in building a performance culture, what practices they used to successfully drive change. To meet this goal, we convened focus groups of top performance improvement leaders and staff at the subcomponent level in six large departments to discuss practices they have used to drive a performance culture and to glean which strategies were most successful.This guide builds on a body of work by the Partnership for Public Service and Grant Thornton during the past five years to document lessons learned about agency performance, specifically about the role and practices of agency performance staff. For two reports, issued in 2011 and 2013 -- "A Critical Role at a Critical Time: A Survey of Performance Improvement Officers" and "Taking Measure: Moving from Process to Practice in Performance Management" -- we interviewed and surveyed department-level PIOs to identify the steps agencies have taken to move performance management beyond being simply a time-consuming compliance exercise. Both of these studies highlighted the importance of leadership support and the role of agency PIOs in driving agency performance.Partnership for Public ServiceP4PS_3ee1d581-8509-4332-acba-06516c635ab0The Partnership for Public Service is a nonprofit, nonpartisan organization that believes good government starts with good people. We help government serve the needs of all Americans by strengthening the civil service and the systems that support it. We pursue this goal by: * Providing assistance to federal agencies to improve their management and operations, and to strengthen their leadership capacity * Conducting outreach to college campuses and job seekers to promote public service * Identifying and celebrating government’s successes so they can be replicated across government * Advocating for needed legislative and regulatory reforms to strengthen the civil service * Generating research on, and effective responses to, the workforce challenges facing our federal government * Enhancing public understanding of the valuable work civil servants performBen BennettAssociate ManagerJudy England-JosephStrategic AdvisorMallory Barg BulmanManaging Editor of ResearchMax Ingraham-RakatanskyResearch FellowShafiq IslamResearch FellowBevin JohnstonCreative DirectorEllen PerlmanStaff WriterAudrey PfundAssociate Design ManagerBeth SchillManagerLara ShaneVice President of Research and CommunicationsMax StierPresident and CEOFederal AgenciesChief Operating OfficersPerformance Improvement OfficersAgency Performance StaffContributorsBetween November 2014 and January 2015, the Partnership for Public Service and Grant Thornton LLP conducted focus groups with performance staff or their equivalent at the subcomponent level in selected federal agencies. We conducted focus groups at six large federal agencies: the Departments of Agriculture, Commerce, Homeland Security, Housing and Urban Development, Labor and the Treasury. We highlighted examples and promising practices from the performance improvement community below the level of the departmental PIO for the purpose of understanding to what degree a performance culture is being embedded within agencies. We narrowed our research to actions performance staff at the subcomponent level are taking to advance their organization’s performance culture, the extent of staff access to valid and reliable cost and performance data, and to what degree program evaluation is integrated into performance initiatives. These focus groups, which took place either as part of a regularly scheduled performance meeting or were convened independently, involved the head of the performance staff in agency subcomponents. Additionally, we held follow-up conversations with members of the departmental PIO office in each agency to gain additional context and clarification. All participants were asked to complete a set of closed-ended questions that were aggregated for the quantitative portions of this report. These were completed by 60 respondents.Don BiceAssociate Director (Performance Improvement Officer), Office of Budget and Program Analysis, Department of AgricultureSue E. BussellsDeputy Chief Information Officer, Office of the Chief Information Officer, Department of AgricultureBill CrewsStrategic Planner and Performance Management Analyst, Risk Management Agency, Department of AgricultureElizabeth DannSenior Strategist, Strategic Planning and Performance Office, Food Safety and Inspection Service, Department of AgricultureSusan Gabriel-SmithManagement Analyst, Office of the Chief Information Officer, Department of AgricultureTherron HagenEconomist, Foreign Agricultural Service, Department of AgricultureKatrina JohnsonProgram Analyst, Office of Budget and Program Analysis, Department of AgricultureKendra KlineAssistant to the Deputy Administrator, Federal Grain Inspection Service; Grain Inspection, Packers and Stockyards Administration, Department of AgricultureCarl MuhlbackerProgram Analyst, Rural Housing Service, Rural Development, Department of AgricultureChris NelsonAnalyst, Office of Budget and Program Analysis, Department of AgricultureJohn RappPerformance Management Branch Chief, Forest Service, Department of AgricultureWinona ScottAssociate Assistant Secretary for Civil Rights, Office of the Assistant Secretary for Civil Rights, Department of AgricultureLea-Ann BigelowDeputy Director, Performance Systems Office of Performance, Evaluation and Risk Management, Office of the Chief Financial Officer and Assistant Secretary for Administration, Department of CommerceKathryn ChantryChief Financial Officer and Director of Administration, Bureau of Industry and Security, Department of CommerceStephen DevineProgram Analyst, Performance and National Programs Division, Economic Development Administration, Department of CommerceHarry KnightProgram Evaluation Lead, Office of Performance, Evaluation and Risk Management, Office of the Chief Financial Officer and Assistant Secretary for Administration, Department of CommerceRick LattimerPolicy Analyst, Economics and Statistics Administration, Department of CommerceKerstin MilliusSenior Program Analyst, Performance and National Programs Division, Economic Development Administration, Department of CommerceBeth NortonProgram Analyst, National Oceanic and Atmospheric Administration, Department of CommerceJeffrey PressDeputy Director, Performance Strategy, Office of Performance, Evaluation and Risk Management, Office of the Chief Financial Officer and Assistant Secretary for Administration, Department of CommerceSamantha SchasbergerProgram Analyst, Performance and National Programs Division, Economic Development Administration, Department of CommerceKenneth StricklettManagement and Program Analyst, Office of Program Planning and Integration, National Oceanic and Atmospheric Administration, Department of CommerceBlanche ZirDirector, Performance Management and Employee Programs, International Trade Administration, Department of CommerceGerald BlackAnalyst, Department of Homeland SecurityLyndon BonaparteOperations Research Analyst, Department of Homeland SecurityThomas CrimanManagement and Program Analyst, Federal Emergency Management Agency, Department of Homeland SecurityAmy CulbertsonAssistant Director for Performance Management, Program Analysis and Evaluation Division, Department of Homeland SecurityJae DelaneyProgram Manager for Strategic Planning, Federal Law Enforcement Training Centers, Department of Homeland SecurityKyle DeWaldChief, Planning and Program Analysis, Citizenship and Immigration Services, Department of Homeland SecurityMelissa EvansPerformance Analyst, Department of Homeland SecurityRuss FreshwaterPerformance Analyst, Program Analysis and Evaluation Division, Office of the Chief Financial Officer, Office of the Undersecretary for Management, Department of Homeland SecuritySarah GrimmProgram Analyst, Transportation Security Administration, Department of Homeland SecurityAnthony HatchettProgram Analyst, Department of Homeland SecurityCarolyn HadijiWorkforce Strategy Analyst, Office of the Chief Human Capital Officer, Office of the Undersecretary for Management, Department of Homeland SecurityDick HershDivision Chief, Performance, Planning and Evaluation, Coast Guard, Department of Homeland SecurityTracey JenkinsBudget Analyst, Office of Operations Coordination and Planning, Department of Homeland SecurityLinda KingDeputy Performance Improvement Officer, Strategic Planning and Performance Management Division, Office of the Chief Readiness Officer, Office of the Undersecretary for Management, Department of Homeland SecurityRachel LewisPerformance Analyst, Science and Technology Directorate, Department of Homeland SecurityNicole LondonManagement and Program Analyst, Department of Homeland SecurityRaymond MillsChief of Special Programs, Director of Communications, Citizenship and Immigration Services, Department of Homeland SecurityMichael MurfeeChief Financial Officer (Acting), Office of Health Affairs, Department of Homeland SecurityKevin MorganManagement and Program Analyst, Customs and Border Protection, Department of Homeland SecurityEd NormanProgram Analyst, Department of Homeland SecurityBen ParkerChief, Program Analysis and Evaluation, Immigration and Customs Enforcement, Department of Homeland SecurityShelley PenmenStatistician, Department of Homeland SecurityTim ShaughnessySenior Technical Advisor, Office of the Chief Procurement Officer, Office of the Undersecretary for Management, Department of Homeland SecuritySharif SokkaryProgram Analyst, Department of Homeland SecurityBrian SterlingProgram Analyst, Office for Civil Rights and Civil Liberties, Department of Homeland SecuritySandy ThomasProgram Analyst, Transportation Security Administration, Department of Homeland SecurityLawrence TobinSenior Program Analyst, Office of Program Accountability and Risk Management, Office of the Undersecretary for Management, Department of Homeland SecurityCaroline TranDeputy Director, Compliance and Technology, Department of Homeland SecurityMorgan VernonManagement and Outreach Specialist, Office of the Chief Security Officer, Office of the Undersecretary for Management, Department of Homeland SecurityPeter WilsonDeputy Director, Business Operations and Support, Department of Homeland SecurityMegan WoolwineStatistician, Management and Organization Division, Department of Homeland SecurityZakiyyah Day Strategic Workforce Planning Manager, Office of the Chief Human Capital Officer, Support Services, Department of Housing and Urban Development Peter DuklisDirector, Office of Field Support and Operations, Office of Multifamily Housing Programs, Department of Housing and Urban DevelopmentAllison FalconePerformance Management Office, Office of the Chief Human Capital Officer, Department of Housing and Urban DevelopmentHenry HensleyPerformance Improvement Officer, Office of Strategic Planning and Management, Department of Housing and Urban DevelopmentJennifer HoSenior Advisor on Housing and Services, Office of the Secretary, Department of Housing and Urban DevelopmentJack MalgeriDirector, Program Standards and Compliance, Office of Fair Housing and Equal Opportunity, Department of Housing and Urban DevelopmentMary McBrideAssistant Deputy Secretary, Office of Field Policy and Management, Department of Housing and Urban DevelopmentAnn Marie OlivaDirector, Office of Special Needs Assistance Programs, Department of Housing and Urban DevelopmentNicho PruettChief, Accountability Branch, Policy Development and Oversight Division, Office of the Chief Human Capital Officer, Support Services, Department of Housing and Urban DevelopmentWilliam SnowSpecialist, Office of Special Needs Assistance Programs, Office of Community Planning and Development, Department of Housing and Urban DevelopmentTerri SwannProgram and Management Analyst, Office of the Chief Human Capital Officer, Department of Housing and Urban DevelopmentChristopher WalshDeputy Performance Improvement Officer, Office of Strategic Planning and Management, Department of Housing and Urban Development Judith BinderBranch Chief, Planning, Policy and Performance Management, Office of Workers' Compensation Programs, Department of LaborDoug BordenLead Program Analyst, Planning, Policy and Performance Management, Office of Workers' Compensation Programs, Department of LaborBraye CloudSpecial Assistant, Office of the Assistant Secretary for Administration and Management, Department of LaborHolly DonnellyDirector, Performance Management Center, Office of the Assistant Secretary for Administration and Management, Department of LaborDave FredericksonDirector, Office of Performance Monitoring, Office of the Assistant Secretary for Administration and Management, Department of LaborChad HancherProgram Analyst, Mine Safety and Health Administration, Department of LaborCharlotte HayesDeputy Assistant Secretary, Office of the Assistant Secretary for Administration and Management, Department of LaborJohn HaymakerPerformance Branch Chief, Office of Federal Contract Compliance Programs, Department of LaborKristin LantzLead Program Management Specialist, Office of Trade and Labor Affairs, Bureau of International Labor Affairs, Department of LaborKaren LivingstonDirector, Wage and Hour Division, Department of LaborRebecca MacMillanSpecial Assistant, Mine Safety and Health Administration, Department of LaborSarah MillerProgram Analyst, Women's Bureau, Department of LaborHeather ParkerWorkforce Analyst, Employment and Training Administration, Department of LaborTraci SmithSpecial Assistant, Office of the Assistant Secretary for Administration and Management, Department of LaborDavid FriedmanRotational President's Management Council Fellow, Office of Strategic Planning and Program Evaluation, Alcohol and Tobacco Tax and Trade Bureau, Department of the Treasury Amanda JonesPresident's Management Council Detailee, Office of Strategic Planning and Performance Improvement, Department of the TreasuryKatie MalagueDirector, Office of Strategic Planning and Performance Improvemen, Department of the Treasury Jill MurphyDirector, Office of Strategic Planning and Program Evaluation,Alcohol and Tobacco Tax and Trade Bureau, Department of the TreasuryJohn PekarikAssociate Chief Financial Officer, Corporate Planning and Internal Control Internal Revenue Service, Department of the TreasuryRenee PruittDivision Chief, Policy, Planning and Budget United States Mint,Department of the TreasuryLenora StilesAnalyst, Office of Strategic Planning and Performance Improvement,Department of the TreasuryGrant ThorntonGT_49b10bfc-245b-11e5-8d34-53758515d2e8Grant Thornton's Global Public Sector, based in Alexandria, Va., is a global management consulting business with the mission of providing responsive and innovative financial, performance management and systems solutions to governments and international organizations. We provide comprehensive, cutting-edge solutions to the most challenging business issues facing government organizations. Our in-depth understanding of government operations and guiding legislation represents a distinct benefit to our clients. Many of our professionals have previous civilian and military public sector experience and understand the operating environment of government. Visit Grant Thornton's Global Public Sector at www.grantthornton.com/publicsector. In the U.S., visit Grant Thornton LLP at www.GrantThornton.com.Demek AdamsAmber GaribMarc HebertRobert MischShelley RappaportRobert SheaShiva VermaSuhany Zaimah_49b111ba-245b-11e5-8d34-53758515d2e8To build a performance culture grounded in continuous improvement. _49b1144e-245b-11e5-8d34-53758515d2e8PerformanceContinuous ImprovementActivities & PrioritiesConnect program activities to agency priorities_49b11610-245b-11e5-8d34-53758515d2e81Alcohol and Tobacco Tax and Trade BureauThe Alcohol and Tobacco Tax and Trade Bureau at the Department of the Treasury carried out a sustained effort to transform the organization's strategic planning process and connect bureau priority goals to employees’ work. The director of the Office of Strategic Planning and Program Evaluation, who serves as that bureau's top performance staff member, led a three-year effort to develop a modified balanced scorecard. The staff rely on the score-card—a performance improvement tool that integrates the bureau’s strategic objectives with key metrics— as the bureau’s new strategic management framework. Performance staff wanted to create an easily understandable management tool for employees at all levels, and use it to set organizational priorities and improve agency performance. They convened a core group of senior and mid-level management and used employee feedback along the way to adapt the effort. The resulting strategic management framework is more effective at aligning employees’ work to the bureau’s goals.Department of CommerceAgency leaders and performance staff at the Department of Commerce used the strategic planning process to increase collaboration among its diverse subcomponents. In March 2014, the department released a new strategic plan that demonstrates how subcomponents work together to achieve key department-wide goals. The redesigned plan—now a concise 54 pages, down from the 140 pages that made up the previous 2014 to 2018 strategic plan—identifies five goals and 19 objectives. It also specifies which subcomponents are responsible for contributing to each goal, connecting subcomponents to shared priorities. For example, 13 of the 19 objectives require that three or more subcomponents collaborate on implementation. Before developing the new strategic plan, Commerce subcomponents only occasionally collaborated. Through this new planning approach, headquarters performance staff have set their sights on greater collaboration among subcomponents by highlighting how they could work together to achieve the agency’s mission. "We realize that we can accomplish so much more collectively than we can individually, and the strategic plan and the way it was developed supports that mindset," said a performance staff member.Department of LaborDepartment of Labor performance staff have more closely linked strategic planning to operational planning, as required by the GPRA Modernization Act. These subcomponent performance staff take opportunities to talk with program staff in regional offices to identify tactical strategies that will allow the subcomponent to accomplish its strategic goals. Through operational planning, performance staff have been able to demonstrate more clearly the relationship between the department’s strategic plan and the work that program staff perform both in the field and at headquarters. Performance staff said they also added performance targets to individuals’ performance plans, creating greater accountability and providing clarity to program staff about their specific roles in helping the department achieve its mission. Performance staff in one agency said it is more difficult to connect employees to an agency’s priorities when an organization is not working directly on one of the agency’s highpriority goals. Some staff may be less committed to performance improvement efforts when that is the case, the agency’s PIO said. “There’s a cultural resistance here that if the secretary or deputy secretary doesn’t have a direct meeting with you on it, well, then why would I do it on my own,” the PIO said. The director of one subcomponent found another way to engage employees. He led an effort in regional offices to require program staff to develop an operating plan directly related to the agency’s strategic plan. Following this exercise, staff had a clearer focus on how their activities contributed to accomplishing their organization’s mission. opportunities to talk with program staff in regional offices to identify tactical strategies that will allow the subcomponent to accomplish its strategic goals. Through operational planning, performance staff have been able to demonstrate more clearly the relationship between the department’s strategic plan and the work that program staff perform both in the field and at headquarters. Performance staff said they also added performance targets to individuals' performance plans, creating greater accountability and providing clarity to program staff about their specific roles in helping the department achieve its mission.Department of Homeland SecurityAt the Department of Homeland Security's Office of the Chief Procurement Officer, performance staff have made the department’s strategic planning process more meaningful by creating a health assessment of procurement using performance information and targets from the department's strategic plan. This tool allows staff across the agency to track progress toward agency goals more easily and see how their work contributes to accomplishing the agency's mission. Staff involved in contracting activities at any level can access the assessment, which is an organized display of metrics that procurement staff oversee, allowing any staff member with a role in procurement to easily track agency progress toward goals. "Building a systematic view of those performance measures, rather than capturing it in something like a strategic plan, is the model that we use," according to a performance staff member in the DHS procurement office.Food Safety and Inspection ServiceA performance staff member at the Department of Agriculture's Food Safety and Inspection Service said her organization uses a strategic performance workgroup -- which includes program and performance staff ranging from front-line employees up to executives -- to build and maintain a strong tie between staff activities and the inspection service's key goals.Department- and subcomponent-level performance officers reported that several GPRA Modernization Act requirements have helped build a performance culture. Many specifically mentioned that the tighter focus on agency priority goals was helping drive performance improvement and has become a good management tool for driving their agencies’ performance. Subcomponent performance staff who believe they have made significant progress on their organization’s performance improvement efforts say they have taken actions to create a clear connection between department priorities and subcomponents’ strategic goals and operational plans. They said they tightened the link between the department’s goals and program staff’s day-to-day work, encouraging greater employee buy-in and commitment to improving performance. This is consistent with recent Government Accountability Office findings. GAO surveyed about 2,400 federal managers regarding their use of performance information and created an index to assess agencies’ use of that information. GAO found that alignment between program performance measures and agency-wide goals and objectives significantly predicted managers’ use of performance information.1 But this alignment is not achieved solely by reporting information in a document. Our conversations with agency PIOs prior to the passage of the GPRA Modernization Act highlighted the importance of involving program staff in strategic planning. For example, one PIO drove performance by working closely with staff to map all agency goals and performance measures, explaining that involving staff at all levels could help facilitate an understanding of where programs fit into overall goals.GoalsCreate clear agency priority goals that are clearly communicated through strategic planning documents._49b1185e-245b-11e5-8d34-53758515d2e81.1ResponsibilitiesDevelop simpler strategic planning documents so staff can see how their individual responsibilities map to the accomplishment of organizational goals._49b11b60-245b-11e5-8d34-53758515d2e81.2Input, Suggestions & FeedbackSolicit employee input, suggestions and feedback from staff who are familiar with programs when creating strategic planning documents._49b122e0-245b-11e5-8d34-53758515d2e81.3EmployeesStaffAnalytical TalentGet the analytical talent you need_49b125ce-245b-11e5-8d34-53758515d2e82Agency Data AnalystsAgency data analysts need diverse skills. Recognizing this, the GPRA Modernization Act required that the Office of Personnel Management identify the competencies for agency staff in the area of analyzing and using performance information to improve government efficiency and effectiveness. In January 2012, after consultation with the Performance Improvement Council and the Office of Management and Budget, the OPM director issued a memorandum that listed a range of skills for agency performance improvement staff. It included: * Accountability * Attention to detail * Customer service * Influencing/negotiating * Information management * Oral communication * Organizational awareness * Organizational performance analysis * Partnering * Performance measurement * Planning and evaluating * Problem solving * Reasoning * Technical competence * Written communicationDepartment of AgricultureOrganizations face challenges finding and recruiting employees with specific skills the agency needs, even when the job title contains the word "analyst," due to the way jobs are classified. One performance staff member at the Department of Agriculture emphasized the agency's need for staff with more advanced statistical skills to pull information from the data the agency collects. "We have management analysts and program analysts by series and choice of grades, but are they truly doing the analytic work? How do we redefine that and work with OPM so that we do have, in fact, the skill sets that are needed?" she said. Now that staff have identified this gap, they are beginning to focus their recruiting efforts to look for potential employees with the statistical skills the agency needs.USDA Office of the Chief Information OfficerOne participant from USDA's Office of the Chief Information Officer recognized that the department could benefit from data analysts with diverse skills. In addition to aggregating and analyzing the data, staff need to be able to synthesize the information in a way that is easily understandable to other USDA employees and agency leaders. "You need to make sure you can produce those reports very quickly, and get it in front of the leadership team so they can look at it and make an informed decision," one participant said.Performance staff in many subcomponents are ... realizing that although their current staff possess basic data analysis skills, they need more sophisticated analysis skills for turning the data into information for decision-making. Department-level PIOs have stated the need for staff with both advanced analytical and communication skills. One PIO said these employees should "be more than number crunchers," explaining that they should possess the analytical capability to understand the data and identify useful insights. Agencies will need to change the way they recruit, hire and train staff to bring in people with essential skills for better performance management. Subcomponent performance staff also said they lack personnel with statistical skills for analyzing and interpreting data. Two-thirds of survey respondents said they and their teams, to a "great" or "very great" extent, have the analytical capacity and skills to assess performance. However, during focus group discussions, they seemed to rethink this assessment, saying they believe they are getting the basic skills they need but would benefit from having people with more advanced capabilities or an unconventional combination of skills. The skills they needed ranged from in-depth knowledge of statistical methods to the ability to communicate effectively about the implications of the data. Performance staff from a few agencies said that they were "data rich and information poor," and recognized the need for greater data analysis by staff with more advanced statistical skills. Participants also said that performance staff would find it helpful to have employees with advanced statistical skills who could turn data into useful and understandable information for making informed decisions. "I'm looking for really, really strong analytical people and creative people -- smart people who can look at the data and understand the connection to the outcomes of a program," said a Department of Commerce performance staff member.SkillsThink creatively about the skill sets the organization needs to assess its performance and lead continuous improvement efforts._49b12830-245b-11e5-8d34-53758515d2e82.1RecruitmentRecruit for analysts with diverse skills, which could include experience in communications, team-building or graphic design._49b1300a-245b-11e5-8d34-53758515d2e82.2TrainingTrain staff in not only data analysis, but also how to clearly present and communicate insights from the analysis._49b13316-245b-11e5-8d34-53758515d2e82.3StaffRelationshipsBuild meaningful relationships_49b13596-245b-11e5-8d34-53758515d2e83Economic Development AdministrationSelect performance staff at the Department of Commerce's Economic Development Administration -- which stimulates regional economic development by strategically investing in economically distressed areas -- said they meet regularly with program field staff around the country. Every quarter, performance staff from Washington, DC, headquarters travel to each of the six regional offices to participate in the review and selection process for new projects. Headquarters performance staff said these conversations have given them the opportunity to learn about the performance of program field staff in much more detail, giving headquarters staff the proper context to understand trends from field offices -- the stories behind the numbers. These conversations also give field program staff the opportunity to ask headquarters performance staff for clarification about expectations and goals, according to headquarters performance staff. Ultimately, they said, these visits create a stronger connection between the regional program staff and headquarters staff with responsibility for improving performance.Food Safety and Inspection ServiceOne of USDA's high-priority goals is to reduce foodborne Salmonella illnesses associated with products regulated by the Food Safety and Inspection Service, such as meat or eggs. When performance staff at the service noticed troubling trends in the quality of their organization’s performance data, they convened the strategic performance working group to address it. At meetings, team members focused on data quality and information that meaningfully demonstrated progress as well as practices not as effective at gauging program performance. Here, again, both the program and performance staff routinely communicate their issues and concerns. Program staff gain greater awareness of effective performance management practices, and performance staff come away with a more thorough understanding of organizational performance and the challenges program staff have in collecting or improving data quality, focus group participants said.Customs and Border ProtectionAt the Department of Homeland Security, performance staff with Customs and Border Protection said they made discussions more meaningful by engaging offices primarily focused on operations, many of which collect large amounts of data but do not have the capacity to analyze it. CBP’s headquarters performance office, which has the staff and analytical capability, reached out to some of the offices, asking to work with their data. Some operational staff were hesitant initially because they feared the data might be used out of context, a focus group participant said. However, after analyzing the data, the headquarters performance staff found information valuable for figuring out how the office’s operations could be enhanced, which allowed performance staff to target specific areas for improvement to help the office achieve its goals. The performance staff at CBP used these success stories to demonstrate to its other operational program offices how sharing data with headquarters performance staff can help them use the information more effectively, and it helped build trust between headquarters and program field staff. Program staff now are eager to collaborate with performance staff and hold productive performance conversations. One performance staff member at CBP headquarters said: "You want to get from the standpoint of trying to chase them down when they see you coming, to the point that they’re waiting outside your door when you get in in the morning. When that happens, you know you've changed the culture."Department of LaborMany performance staff in Department of Labor subcomponents said performance culture evolved significantly thanks to the deputy secretary’s genuine interest in effective performance management. "We had a deputy secretary who was very much into performance management… and he ensured that his leadership team understood what their goals were," one subcomponent performance staff member said. "It wasn't about a compliance exercise." The secretary's responses to the data fostered greater trust among agency performance staff. Staff told us that if a subcomponent reported that all goals were met, the secretary asked if the goals were too easily achieved. If the goal was not met, the secretary was interested in understanding why that happened.Performance staff members representing five subcomponents in multiple agencies said they meet regularly with program field staff, believing that in-person conversations about program activities can provide more context than simply reading a report about those activities. These conversations helped performance staff move beyond discussing the numbers to gaining a qualitative appreciation of improvements being made as well as the challenges keeping program staff from improving. These meetings also help performance staff build meaningful relationships with program staff in field offices and gain a greater understanding of program progress. The collaborative relationships have fostered a more open, positive environment around performance conversations and brought people together to tackle performance management challenges. Similarly, it was important for department-level PIOs to build relationships with their program staff, and also with department leaders. Most saw value to having regular conversations with program staff to focus on understanding the data better and identifying opportunities for improvement. In addition, some PIOs said it was important to involve agency leaders in performance conversations about the implications of the data collected to move program staff beyond simply reporting the data. These PIOs built relationships with agency leaders to equip them to use the information to make decisions. “We have to make sure leadership is engaged. Reporting doesn’t stand by itself,” one PIO said. Meaningful, collaborative relationships between performance and program staff has fostered a more sustainable performance culture, and interviewees said integrating performance management activities is a practice critical to instilling performance into a culture. Regular data reviews provide opportunities for performance staff to learn more about program performance, while also allowing them to gain the trust of program staff. While some subcomponent performance staff noted promising shifts toward more open, effective performance conversations, many admitted that challenges remain. For example, many subcomponent performance staff said they have had negative experiences when reporting less-than-positive results to department-level leaders. Department- level performance staff emphasized efforts to address this issue by focusing on working with sub-component staff to identify negative issues early, sharing data and working more collaboratively. Performance staff in one department- level PIO office, however, said they struggle to foster trust with subcomponent staff. These PIO staff said some subcomponent performance staff admitted they felt uncomfortable in agency-wide performance review meetings when they could not report progress on certain measures. In recent years, the current PIO and PIO office staff reached out to subcomponents that need assistance with improving performance. The department-level staff emphasized they are making an effort to talk more openly with subcomponents about performance issues and working with subcomponent performance staff to identify and fix problems early. They are also trying to increase the quality and frequency of collaboration. Despite lingering challenges, a majority of subcomponent performance staff interviewed in 2015 said their agencies had made progress in driving a stronger culture of performance. Nearly two-thirds of focus group respondents (61.7 percent) said they thought their agencies had made progress in driving a performance culture to a "great" or "very great" extent.ConversationHold conversations between headquarters and subcomponent performance staff to discuss the context behind the numbers._49b13d7a-245b-11e5-8d34-53758515d2e83.1CollaborationBring together performance and program staff who serve diverse roles in the organization to learn and work together. _49b1409a-245b-11e5-8d34-53758515d2e83.2Performance StaffProgram StaffTrustFoster trust with program staff in the field by working with them to discuss and resolve performance challenges._49b149aa-245b-11e5-8d34-53758515d2e83.3Program StaffInformationMove from data to information_49b14bda-245b-11e5-8d34-53758515d2e84Office of Multifamily Housing ProgramsDuring our conversations with agency performance staff, we found that while most agencies had not begun using data to drive decisions effectively, staff in some subcomponents had ideas for the best way to improve their use of available data. The challenge facing performance staff in the Department of Housing and Urban Development's Office of Multifamily Housing Programs -- which facilitates the construction and refinancing of multifamily housing developments -- is with aggregating data from regional offices. The HUD office relies on information collected in a standard way, but some regional offices collect and tailor the data differently, seeking a different data display for their regional operations. This leads to inconsistent data sets from one region to the next and creates challenges when trying to combine them. When the data deviate in this way, it can be difficult or costly for staff to aggregate information and ensure the data are still accurate. The multifamily housing office may be able to resolve this by creating consistent reporting requirements across the regions. "The standardization of reporting would probably help and would be a very cost-effective means of getting all the data down," one performance staff member said.Customs and Border ProtectionPerformance staff at Customs and Border Protection's headquarters office recognize that many offices collect large amounts of data, from immigration statistics to international trade information, that include cost information reported by offices throughout the country. One performance staff member at CBP headquarters said the information collected from these different offices may not always be connected from one region to another in a way that’s effective for making decisions. He explained that by combining and comparing data from various supporting offices, his staff was able to make useful connections and insights, and ask meaningful questions that helped staff in the offices improve performance. "You begin to notice anomalies, like why does it cost more to have a border patrol agent in New Orleans as opposed to El Paso," said one performance staff member. "Should that be the case? You start to then have some meaningful questions."Alcohol and Tobacco Tax and Trade BureauStaff at the Department of the Treasury's Alcohol and Tobacco Tax and Trade Bureau have witnessed growth of the alcohol and tobacco industries in the United States, with the bureau processing an increasing number of applications for permits to manufacture or trade alcohol or tobacco products. The staff has a large amount of information at its disposal but the way the data are structured limits employees’ ability to work with that data. "We have an electronic filing system for all of our core business lines now, but they were not built in such a way that we would be able to dig into it," said a performance staff member.HUDIn other instances, officials described data collected from multiple sources that cannot be combined or aggregated easily. Some data are collected based on program requirements in ways that limit their overall usefulness. For example, performance staff at HUD said that data collection requirements on programs for providing housing for the homeless limit staff insights into the data. To protect the privacy of individuals in the homeless community, HUD staff are only allowed to get data aggregated by community. At the same time, Congress may ask for individual- level results to demonstrate program success. To address this challenge, HUD performance staff looked at complementary data sets to understand more fully the effectiveness of agency programs for reducing homelessness. "A lot of the work we've been able to do comes from the strength of using multiple data sets to get the best information available to us," said one performance staff member. However, it is important to be aware of potential data limitations when combining multiple data sets. For example, data reported by recipients of different types of grants may vary in both reliability and frequency.Many subcomponent performance staff said in recent forums that they know what they need to do to make the data more useful to them, but face challenges getting it done, whether due to lack of resources or outdated requirements. The practice of turning data into information has not become a widespread reality, but agencies should make this a goal and strive to accomplish it. By translating available data into actionable insights, performance staff increase the likelihood that this information will be used by senior leadership to drive decisions. As agencies' efforts on performance management and evaluation evolve, performance staff are becoming increasingly aware of the limitations of the data available for decision-making. Several performance staff said they have too much data and not nearly enough information. They are simply collecting too much of the wrong data. In some cases, the agency is collecting extensive amounts of output data, which was useful at one point but is no longer relevant to the program’s mission and goals. Agency officials described databases full of numbers that were not helpful in determining if the agency was accomplishing its mission. These are ongoing issues at the departmental and subcomponent levels. Agencies often struggle to sort through available data to identify measures that best gauge agency performance against the agency’s mission. "There is plenty of information available, it’s just how you find it," said one PIO. "Then it is hard to translate it into usable data. We're good at collecting data but not so good at analyzing it." In 2013, only 38.8 percent of PIOs reported they were measuring the right things to be successful. It appears this did not improve much two years later, when only 41.7 percent of subcomponent performance staff reported that their agency is measuring the right things to assess progress toward key agency goals.Data StandardizationStandardize data collection across regions or offices to make it easier to aggregate data across subcomponents._49b14edc-245b-11e5-8d34-53758515d2e84.1Data CollectionsEliminate outdated data collection requirements where possible._49b15170-245b-11e5-8d34-53758515d2e84.2Data OutreachReach out to other organizations that collect data that could inform or complement the information currently available to agency staff._49b153aa-245b-11e5-8d34-53758515d2e84.3ROIDemonstrate return on investment_49b156b6-245b-11e5-8d34-53758515d2e85Economic Development AdministrationThe performance staff at the Department of Commerce's Economic Development Administration acknowledges that program evaluation could identify where performance is improving, and draw a connection to the resources that contributed to achieving that progress. Individuals with these evaluation skills have been difficult to find and bring on board, however. As a result, staff at EDA are working with staff in other subcomponents, such as the Economics and Statistics Administration, to improve the rigor of evaluations.Food Safety and Inspection ServiceOther interviewees encountered similar difficulties. Some subcomponents have a more pressing need for combinations of skill sets, leading several to start changing how they hire—creating a position requiring skills in both evaluation and performance management. For example, although performance staff at the Department of Agriculture’s Food Safety and Inspection Service recognize the importance of linking performance management and program evaluation, they face a logistical challenge. The performance management and program evaluation functions are located in two different offices, which can limit staff collaboration. Recently, the Food Safety and Inspection Service created a new position for an employee whose time and job responsibilities are split between the two offices -- linking strategic planning and performance improvement with program evaluations. This employee is helping performance staff understand how best to measure the impact of the service's programs to accomplish strategic goals. The new employee "brings both of those pieces of the puzzle together," said a performance staff member, and contributes to the organization's efforts to integrate performance management and program evaluation.Alcohol and Tobacco Tax and Trade BureauThe Department of the Treasury's Alcohol and Tobacco Tax and Trade Bureau is structured in a similar way. The director of the bureau's Office of Strategic Planning and Program Evaluation said: "I think we recognize the value of combining agency objectives with program activity and key metrics through the strategic plan, and then we evaluate what we are achieving," explaining that the office closely connects the planning and evaluation functions and makes staff more aware of the relationship between them.National Oceanic and Atmospheric AdministrationPerformance staff at Commerce's National Oceanic and Atmospheric Administration have taken steps toward conducting more consistent program evaluations across the organization. NOAA has several offices with advanced evaluation capabilities, such as the National Weather Service, the Office of Education and the National Ocean Service, who have skilled evaluation staff focused on specific programs. Performance staff recognized the need for improved program evaluation across NOAA and formed a working group that convened employees from different offices to discuss how program evaluation is used in different parts of the organization. Staff discovered that the definition and rigor of program evaluation varied. "One challenge is the scope of NOAA’s programs. We needed to build capacity across the organization so that project and program managers routinely use sound research methods," said one performance staff member. The working group’s discussions resulted in the establishment of a standing evaluation committee. The committee sponsors educational activities and developed a program evaluation guide for NOAA staff with tips to design and conduct more effective program evaluations.Department of LaborOf the departments we reviewed that used program evaluation, the Department of Labor was among the most advanced. Performance staff at one of our forums said the key to effective and consistent use of evaluations across the department is to have a centralized program evaluation office. Performance staff at Labor subcomponents can turn to the Office of the Chief Evaluation Officer for expertise and guidance on effective evaluation practices. "We give a lot of attention to evaluation, and a lot of that is possible because we have a chief evaluation officer in the department that provides funding and helps guide a broader evaluation agenda across the department," one performance staff member said. Program and performance staff from Labor's Employment and Training Administration work with the department’s evaluation office to develop ways to gauge and understand the effectiveness of the technical assistance the administration offers to its grantees, to establish which assistance techniques work best and to determine which techniques may not be as effective. ETA serves its stakeholders more effectively by evaluating its efforts and using its findings to make changes. These evaluations are all the more effective because subcomponent staff familiar with the programs work with evaluation office staff who have specialized expertise in program evaluation. "It’s definitely a symbiotic relationship," one performance staff member said.Department of Housing and Urban DevelopmentThe Office of Special Needs and Assistance Programs is the Department of Housing and Urban Development office that funds nonprofit organizations and state and local governments so they can provide housing to homeless Americans. Performance staff there improved how they respond to congressional inquiries by tracking cost data and return on investment. Each year, performance staff determine the budget they need to meet the administration’s goals to end homelessness by evaluating how much it costs grantees to provide housing and other services. In the analysis, staff make a deliberate effort to link how each dollar spent on homelessness assistance helps the recipients receive or maintain permanent housing. Performance staff from multiple HUD subcomponents said they realize that increasing interest by Congress and others in measuring the benefit gained for dollars spent will force greater attention to the department’s costs of doing business.Internal Revenue ServiceTreasury's Internal Revenue Service performance staff collect and analyze a wealth of agency cost data. They can track metrics like the type of cases filed and estimate the average hours of staff time needed to process each type of case. The data enable performance staff to calculate more accurately the number of cases each full-time employee can process. IRS leadership then can demonstrate to Congress the agency’s capacity based on the number of IRS personnel working on cases.Office of Management and BudgetOur research shows the Office of Management and Budget has played a positive role in encouraging agencies to demonstrate return on investment. According to one PIO: "They said, 'Stop sending us these packages without telling us what you're doing to achieve your goals.' So now nobody will get an increase unless there are metrics."Natural Resources Conservation ServiceFor organizations with less easily quantifiable activities, staff are exploring alternative ways to demonstrate the value of investments in programs. Performance staff at USDA's Natural Resources Conservation Service, which offers technical and financial assistance to farmers and private landowners to encourage environmentally sustainable practices, have begun examining the cost of their services. The Natural Resources Conservation Service believes it needs to demonstrate investment value to private landowners who pay some of the cost of conservation efforts, said a performance staff member. Although that value may not be easy to quantify, performance staff have examined the link between the organization's efforts and costs and the value to stakeholders. "The secretary can confidently say these are the kinds of things we are producing, and we know that we’re getting this kind of benefit," said a performance staff member. "We can talk about the tons of soil and then how that equates to the amount of food or fiber we’re producing." In this way, performance staff demonstrate to farmers and private landowners what their efforts mean for their local communities. Performance staff are also able to provide useful information to departmental leadership that allows them to justify agency programs and use of resources.Performance staff we spoke with stressed the importance of demonstrating programmatic return on investment by rigorously evaluating the cost and performance of programs. We found that only 31.7 percent of subcomponent performance staff surveyed in 2015 reported that program evaluation is integrated into overall performance initiatives in their agency. Two years ago, we also found that program evaluations were not playing an important role in agency performance management efforts at the department level. We recommended that the Office of Management and Budget invest in program evaluation activities that enhance understanding of performance and program results and use the knowledge gained to improve performance management. Better practices likely would be adopted more quickly if there were coordination among those who set goals and measure performance and those who lead evaluation initiatives. A recent Government Accountability Office report corroborated this finding by noting that agencies' capacity to conduct program evaluations is uneven. About one-third, or seven of the agencies GAO reviewed, reported having assigned responsibility to a single, high-level official to oversee their evaluation studies. In the most recent focus groups, we found that many staff did not fully understand program evaluation -- long-term, in-depth studies to measure a program’s lasting impact and effectiveness. In other instances, performance staff recognized the value of conducting evaluations but have not made them a priority, even if they would like to, because they lack the capacity to conduct and use program evaluations to determine program effectiveness. The GPRA Modernization Act requires agencies’ strategic plans to describe how program evaluations influenced goal setting. There is value in linking program evaluation to performance management, said a PIO and deputy PIO in an agency with a strong program evaluation office. They said coordinating and sharing information between the two offices provided a clearer picture of what was happening in different programs and what performance issues needed to be addressed. A recent guide on evidence-based policy making by the Pew Charitable Trusts and the MacArthur Foundation emphasized this point and highlighted steps to help governments make better choices through evidence-based policy making. Though it has not been a focus of evaluation efforts in the past, many performance offices have also begun exploring how to demonstrate financial return on investment and estimate the cost of achieving program objectives. With constrained resources, agency staff face pressure to demonstrate that their organizations are using resources efficiently, and staff are taking greater interest in quantifying the public benefit from the money agencies spend. By exploring ways to quantify return on investment, performance staff are more likely to gain the attention of agency leadership and influence decision-making. Some agencies lack reliable cost data, such as data about the cost of materials, labor or other factors needed to deliver a product. This is leading performance staff in some subcomponents to explore alternative ways to measure the cost of delivering programs and demonstrate to stakeholders the return on taxpayer dollars. These efforts are allowing performance staff to have a better understanding of the effectiveness of their organizations' programs, which could help inform evaluations.Barriers & ConnectionsTo more effectively measure the value of programs, break down organizational barriers and connect staff who have performance management, program evaluation and budget expertise throughout the organization._49b15954-245b-11e5-8d34-53758515d2e85.1Cost InformationWork with budget staff to obtain accurate cost information._49b15bca-245b-11e5-8d34-53758515d2e85.2Program EvaluationEstablish a common definition of program evaluation throughout the organization to ensure staff efforts to demonstrate return on investment are consistent._49b15f58-245b-11e5-8d34-53758515d2e85.32015-06-302015-07-06http://fcw.com/articles/2015/06/23/~/media/GIG/FCWNow/Documents/2015/0623Performance_pieces.pdfOwenAmburOwen.Ambur@verizon.net