Considerations for Evaluating Automated Transit Bus ProgramsGiven the potential of transit-bus automation, it is critical to evaluate the benefits and challenges from early implementations. A well-designed evaluation can quantify such societal benefits as improving travel time, increasing mobility, and raising transit ridership. This guide aims to assist transit stakeholders with designing and implementing evaluations of automated transit-bus programs. In designing evaluations, transit agencies and other stakeholders should identify program goals and audiences affected by the technology; develop a logic model that maps project inputs, activities, and outcomes; choose an appropriate evaluation design; and collect and analyze data on key performance indicators related to their program goals.With advances in technology and data gathering, it is expected that program evaluation will be conducted rigorously. This guide provides recommendations on designing and implementing a useful, effective evaluation of a transit bus automation project/pilot/demonstration to measure its impacts and record key lessons learned. However, this guide recognizes that transit agencies face time and budget constraints. This guide highlights important, general principles that can be applied to evaluations of various transit-automation projects. Given the number of factors that affect the quality of evaluations, FTA recommends planning evaluation activities from the start of a program. Figure 1-1 illustrates the recommended steps for designing and implementing an evaluation, and this guide explains each step. For a checklist of key evaluation components, please refer to Appendix A.Federal Transit AdministrationFTA_2b40042e-389d-11ea-b465-dcb92483ea00As described in the Strategic Transit Automation Research (STAR) Plan, the Federal Transit Administration (FTA) is sponsoring research and demonstrations of transit bus automation to help transit agencies, stakeholders, and industry make informed decisions.U.S. Department of TransportationOffice of Research, Demonstration and InnovationJohn A. Volpe National Transportation Systems CenterAdvanced Vehicle Technology DivisionJoseph LunaCo-AuthorElizabeth MachekCo-AuthorSean PeirceCo-AuthorTransit StakeholdersTransit AgenciesTransit IndustryTransit agencies, stakeholders, and industry make informed decisions._2b400546-389d-11ea-b465-dcb92483ea00To assist transit stakeholders with designing and implementing evaluations of automated transit-bus programs._2b40060e-389d-11ea-b465-dcb92483ea00AutomationGiven both the potential of transit automation and the unknowns associated with it, the benefits, challenges, and lessons learned from early demonstrations need to be evaluated and shared.DemonstrationsLearningEvaluationA well-designed evaluation can quantify such societal benefits as improving travel time, reliability, and throughput; increasing mobility (spatial and temporal); enhancing safety; raising transit ridership; and saving money on operations and maintenance.AccountabilityEvaluation also demonstrates agency commitment to accountability and offers agencies the opportunity to engage the public and identify unforeseen areas for improvement. Ultimately, evaluation advances knowledge.EngagementKnowledge-SharingAs agencies share experiences with each other, the benefits and cost savings multiply. Evaluation and knowledge-sharing help agencies plan for future deployments and better position themselves to advocate for public-transportation funding.Goals & AudienceIdentify Program Goals and Audience_2b4006c2-389d-11ea-b465-dcb92483ea00Step 1GoalsIdentify a transit program’s goals._2b400776-389d-11ea-b465-dcb92483ea001.1It is crucial to identify first a transit program’s goals for a given deployment. What does your program aim to accomplish? Why is this program needed? Figure 2-1 presents example program goals for deployments of a smooth acceleration and deceleration advanced driver-assistance system (ADAS) and automated feeder bus service. Identifying program goals establishes direction for a given program. Although goals may evolve during program implementation as a result of unforeseen circumstances, establishing clear goals at an early stage can guide any program changes.AudiencesPinpoint the audiences that will be impacted by a project._2b40082a-389d-11ea-b465-dcb92483ea001.2Along with identifying program goals, it is crucial to pinpoint the audiences that will be impacted by a project. Who would benefit from the new technology? Who might be negatively impacted by the new technology? Potential audiences include: • Users – regular riders (e.g., commuters), infrequent riders, persons with disabilities • Non-users – motorists, pedestrians, bicyclists, local businesses • Agency staff – drivers, managers and supervisors, maintenance, dispatchers, planners, unions Listing potentially-impacted audiences at an early stage not only helps to clarify goals, but also identifies groups to interview to measure whether goals are being achieved. ModelDevelop Logic Model_2b4008de-389d-11ea-b465-dcb92483ea00Step 2Program ManagersEvaluatorsOnce program goals have been identified, it is useful for program managers, in conjunction with evaluators and other agency staff, if possible, to develop a logic model. Logic models summarize how a program achieves its goals; that is, how do a program’s inputs and activities achieve the outcomes observed? Figure 3-1 presents an example logic model.1 As depicted, typical logic models consist of Inputs, which feed into Activities, which result in Outcomes (both short- and long-term).InputsSummarize inputs._2b40099c-389d-11ea-b465-dcb92483ea002.1Inputs consist of the financial, organizational, and human resources that a program has available to meet its goals.ActivitiesSummarize activities._2b400a50-389d-11ea-b465-dcb92483ea002.2Activities refer to what a program actually does. These activities could comprise new processes, research, tools, technology, events, outreach, and so forth. Activities help a program achieve its goals.OutcomesSummarize outcomes._2b400b0e-389d-11ea-b465-dcb92483ea002.3Outcomes correspond to changes in knowledge and behavior of a program’s target audiences. Short-term outcomes could include improved public awareness and operator acceptance of a new technology. Long-term outcomes could include improvements in safety, agency cost savings, and operational performance.2 External FactorsConsider the external factors._2b400cbc-389d-11ea-b465-dcb92483ea002.4In addition to inputs, activities, and outcomes, it is also critical to consider the external factors that might affect a program’s intended goals, e.g., changes in legislation, declines in the broader economy, harsh weather, etc.3 For example, a weakening job market might reduce overall transit ridership, cancelling out ridership gains expected from deployments. As part of evaluation planning, program managers and evaluation staff should brainstorm and identify such external factors. Taking the example of adopting AVs for feeder services, a program logic model could resemble (Figure 3-2)EvaluationChoose Evaluation Design._2b400d8e-389d-11ea-b465-dcb92483ea00Step 3There are several components to consider when choosing an evaluation design, including evaluation questions, evaluation types (process and outcome), counterfactual scenarios, baseline data, and measures of effectiveness.QuestionsDerive questions that an evaluation seeks to answer._2b400ea6-389d-11ea-b465-dcb92483ea003.1Evaluation Questions and Evaluation Types -- Drawing from a program’s goals and its logic model, evaluators derive questions that an evaluation seeks to answer. These questions are similar to hypotheses that are tested in a scientific experiment. As examples, the following evaluation questions might be applicable to a test of ADAS, although the specific questions will need to be tailored to the nature of the deployment: • Did ADAS-equipped buses save fuel relative to non-equipped buses on the same route? • Did transit drivers use the ADAS as intended? Did they find them useful? • Did ADAS reduce variability in headway times? • How effective was the program’s public engagement effort in terms of raising public understanding of the project? • How effectively did the program respond to maintenance challenges during the pilot demonstration (alternative challenges: schedule, procurement, etc.)? ^ An evaluation question should be clear, specific, objective, and politically neutral; further, the terms in an evaluation question should be readily-defined and measurable, whether quantitatively or qualitatively (GAO 2012).4 Evaluation questions should be linked to the audiences, activities, and goals laid out by a project.5 Evaluation questions that are ambiguously written or include multiple combinations of activities and outcomes can be complicated to measure and may yield misleading recommendations. Such questions to avoid could include, for example: • Did AVs change the passenger experience? • Did ADAS reduce collisions and stress amongst transit drivers? • Given schedule improvements resulting from AVs, does route ridership increase? ^ The first question is ambiguous and can invite multiple interpretations. What is meant by change? Would such change be positive or negative? Passenger experience can encompass a wide range. The second question is doublebarreled: it incorporates two possibilities that may not necessarily be correlated. For instance, it is possible to reduce collisions with ADAS, but at the cost of increased stress on transit drivers if there are many false alarms. The third question is a leading question that could bias the interpretation of data; it assumes that schedule improvements would result from AVs, but such improvements could be influenced by a variety of other factors. As a result, observed ridership increases could be falsely attributed to schedule improvements due to AV technology.6 As evaluators derive evaluation questions, it is helpful to keep in mind two main types of program evaluations: process and outcome evaluations. Evaluation questions often fall into one of those two categories. Process evaluations examine the management and execution of a given program’s activities. For example, questions on how a program manages its public-engagement activities and how a program responds to challenges during the pilot would pertain to a process evaluation. Process evaluations often occur while a program is in progress; as such, the findings of a process evaluation can be used to improve program management in real time. Alternatively, outcome evaluations focus on a program’s outcomes to measure whether a program met its intended goals. Outcome evaluations typically occur once a program has completed its activities.7 Although some programs can budget for multiple evaluation teams, many agencies can field only one evaluation for a program. For agencies that can field only one evaluation team, the team could include both process and outcome questions in its evaluation. Transit-automation technologies are still new, and there are important questions to address, such as whether a given technology works and whether it is suitable for transit applications. Therefore, evaluations of transit automation should be leveraged to contribute to this growing field of knowledge to ease future deployments.Baselines & CounterfactualsCompare program accomplishments to alternative scenarios._2b400f6e-389d-11ea-b465-dcb92483ea003.2Counterfactual Scenarios and Baseline Data -- Evaluations compare what a program has accomplished (or is accomplishing) to what would have happened had there been (a) no program at all (no-build scenario) or (b) a different program in its place (next-best alternative). If an evaluation focuses only on the outcomes and benefits of a program, it would not be clear whether those outcomes or benefits would have been achieved without the program or perhaps with a less-costly alternative. Would a given transportation situation worsen in the absence of the proposed program? What would happen if a transit system continued to use conventional technologies? Comparing a project’s benefits to a no-build scenario (also known as a “counterfactual”) or next-best alternative not only provides a more accurate evaluation, but such comparison strengthens justification for further program support. Once counterfactual or alternative scenarios are established, evaluation teams should assess what a given transportation situation looked like prior to program implementation.8 Program managers and evaluation teams, consulting their established program goals, should identify the data needed for measuring a baseline transportation situation and changes to that situation that result from the program. The National Transit Database (NTD) Glossary can help program managers and evaluators in selecting consistent metrics for measuring baselines, since these metrics are already collected as part of NTD reporting. However, there may be other relevant metrics that go beyond NTD’s scope.EffectivenessDetermine whether a program is meeting its stated goals._2b40104a-389d-11ea-b465-dcb92483ea003.3Measures of Effectiveness9 -- In identifying baseline data and monitoring changes to that data, program managers and evaluators are pinpointing measures of effectiveness (also known as key performance indicators or KPIs) to determine whether a program is meeting its stated goals. Changes in values of these KPIs can help an agency determine whether a particular investment in automated transit bus programs has “moved the needle” in such areas as customer satisfaction, safety, and so forth. Please refer to Appendix B for KPIs drawn from a variety of reports related to automated transit. Although these KPIs are offered as examples for reference, KPIs should be tailored to each technology deployment. Other potential measures/KPIs can be drawn from the NTD, the Transit Cooperative Research Program (TCRP), and the National Cooperative Highway Research Program (NCHRP).Evaluation DesignsCompile a strategy to answer evaluation questions._2b401130-389d-11ea-b465-dcb92483ea003.4Evaluation Designs -- An evaluation design is the overall strategy that is used to answer evaluation questions; data collection and analysis methods (discussed in the next section) are tactics used in executing the evaluation design. There are many different designs that a program evaluation can adopt. GAO (2012) notes that good evaluation design should be appropriate for the evaluation questions and context, adequately address the evaluation questions, fit available time and resources, and rely on sufficient, credible data.10 Descriptions of a few common evaluation designs, adapted from GAO (2012), are summarized below. Many evaluations mix different designs.Case StudiesExplore issues in depth, from both qualitative and quantitative perspectives._2b401216-389d-11ea-b465-dcb92483ea003.4.1Case studies allow evaluators to explore issues in depth, from both qualitative and quantitative perspectives. Case studies are particularly suitable for process evaluations, but they are also relevant for outcome evaluations. Case studies typically have a smaller sample size—that is, they focus on only a few projects or components—than other evaluation designs, but allow for deeper study of each project or component. Because of their small sample size, case studies are generally not statistically representative, but nevertheless should be chosen carefully to ensure that there is representation across the relevant variables of interest.ExperimentsCompare outcomes for treated groups with control groups._2b4012fc-389d-11ea-b465-dcb92483ea003.4.2Randomized experiments are considered the ideal form of evaluation. In a randomized experiment, the “treatment” (program intervention, such as a funding grant) is assigned to participants (e.g., transit agencies, State transportation departments) randomly. This random assignment controls for any biases in the population that could affect outcomes. The group of participants that receives the program intervention would be known as the “treatment group,” and those that did not receive the intervention would be called the “control group.” Evaluators would then compare the outcomes observed for the treated group with the control group that did not receive the intervention to establish the effectiveness of a given program. Randomized experiments need to be designed and implemented at the start of a program, and random assignment can be difficult to implement in realworld transportation settings. Further, these experiments are time- and resource-intensive, so they often are not suitable for many scenarios.Quasi-ExperimentsCompare those that have been affected by a program to similar control groups._2b401400-389d-11ea-b465-dcb92483ea003.4.3Quasi-experiments offer a compromise solution to randomization. In many quasi-experiments, the treatment is not assigned randomly. However, evaluators can compare those that have been affected by a program to a control group that is similar to the treatment group—but that have not been exposed to the treatment. For example, if a new transit technology is deployed on two routes, evaluators could compare outcomes on those two routes with two other routes (lacking the new technology) that have similar ridership, length, traffic patterns, etc., to establish a new technology’s effect.11StatisticsIdentify ways in which a program led to outcomes._2b4014f0-389d-11ea-b465-dcb92483ea003.4.4Statistical analysis offers another design possibility where randomized and quasi-experiments are not possible. Such analysis can be done via a variety of quantitative methods that describe the relationship between a program and its outcomes or that identify ways in which a program specifically led to outcomes. Although statistical analysis can demonstrate different relationships amongst observed data, such analysis should be viewed cautiously because it can be difficult to establish how a data relationship was caused.12DataCollect and Analyze Data._2b4015ea-389d-11ea-b465-dcb92483ea00Step 4Once an evaluation design (e.g., case studies, quasi-experiment, statistical analysis) is selected, evaluators should choose appropriate data-collection methods to assess a program’s measures of effectiveness. Some typical methods include:Surveys & QuestionnairesAssess perceptions of passengers and agency staff._2b4016ee-389d-11ea-b465-dcb92483ea004.1PassengersAgency StaffGAOGAO (1993) notes that questionnaires are useful when a large amount of standardized information must be collected, when different sets of people are involved, and when those people are located in widely separated locations.13 Questionnaires can collect a wide variety of information, from facts to statistics to opinions. However, evaluators must consider several elements to design a valid questionnaire. Has the survey sample been chosen in an unbiased manner? Have the questions been written appropriately? Please refer to GAO (1993) for resources on designing and deploying survey questionnaires.EvaluatorsRegardless of the data-collection method chosen, evaluators should take steps to reduce bias. For instance, in questionnaires respondents may give an answer that they think will please the interviewer (social-desirability bias). In other cases, respondents will lack knowledge sufficient to provide a response but will provide a response anyway. There are several methods for reducing respondent bias. One such method is conjoint analysis, where respondents review pairs of scenarios in which key criteria have been randomized. Respondents then indicate which of the paired scenarios they prefer. Such a design can allow researchers to determine which criteria are most important to respondents while reducing potential bias.14 Anchoring vignettes are another method for reducing respondent bias; these are short, hypothetical stories that help “anchor” respondent responses to normative questions.RespondentsBecause respondents may have different definitions of how much they agree with a given item, anchoring vignettes normalize responses across respondents.15 Depending on available time and budget, it is recommended that evaluation teams “pilot test” questionnaires and other data-collection methods with a small sample of respondents or experts. Such pilots can catch and mitigate response biases prior to full deployment.Within questionnaires, a Likert-scale question is a common method for assessing the extent to which respondents agree with a given item. For example, a transit driver may be asked the following question with these Likert-scale responses: Overall, how satisfied or dissatisfied were you with the ADAS user interface? (Choose one.) – Very satisfied – Satisfied – Neither satisfied nor dissatisfied – Dissatisfied – Very dissatisfied For Likert-scale questions, it is important to ensure that questions do not “lead” respondents toward one answer or another. Such questions must also be written clearly, avoiding language that would be confusing to a respondent... In terms of analyzing data once it has been gathered, there are two main categories of analysis—descriptive and inferential. Descriptive analysis presents characteristics of data without necessarily discussing how those characteristics came about. For instance, such data characteristics as mean, median, range, variance, and mode are considered descriptive. Descriptive statistics are often visualized through histograms, line graphs, scatter plots, and various other graphics. Inferential analysis, on the other hand, intends to establish causality—that is, how did a particular set of findings come about? Methods such as quasi-experiments and statistical regression are employed in inferential data analysis, but such methods should be used cautiously and with a strong understanding of confounding factors. Qualitative data gathering, such as through surveys and interviews, can help to establish a causal story on top of data analysis. Such qualitative data from surveys and interviews also can be statistically analyzed through content-analysis software.SensorsGather safety- and operations-related data._2b4017e8-389d-11ea-b465-dcb92483ea004.2(e.g., on-vehicle LIDAR, cameras, roadside sensors)RecordsCapture impacts on transit run times, ridership, safety incidents, labor costs, and other elements that typically are recorded._2b4019d2-389d-11ea-b465-dcb92483ea004.3Interviews & Focus GroupsAscertain in-depth opinions of passengers and agency staff._2b401b08-389d-11ea-b465-dcb92483ea004.4PassengersAgency Staff2019-12-312020-01-16https://www.transit.dot.gov/sites/fta.dot.gov/files/docs/research-innovation/146801/considerations-evaluating-automated-transit-bus-programs-fta-repor-no0149.pdfOwenAmburOwen.Ambur@verizon.net