Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over PublishabilityIn our view, the key for improving the efficiency of knowledge accumulation is to capitalize on existing motivations to be accurate and to reduce the emphasis on publication itself as the mechanism of achievement and advancement. Scientists have strong accuracy motivations. And in the long run, getting it right has a higher payoff than getting it published. However, the goal to publish is immediate, palpable, and concrete; the goal to be accurate is distal and abstract. As a consequence, the short-term payoffs of publishing can be inordinately influential (Liberman & Trope, 1998; Trope & Liberman, 2003), particularly for early-career scientists for whom there is relative urgency for markers of achievement. To address this, the conditions of daily practice must elevate the importance of the more abstract, longer-term goals in comparison to the persisting importance of the concrete, shorter-term goals... we suggest new or altered practices to meet these objectives.We titled this article “Scientific Utopia” self-consciously. The suggested revisions to scientific practice are presented idealistically. The realities of implementation and execution are messier than their conceptualization. Science is the best available method for cumulating knowledge about nature. Even so, scientific practices can be improved to enhance the efficiency of knowledge building. The present article outlined changes to address a conflict of interest for practicing scientists—the rewards of getting published that are independent of the accuracy of the findings that are published. Some of these changes are systemic and require cultural, institutional, or collective change. But others can emerge “bottom-up” by scientists altering their own practices.Brian A. NosekBANCo-Author -- We, the present authors, would like to believe that our motivation to do good science would overwhelm any decisions that prioritize publishability over accuracy. However, publishing is a central, immediate, and concrete objective for our career success. This makes it likely that we will be influenced by self-serving reasoning biases despite our intentions. The most effective remedy available for immediate implementation is to make our scientific practices transparent. Transparency can improve our practices even if no one actually looks, simply because we know that someone could look.Jeffrey R. SpiesCo-AuthorMatt MotylCo-AuthorCommentersAcknowledgments -- We thank Yoav Bar-Anan, Roger Giner-Sorolla, Jesse Graham, Hal Pashler, Marco Perugini, Bobbie Spellman, N. Sriram, Victoria Stodden, and E. J. Wagenmakers for helpful comments.Yoav Bar-AnanCommenterRoger Giner-SorollaCommenterJesse GrahamCommenterHal PashlerCommenterMarco PeruginiCommenterBobbie SpellmanCommenterN. SriramCommenterVictoria StoddenCommenterE. J. WagenmakersCommenterAcceleration of the Accumulation of KnowledgeTo develop strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.Scientific MethodThe chief thing which separates a scientific method of inquiry from other methods of acquiring knowledge is that scientists seek to let reality speak for itself, and contradict their theories about it when those theories are incorrect. . . RealityHypothesesScientific researchers propose hypotheses as explanations of phenomena, and design experimental studies to test these hypotheses via predictions which can be derived from them. PredictionsRepeatabilityThese steps must be repeatable, to guard against mistake or confusion in any particular experimenter. . . . DocumentationScientific inquiry is generally intended to . . . document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, giving them the opportunity to verify results by attempting to reproduce them.ArchivalData SharingVerificationReplicationOpennessThree areas of scientific practice—data, methods and tools, and workflow—are largely closed in present scientific practices. Increasing openness in each of them would substantially improve scientific progress.Scientific ProgressTransparencyResponsibilityCredibilityAccuracyTechnologyExisting technologies allow us to translate some of this ideal into practice. We make our unpublished manuscripts available at personal Web pages (e.g., http://briannosek.com/) and public repositories (http://ssrn.com/). We make our study materials and tools available at personal Web pages (e.g., http://people.virginia.edu/~msm6sw/materials.html; http://people.virginia.edu/~js6ew/). We make data available through the Dataverse Network (e.g., http://dvn.iq.harvard.edu/dvn/dv/bnosek), and we are contributing to the design and construction of the Open Science Framework for comprehensive management and disclosure of our scientific workflow (http://openscienceframework.org/). AccountabilityOpening our research process will make us feel accountable to do our best to get it right and, if we do not get it right, to increase the opportunities for others to detect the problems and correct them. HumanityOpenness is not needed because we are untrustworthy; it is needed because we are human.Paradigm-Driven ResearchPromote and reward paradigm-driven research1Research AuthorsIt is easy to do more paradigm-driven research if authors make their paradigms available to others. The primary risk of paradigm-driven research is that research questions can evolve to being about the method itself rather than the theory that the method is intended to address. Using a single methodology for a theoretical question can reify idiosyncratic features of that methodology as being the phenomenon. This is where conceptual replication provides substantial added value. Paradigm-driven research provides confidence in the accuracy of findings. Conceptual replication ensures that the findings are theoretically general, not methodologically idiosyncratic.Whereas conceptual replication is used only to confirm prior results, another relatively common research strategy, paradigm-driven research, can be used for both confirming and disconfirming prior results. Paradigm-driven research accumulates knowledge by systematically altering a procedure to investigate a question or theory, rather than varying many features of the methodology—by design or by accident. This offers an opportunity to incorporate replication and extension into a single experimental design (Roediger, 2012). Paradigm-driven research balances novelty and replication by building new knowledge using existing procedures. Effective use of this approach requires development of standards, sharing and reuse of materials, and deliberate alteration of design rather than wholesale reinvention. For example, the Deese-Roediger-McDermott paradigm for studying false memories (Roediger & McDermott, 1995) has been adapted to examine how aging (Butler, McDaniel, Dornburg, Price, & Roediger, 2004), mood (Storbeck & Clore, 2005), and expectations (Schacter, Israel, & Racine, 1999) influence the frequency of false memories (see Gallo, 2010, for a review). The subsequent findings reinforce the original results through direct replication and extend those findings by identifying moderating influences, mechanisms, and boundary conditions. A paradigm-driven approach provides confidence in the validity of an effect (or doubt if it fails to replicate) while simultaneously extending knowledge in new directions.ChecklistsImprove authoring, reviewing, and editing by using checklists.2Authors Authors already follow some checklist-like requirements, such as the formatting prescribed by the style manuals of the American Psychological Association (APA) or the Modern Language Association. It is easy to conceive of statistical and disclosure checklists for authors and editorial teams. CONSORTFor example, CONSORT has a 25-item checklist describing minimum standards for reporting randomized controlled trials (http://www.consort-statement.org/). Checklists can ensure disclosure of obvious items that are sometimes forgotten: sample sizes, statistical tests, effect sizes, and covariates included in analysis. They can also define best practices and methodological standards for domain-specific applications.ReviewersAuthors’, reviewers’, and editors’ examination of each article is almost entirely ad hoc. EditorsSocietiesSocieties, journals, and individuals could maintain simple checklists of standard requirements to prevent errors and improve disclosure.JournalsAuthor, reviewer, and editor checklists -- Earlier we expressed some doubt about raising expectations of reviewers for catching errors, with one exception—easy to implement checklists such as that suggested by Simmons and colleagues (Simmons et al., 2011). Checklists are an effective means of improving the likelihood that particular behaviors are performed and performed accurately (Gawande, 2009). Information & Practices[Ensure that] key information is included and advisable methodological practices are identified “naturally” and systematically in the review process.2.1Why are checklists needed? The most straightforward reason is that key information is left out with stunning frequency, and advisable methodological practices are not identified “naturally” or systematically in the review process. For example, the value of reporting effect sizes has been widely disseminated (Cohen, 1962, 1969, 1992; Wilkinson and Task Force on Statistical Inference, 1999). Nonetheless, reporting effect sizes has become common only in recent history and is still not standard practice. A checklist requiring their inclusion before publication would change this. Further, Bouwmeester and colleagues examined 71 prediction studies from high-impact medical journals and found pervasive methodological shortcomings in design, reporting, and analysis decisions, such as clear specification of predictor and outcome variables, description of participant exclusion criteria, and handling of missing values (Bouwmeester et al., 2012). They concluded that “The majority of prediction studies in high impact journals do not follow current methodological recommendations, limiting their reliability and applicability.” High standards for publication do not translate into specific standards for reporting. Mindsets & IncentivesChallenge mindsets that sustain the dysfunctional incentives3Hiring CommitteesThis anecdote suggests that some degree of publishing productivity is essential to get into the pool of competitive candidates, but after that, other factors are more important for getting the job. Without sufficient evidence, we speculate that publication numbers and journal prestige heuristics do play a role in initial selection from a large hiring pool and then play a much smaller role when the pool is narrowed and the hiring committees can look deeply at each candidate. At that point, the committees can invest time to examine quality, potential impact, and direction of the research agenda. In tenure and promotion cases, the depth of processing ought to be even more acute as it is a detailed review of a single candidate’s record.Early-Career ScientistsThis conclusion is based on anecdotal data. Early-career scientists would get useful information from a systematic review of the degree to which publication numbers and journal prestige predict hiring and promotion. Multiple departments could pool and share evidence. The aggregate data might confirm the prevailing perception that publication numbers and journal prestige are the key drivers for professional success, or as we believe, they would illustrate notably weaker predictive validity when the evaluation committee has resources to examine each record in detail.Earlier we stated: “With an intensely competitive job market, the demands for publication might seem to suggest a specific objective for the early-career scientist: publish as many articles as possible in the most prestigious journals that will accept them.” Although this is a common perception, particularly among early-career scientists, we also believe that there are good reasons—though not yet sufficient evidence—to challenge it. For example, the first author regularly presents to graduate students summary data of the short list from a past search for an assistant professor in psychology at the University of Virginia. For this particular search, more than 100 applications were received. Table 1 presents the 11 applicants that made it to the short list. All short-list candidates had at least four publications and at least one first-authored publication. On the basis of publication numbers, there are clear standouts from this group, such as the postdoc with 35 publications, an assistant professor with 21 publications, and a graduate student with 10 publications. Further, these candidates published in prestigious outlets. However, none of these three were selected as a finalist. In fact, two of the three interviewed candidates were among the least productive on the short list.Replication Metrics[Develop] metrics to identify what is worth replicating4Even if valuation of replication increased, it is not feasible—or advisable—to replicate everything. The resources required would undermine innovation. A solution to this is to develop metrics for identifying replication value (RV)—what effects are more worthwhile to replicate than others? The Open Science Collaboration (2012b) is developing an RV metric based on the citation impact of a finding and the precision of the existing evidence of the effect. It is more important to replicate findings with a high RV because they are becoming highly influential, and yet their truth value is still not precisely determined. Other metrics might be developed as well. Such metrics could provide guidance to researchers for research priorities, to reviewers for gauging the “importance” of the replication attempt, and to editors who could, for example, establish an RV threshold that their journal would consider as sufficiently important to publish in its pages.Crowd SourcingCrowd sourcing replication efforts5Open Science CollaborationFor example, in 2011, the Open Science Collaboration began investigating the reproducibility of psychological science by identifying a target sample of studies from published articles from 2008 in three prominent journals: the Journal of Personality and Social Psychology, the Journal of Experimental Psychology: Learning, Memory, and Cognition, and Psychological Science (Carpenter, 2012; Yong, 2012). Individuals and teams selected a study from the eligible sample and followed a standardized protocol. In the aggregate, the results were intended to facilitate understanding of the reproducibility rate and factors that predict reproducibility. Journal of Personality and Social PsychologyJournal of Experimental PsychologyOpen Science CollaboratorsFurther, as an open project, many collaborators could join and make small contributions that accumulate into a large-scale investigation. The same concept can be incorporated into replications of singular findings. Some important findings are difficult to replicate because of resource constraints. Feasibility could be enhanced by spreading the data collection effort across multiple laboratories.Individual scientists and laboratories may be interested in conducting replications but not have sufficient resources available for them. It may be easier to conduct replications by crowd sourcing them with multiple contributors. Peer Review Standards[Develop] peer review standards focused on the soundness, not importance, of research6JournalsPeer ReviewersPLoS ONEPeer review at PLoS ONE is explicitly an evaluation of research soundness and not its perceived importance. Since its introduction in 2006, PLoS ONE’s growth has been astronomical. In 2011, 13,798 articles were published (70% acceptance rate), making it the largest journal in the world. Given the disregard for importance in the review process, one might surmise that PLoS ONE’s impact factor would be quite low. In fact, its 2011 impact factor was an impressive 4.41. This put it in the top 25% of general biological science journals and nearly as high as Psychological Science (4.7). This casts further doubt on reviewers’ ability to predict importance (Gottfredson, 1978) or at least one indicator of importance: citation impact. With a publishing model focused on soundness, negative results and replications are more publishable, and the journal identity is not defined as publishing research that is otherwise unpublishable.Psychological ScienceBiological Science JournalsJournals with peer review standards focused on the soundness, not importance, of research -- The basis of rejection for much research is that it does not meet the criterion of being sufficiently “important” for the journal considering it. Many manuscripts are rejected on this criterion, even if the reviewers identify the research as sound and reported effectively. Despite evidence of the unreliability of the review process for evaluation and identifying importance (Bornmann, Mutz, & Daniel, 2010; Cicchetti, 1991; Gottfredson, 1978; Marsh & Ball, 1989; Marsh, Jayasinghe, & Bond, 2008; Peters & Ceci, 1982; Petty, Fleming, & Fabrigar, 1999; Whitehurst, 1984), this is a reasonable criterion given that journals have limited space and desires to be prestigious outlets. However, in the digital age, page limits are an anachronism (Nosek & Bar-Anan, 2012). Digital journal PLoS ONE (http://plosone.org/) publishes research from any area of scientific inquiry. Barrier ReductionLower or remove the barrier for publication7arXivSuccessful models already exist, such as arXiv, the public repository for physics and other fields (http://arxiv.org; see also http://ssrn.com/ and http://repec.org/). By submitting their manuscripts to arXiv, authors make their work publicly available to the physics community. Peer review—through the “typical” journals—occurs independently of disseminating manuscripts through the repository. If physicists want to wait for peer review to determine everything they read, they can still do so. But most physicists use arXiv to keep up to date on what other laboratories are doing in their specialty.Research AuthorsBy making it trivial to publish, the act itself is no longer much of an incentive. Anyone can publish. The incentives would then shift to evaluation of the research and its impact on future research (i.e., its contribution to cumulating knowledge). Also, the priorities in the peer review process would shift from assessing whether the manuscript should be published to whether the ideas should be taken seriously and how they can be improved (Nosek & Bar-Anan, 2012). Further, this would remove a major barrier to publishing replications and negative results if and when they occur. The only barrier left would be the authors’ decision of whether it is worthwhile to write up a report at all.Research SystemFinally, this change would alter the mindset that publication is the end of the research process. In the present system, it is easy to perceive the final step in research occurring when the published article is added to one’s vita. That is the incentive of publication but not of knowledge building. Knowledge building incentives are satisfied when the research has impact on new investigations. By reducing the value of publication, the comparative value of having impact on other research increases (see Nosek & Bar-Anan, 2012, for a detailed discussion and addressing of common concerns about the impact of moving to a postpublication peer review model).A more radical fix than the PLoS ONE model is to discard publishing as a meaningful incentive. How? Make it trivial to publish. The peer review process presently serves as both gatekeeper and evaluator. Postpublication peer review can separate these concepts by letting the author decide when to publish. Then, peer review operates solely as an evaluation mechanism (Armstrong, 1997; Nosek & Bar-Anan, 2012; Smith, 1999). Nosek and Bar-Anan (2012) provide in-depth discussion for how this is achievable by embracing digital journals and public repositories and by restructuring the review process. Ultimate SolutionOpening Data, Materials, and Workflow8ScientistsOpenness provides scientists with confidence in the claims and evidence provided by other scientists. Further, reputation enhancement is a primary mechanism for reward in unstructured contribution systems. Scientists gain and lose status by their public contributions to scientific progress. As such, public reputation management is the primary lever for promoting accountability in academic science.JournalsIn present research practice, openness occurs almost entirely through a single mechanism—the journal article. Buckheit and Donoho (1995) suggested that “a scientific publication is not the scholarship itself, it is merely advertising of the scholarship” to emphasize how much of the actual research is opaque to readers. For the objective of knowledge accumulation, the benefits of openness are substantial. Openness increases accountability (Lerner & Tetlock, 1999); makes it easier to share, adapt, extend, and critique methods, materials, analysis scripts, and data; can eliminate the file-drawer effect; and can improve the potential for identifying and correcting errors (Ioannidis & Khoury, 2011; Ioannidis & Panagiotou, 2011; Schooler, 2011; Stodden, 2011).The Ultimate Solution: Opening Data, Materials, and Workflow -- Implementing the strategies in the previous section will shift the incentives toward more efficient knowledge accumulation. They do not, however, address the core factor that led Motyl and Nosek to conduct a replication in the opening anecdote—accountability. Science is a distributed, nonhierarchical system. As noted by Nosek and Bar-Anan (2012): "Open communication among scientists makes it possible to accumulate a shared body of knowledge. . . . Individual scientists or groups make claims and provide evidence for those claims. The claims and evidence are shared publicly so that others can evaluate, challenge, adapt, and reuse the methods or ideas for additional investigation. . . . Science makes progress through the open, free exchange of ideas and evidence. (p. 217)"Open Data[Establish] open data as a standard practice across all of the sciences8.1SciencesHuman Genome ProjectFor example, the Human Genome Project acknowledges its principle of rapid, unrestricted release of prepublication data as a major factor for its enormous success in spurring scientific publication and progress (Lander et al., 2001). PsychologistsThe concerns about credibility may be well founded. In one study, only 27% of psychologists shared at least some of their data upon request for confirming the original results even though APA ethics policies required data sharing for such circumstances (Wicherts et al., 2006; see also Pienta, Gutmann, & Lyle, 2009). Further, Wicherts et al. (2011) found that reluctance to share published data was associated with weaker evidence against the null hypothesis and more apparent errors in statistical analysis—particularly those that made a difference for statistical significance. This illustrates the conflict between personal interests and scientific progress—the short-term benefit of avoiding identification of one’s errors dominated the long-term cost of those errors remaining in the scientific literature.Research Infrastructure ProjectsMovement toward open data is occurring rapidly. Many infrastructure projects are making it easier to share data. There are field-specific options such as OpenfMRI (http://www.openfmri.org/; Poldrack et al., 2011), INDI (http://fcon_1000.projects.nitrc.org/), and OASIS (http://www.oasis-brains.org/) for neuroimaging data. And, there are field-general options, such as the Dataverse Network Project (http://thedata.org/) and Dryad (http://datadryad.org/). OpenfMRIhttp://www.openfmri.org/INDIhttp://fcon_1000.projects.nitrc.org/OASIShttp://www.oasis-brains.org/Dataverse Network Projecthttp://thedata.org/Dryad http://datadryad.org/JournalsSome journals are beginning to require data deposit as a condition of publication (Alsheikh-Ali, Qureshi, Al-Mallah, & Ioannidis, 2011). Funding AgenciesLikewise, funding agencies and professional societies are encouraging or requiring data availability postpublication (National Institutes of Health, 2003; National Science Foundation, 2011; PLoS ONE, n.d.).Professional SocietiesNational Institutes of HealthNational Science FoundationPLoS ONEResearchersOf course, although some barriers to sharing are difficult to justify—such as concerns that others might identify errors—others are reasonable (Smith et al., 1986; Stodden, 2010; Wicherts & Bakker, 2012). Researchers may not have a strong ethic of data archiving for past research; the data may simply not be available anymore. Many times data that are available are not formatted for easy comprehension and sharing. Preparing data takes additional time (though much less so if the researcher plans to share the data from the outset of the project). Research ParticipantsFurther, there are exceptions for blanket openness, such as inability to ensure confidentiality of participant identities, legal barriers (e.g., copyright), and occasions in which it is reasonable to delay openness—such as when data collection effort is intense and the data set is to be the basis for multiple research projects (American Psychological Association, 2010; National Institutes of Health, 2003; National Science Foundation, 2011). The key point is that these are exceptions. Default practice can shift to openness while guidelines are developed for the justification to keep data closed or delay their release (Stodden, 2010).American Psychological AssociationWith the massive growth in data and increased ease of making it available, calls for open data as a standard practice are occurring across all of the sciences (Freese, 2007; King, 2006, 2007; Schofield et al., 2009; Stodden, 2011; Wicherts, 2011; Wicherts & Bakker, 2012)... Arguments for open data cite the ability to confirm, critique, or extend prior research (Smith, Budzieka, Edwards, Johnson, & Bearse, 1986; Wicherts, Borsboom, Kats, & Molenaar, 2006; Wolins, 1962), the opportunity to reanalyze prior data with new techniques (Bryant & Wortman, 1978; Hedrick, Boruch, & Ross, 1978; Nosek & Bar-Anan, 2012; Poldrack et al., 2011; Stock & Kulhavy, 1989), increased ability to aggregate data across multiple investigations for improved confidence in research findings (Hrynaszkiewicz, 2010; Rothstein, Sutton, & Borenstein, 2006; Yarkoni, Poldrack, Van Essen, & Wager, 2010), the opportunity for novel methodologies and insights through aggregation and big data (Poldrack et al., 2011), and that openness and transparency increase credibility of science and the findings (Vision, 2010)... The rate of errors in published research is unknown, but a study by Bakker and Wicherts (2011) is breathtaking. They reviewed 281 articles and found that 15% contained statistical conclusions that were incorrect—reporting a significant result (p < .05) that was not, or vice versa. Their investigation could only catch statistical errors that were detectable in the articles themselves. Errors can also occur in data coding, data cleaning, data analysis, and result reporting. None of those can be detected with only the summary report. For example, a study looking at sample mix-ups in genome-wide association studies found evidence that every single original data set examined had at least one sample mix-up error, that the total error rate was 3%, and that the worst performing paper—published in a highly prestigious outlet—had 23% of its samples categorized erroneously (Westra et al., 2011). Further, correcting these errors had a substantial impact on improving the sensitivity of identifying markers in the data sets. Making data openly available increases the likelihood of finding and correcting errors and ultimately improving reported results. Simultaneously, it improves the potential for aggregation of raw data for research synthesis (Cooper, Hedges, & Valentine, 2009), it presents opportunities for applications with the same data that may not have been pursued by the original authors, and it creates a new opportunity for citation credit and reputation building (Piwowar, 2011; Piwowar, Day, & Fridsma, 2007). Researchers who create useful data sets can be credited for the contribution beyond their own uses of the data.Methods & Tools[Develop] open methods and tools8.2AuthorsAuthors cannot identify and report every detail that may be important in a method, but many more parts of the methodology can be shared outside of the report itself. For example, it is easy to create a video of the experimental setting and conduct a simulation of the procedure for posting on the Internet.Figshare Figshare (http://figshare.com/) offers a repository for data and methods or materials for private archiving or public sharing. Open Science FrameworkFurther, the Open Science Framework (http://openscienceframework.org/) is a Web-based project management framework for documenting and archiving research materials, analysis scripts, or data, and it empowers the user to keep the materials private or make them public.Open data allow confirmation, extension, critique, and improvement of research already conducted. Open methods have the same effect and also facilitate progress in reuse, adaptation, and extension for new research (Schofield et al., 2009). In particular, open methodology facilitates replication and paradigm-driven research. Published reports of methodologies often lack sufficient detail to conduct a replication (Donoho, Maleki, Rahman, Shahram, & Stodden, 2009; Stodden, 2011). At best, the written report is the authors’ understanding of what is critical for the methodology. However, there are many factors that could be important but go unmentioned—for example, the temperature of the room for data collection, the identities of the experimenters, the time of day for data collection, or whether instructions were delivered verbally or in written form. Moreover, in paradigm-driven research, changes to the methodology are ideally done by design, not by accident. The likelihood of replicating and extending a result is stronger if the original materials are reused and adapted rather than reinvented on the basis of the new researchers’ understanding of the original researchers’ written description... Presently, only the scientific report is cited and valued. Openness with data, methods, and tools makes them citable contributions (Mooney, 2011; Piwowar et al., 2007; http://www.data-pass.org/citations.html). Contributing data or methods that are the basis for multiple investigations provides reputation enhancement for the originator of the resources. Vitas can include citations to the articles, data sets, methods, scripts, and tools that are each independently contributing to knowledge accumulation (Altman & King, 2007). Also, the ready availability of these materials will accelerate productivity by eliminating the need to recreate or reinvent them. Further, reinvention based on another’s description of methods is a risk factor for introducing unintended differences between the original and replicated methodology.Workflow[Develop] open workflow8.3National Institutes of HealthFor example, clinicaltrials.gov is a National Institutes of Health–sponsored study registry for clinical trials. International Committee of Medical Journal EditorsIn 2005, the International Committee of Medical Journal Editors started requiring authors to register their randomized controlled trials prior to data collection as a condition for publication. CompaniesCompanies sponsoring trials have an obvious financial conflict of interest for the outcome of the research. A registry makes it more difficult to hide undesired outcomes. Indeed, using registry data, Mathieu, Boutron, Moher, Altman, and Ravaud (2009) found that 31% of adequately registered trials showed discrepancies between the registered and published outcomes. For those in which the nature of the discrepancies could be assessed, 82% of them favored reporting statistically significant results.ScientistsOf course, money is not the only source of conflict of interest. Scientists are invested in their research outcomes via their interests, beliefs, ego, and reputation. Some outcomes may be more desirable than others—particularly when personal beliefs or prior claims are at stake. Those desires may translate into design, analysis, and reporting decisions that systematically bias the accuracy of what is reported, even without realizing that it is occurring (Kunda, 1990; Mullen, Bauman, & Skitka, 2003). LaboratoriesPublic documentation of a laboratory’s research process makes these practices easier to detect and could reduce the likelihood that they will occur at all (Bourne, 2010). Further, registration of studies prior to their completion solves one aspect of the file-drawer effect—knowing what research was done even if it does not get published (Schooler, 2011).ResearchersAn obvious concern about transparency of workflow is that researchers are not interested in most of the details of what goes on in other laboratories. Indeed, though advocating this strongly, we do not expect that we would routinely look at the details of other laboratory operations. However, there are occasions for which access would be useful. For example, when we are inspired by another researcher’s work and aim to adapt it for our research purposes, we often need more detail than is provided in the summary reports. Access to the materials and workflow will be very useful in those cases.Data.govFurther, although we do not care to look at the public data about U.S. government expenditures ourselves (http://www.data.gov/), we are pleased with the transparency and the fact that someone can look. Indeed, much as investigative journalism provides accountability for government practice, with open workflow, new contributors to science might emerge who evaluate the knowledge accumulation process rather than produce it and are valued as such.Story TellersFinally, using a registry in an open workflow can clarify whether a finding resulted from a confirmatory test of a strong a priori prediction or was a discovery in the course of conducting the research. The current default practice is to tell a good story by reporting findings as if the research had been planned that way (Bem, 2003). However, even if we intend to disclose confirmation versus discovery, our recollection of the project purpose may not be the same as the project purpose when it began. People reconstruct the past through the lens of their present (Schacter, 2001). People are more likely to presume that what they know now was how they conceived it at the beginning (Christensen-Szalanski & Willham, 1991; Fischoff, 1977; Fischoff & Beyth, 1975). Without a registry for accountability, findings may be genuinely and confidently espoused as confirmatory tests of prior predictions when they are written for publication. However, discoveries are more likely to leverage chance than are confirmatory tests. What appears to be “what we learned” could be “what chance told us.” The point of making a registry available is not to have a priori hypotheses for all projects and findings; it is to clarify when there was one and when there was not. When it is a discovery, acknowledge it as a discovery.StatisticiansAs Tukey (1977) summarized in support of discovery: "Once upon a time statisticians only explored. Then they learned . . . to confirm a few things exactly, each under very specific circumstances. As they emphasized exact confirmation, their techniques inevitably became less flexible. The connection of the most used techniques with past insights was weakened. Anything to which a confirmatory procedure was not explicitly attached was decried as 'mere descriptive statistics,' no matter how much we had learned from it. (p. vii)" Discovery is critical for science because learning occurs by having assumptions violated. Strong narratives focusing on what was learned are useful communication devices, and simple disclosures of how it was learned are useful accuracy devices.Given that academic science is a largely public institution funded by public money, it is surprising that there is so little transparency and accountability for the research process. Beyond the published reports, science operates as a “trust me” model that would be seen as laughably quaint for ensuring responsibility and accountability in state or corporate governance. In some areas of science, however, it is understood that transparency in the scientific workflow underlies credibility and accuracy. 2013-08-02http://pps.sagepub.com/content/7/6/615.fullOwenAmburOwen.Ambur@verizon.net