Center for Open Science IMPACT REPORT 2020: Maximizing the impact of science togetherCenter for Open ScienceCOS_f50a94vc-3f02-1e10-b991-79167a64bc2aThe Center for Open Science is a 501(c)(3) nonprofit organization. We work to catalyze grassroots community efforts to change research norms and help provide solutions, such as the OSF, an open-source, public goods infrastructure, to help stakeholders recalibrate reward systems. Brian NosekExecutive Director and Co-founder, Center for Open ScienceResearch CommunityPartnershipCOS partners with stakeholders across the research community to advance the infrastructure, methods, norms, incentives, and policies shaping the future of research to achieve the greatest impact on improving credibility and accelerating discovery. Accelerated discovery, improved credibility_ee557eea-2e8c-11eb-a1d2-7209fa82ea00To increase the openness, integrity, and reproducibility of research._ee557fee-2e8c-11eb-a1d2-7209fa82ea00OpennessScienceReproducibilityInfrastructure, Tools & ServicesOffer open science infrastructure, tools, and services_ee5580a2-2e8c-11eb-a1d2-7209fa82ea001ResearchStrengthen research_ee558192-2e8c-11eb-a1d2-7209fa82ea001.1Strengthening research at each stage of the research lifecycle -- Research serves humanity most effectively when its outcomes are credible and reproducible. Credibility is maximized when rigor and transparency is applied during planning, conducting, reporting, and discovery of research... Sharing findings, data, code, and materials enables discovery for reuse and accessibility for review and extension. More than 260,000 researchers use OSF to collaborate and share their research, and about 4 million researchers discovered content on OSF for their own research in 2020 alone.1.1.1RegistrationsNumber_9ac7ec5e-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_1Unspecified2020-01-012020-12-3117K+ PLANNING - New registrations created on OSF170002020-01-012020-12-311.1.2FilesNumber_9ac7ee66-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_2Unspecified2020-01-012020-12-311M+ CONDUCTING - Number of new files shared publicly10000002020-01-012020-12-311.1.3PapersNumber_9ac7efd8-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_3Unspecified2020-01-012020-12-3118K+ REPORTING - Papers shared via Preprint services180002020-01-012020-12-311.1.4DownloadsNumber_9ac7f870-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_4Unspecified2020-01-012020-12-3123M+ DISCOVERY - Number of files downloaded230000002020-01-012020-12-31Infrastructure & SolutionsProvide open-source, public goods infrastructure and customizable solutions for groups to cultivate culture change in their own communities_ee55823c-2e8c-11eb-a1d2-7209fa82ea001.2Infrastructure for culture change -- Researchers and stakeholders around the world rely on trusted infrastructure and workflows to enable their efforts to conduct rigorous, transparent research. COS provides open-source, public goods infrastructure and customizable solutions for groups to cultivate culture change in their own communities.1.2.1Culture Change_9ac7fc80-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_5Groups cultivate culture change in their own communitiesStatus2020-01-012020-12-31[To be determined]2020-01-012020-12-31RegistriesEmpower research groups, funders, and journals to create research registries and support reproducible methods for their community of practice, awardees, or authors_ee5582dc-2e8c-11eb-a1d2-7209fa82ea001.3Branded Registries -- This year COS launched Branded Registries, a new interface built on the OSF Registries infrastructure that empowers any research group, funder, or journal to create a research registry and support reproducible methods for their community of practice, awardees, or authors. Each branded registry provides customized preregistration templates and workflows, allowing scientific communities to facilitate rigor among their researchers with the infrastructure tailored to their field... Branded Registries, and COS’s other infrastructure products, is one component of our integrated strategy for changing research cultures.1.3.1Registries_9ac7fe74-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_6Create research registriesStatus2020-01-012020-12-31[To be determined]2020-01-012020-12-311.3.2Methods_9ac7ffdc-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_7Support reproducible methodsStatus2020-01-012020-12-31[To be determined]2020-01-012020-12-31TransparencyElevate visibility of journal and funder transparency policies_ee558386-2e8c-11eb-a1d2-7209fa82ea001.3.1JournalsFundersMake it required -- TOP Factor elevates visibility of journal and funder transparency policies. Comparative data rewards progressive policymakers and encourages slower actors to join the reform movement. 1.3.1.1Rewards & Encouragement_9ac8036a-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_8Reward progressive policymakers and encourage slower actors to join the reform movementStatus2020-01-012020-12-31[To be determined]2020-01-012020-12-31IncentivesRealign incentives with Rigor and Transparency initiatives_ee558430-2e8c-11eb-a1d2-7209fa82ea001.3.2Make it rewarding -- Journals and funders collaborate to realign incentives with Rigor and Transparency initiatives using Registered Reports... Learn more about Rigor and Transparency initiatives for funders and publishers at osf.io/ktsur, or visit cos.io/rti to request more information.1.3.2.1Realignment_9ac80680-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_9UnspecifiedStatus2020-01-012020-12-31[To be determined]2020-01-012020-12-31Norms & RigorEnable communities to cultivate preregistration norms and improve rigor_ee5584da-2e8c-11eb-a1d2-7209fa82ea001.3.3Make it normative -- Custom branded registries enable communities to cultivate preregistration norms and improve rigor.1.3.3.1CommunitiesNumber_9ac80810-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_10Unspecified2020-01-012020-12-31[To be determined]2020-01-012020-12-31RegistrationStreamline the registration process_ee55858e-2e8c-11eb-a1d2-7209fa82ea001.3.4Make it easy -- OSF Registries streamlines the registration process with easy-to-use workflows. 1.3.4.1Workflows_9ac80a2c-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_11Provide easy-to-use workflowsEase of Use2020-01-012020-12-31[To be determined]2020-01-012020-12-31PreregistrationEnable preregistration_ee558642-2e8c-11eb-a1d2-7209fa82ea001.3.5Make it possible -- OSF enables preregistration.1.3.5.1Formats_9ac80bda-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_12UnspecifiedStatus2020-01-012020-12-31New registration formats this year broaden inclusion of research methodologies such as qualitative research, secondary data analysis, longitudinal investigations, and systematic reviews, and more.Broader2020-01-012020-12-31CommunitiesEnable open practices_ee5586ec-2e8c-11eb-a1d2-7209fa82ea002Research CommunitiesEnablementCommunities enabling open practicesNII JapanAdaptation & ScalingThe National Institute of Informatics (NII) consulted with COS to adapt OSF’s open source code to scale and launch the Japanese research data management platform GakuNin RDM (GRDM) to adhere to regulatory and compliance requirements for the region. The data management and preservation infrastructure is built using OSF, enhanced by customizations for integration with domestic scholarly platforms and Japanese institutional repositories that enable open sharing, discovery, rigor, and innovation among Japanese and Malaysian researchers. NII engineers and RDM operators collaborated with the open source development team at COS to consult on a local deployment of OSF, customizations, and training to support shared goals of openness, transparency and reproducibility of research around the globe.Yusuke KomiyamaNational Institute of InformaticsEndorsement“GRDM is continuously developing as an OSS for research promotion and research integrity based on Open Science Framework (OSF). From 2019 to 2020, 19 institutions have already adopted GRDM in the demonstration phase, with plans to start the production operation in 2021. NII RDC is also recommended for use in research data management in major competitive grants by the government and increases the number of institutions using the service.”Focused UltrasoundThe Focused Ultrasound Foundation launched FocUS Archive on OSF Preprints in 2017 to accelerate open research sharing in the field of ultrasound treatments and technology. But their commitment to accelerating scientific progress didn’t stop there. In late 2020, they will launch the Focused Ultrasound Foundation Collection on OSF Collections to aggregate the data, materials, and outcomes of their grantees’ work. The collection provides easy-to-use submission and discovery interfaces to foster sharing and reuse.Emily WhiteFocused UltrasoundEndorsement“The Focused Ultrasound Foundation embraces open science practices with our research community because we fundamentally believe implementation of those practices result in research products that have better rigor, reproducibility, and increase the threshold of quality research. Additionally, our use of OSF Preprints and OSF Collections not only help with the quality of the research, but accelerates the timeline this research is available to the larger community. Better discoverability. As a medical research funder, we’re always trying to figure out how to get from bench to bedside both as fast and as safe as possible, and open science practices helps us do that. It’s all about the patients.”Ghent UniversitySupportGhent University supports transparent and rigorous research among its institutional researchers with the OSF Institutions interface. Ghent researchers take advantage of OSF Institutions’ streamlined data management structure, preregistration workflows, collaboration features, and more to generate and share high-impact research. In addition to supporting almost 200 Ghent-affiliated users, over 800 projects, and 200 registrations, the university has partnered with COS to integrate one of the most popular cloud storage providers with OSF, OneDrive for Business, used by institutions and researchers around the world. This storage service will be available as an OSF storage add-on in Q1 2021 to all OSF users. Shared, open infrastructure dramatically extends the impact of infrastructure investments.Myriam MertensGhent UniversityEndorsement“Open Science is, and has been for many years, an important component of Ghent University's research strategy. It is also highly valued by a significant and ever growing segment of our research community. We became a member of OSF Institutions a couple of years ago to support our researchers in their efforts to share research workflows and outputs for validation and reuse purposes, and to help identify our own open science champions. Going forward, we want to continue to expand our OSF Institutions user base, in order to grow our organisation's open science community and further strengthen transparent, reproducible, and collaborative research practices across disciplines.”AfricArXivSupportAfricArXiv began as a pan-African cross-disciplinary preprint platform in 2018, and furthered its commitment in 2020 by launching a roadmap with new opportunities to support open, rigorous, and reproducible research practices. The new project encompasses training training events, collaborative initiatives for the African research diaspora, and incentives for greater openness and transparency in pan-African scholarship.Johanssen ObandaAfricArXivEndorsement“Through partnerships like the one established with COS, we hope to establish an independent, efficient, sustainable and decentralized research repository that serves as an interoperable platform for contributions by African researchers, and for all scientists who work in an African context, with the goal to increase discoverability of African research.”EGAPIntegrationCOS celebrated the integration of the Evidence in Governance and Politics (EGAP) registry with OSF Registries. The EGAP registry provides social scientists an interface to discover existing work, submit a registration of their own, or begin conducting rigorous research using the connected OSF infrastructure as a project workspace.Matthew LisieckiEGAPEndorsement"Over the past several years, the EGAP Registry has been gaining momentum and more awareness by researchers studying subjects related to governance, institutions, politics, and accountability. Simultaneously, OSF offers a flexible registration infrastructure that can support any registry and leverage OSF’s other features for the research life cycle. By making the two systems work together, we aim to reduce the time researchers spend entering duplicate information while increasing the sharing of accurate and transparent information on research projects."Open Scholarship Knowledge BaseKnowledge BaseOpen science advocates and volunteers around the world and across disciplines came together to compile, curate, edit, and review open education resources that ultimately launched the Open Scholarship Knowledge Base (OSKB) in August 2020. The volunteer-led initiative, backed by COS, is a growing body of education and training materials covering the what, why, and how of open scholarship.STEM Education Resource HubResource HubEducation research volunteers coalesced in June 2020 to begin work on the STEM Education Resource Hub. The STEM Hub provides education researchers with connection to all COS services and grassroots organizing to nurture the education community toward open, reproducible research. The Hub will catalyze innovation in education research through community-led events and support for collaboration on research, training, and culture change.Erin BuchananHarrisburg University of Science and TechnologyEndorsement“The Education group was formed to support students and teachers alike to find open access materials for their needs. Access to all types of materials is especially critical for countries that use internet blockades to limit information to their citizens. We welcome everyone to contribute and use the hub!”Many Labs 5CollaborationMore than 170 researchers collaborated on Many Labs 5, the final “Many Labs” project that comprised 11 papers and the entire Fall issue of Advances in Methods and Practices in Psychological Science. Led by Charlie Ebersole, a postdoc at the University of Virginia, the study investigated whether conducting formal peer review by experts in advance could improve replicability of earlier findings that mostly failed to replicate in the Reproducibility Project: Psychology. The evidence countered critics' claims that prior failures to replicate were due to low power studies that did not meet quality standards. COSGNMeta-NetworkingOver 120 open science grassroots communities came together online in 2020 to form the Community of Open Scholarship Grassroots Networks (COSGN) creating a meta-network of mostly volunteer networks with broad representation across regions, disciplines, and research scope. Join the community hub to collaborate and dialog with fellow open science grassroots peers working to disseminate research best practices.Leadership & SupportBack researchers with community leadership and support for reproducible practices_ee5587a0-2e8c-11eb-a1d2-7209fa82ea002.1ResearchersAdoptionResearchers are more likely to adopt reproducible practices when they are backed by community leadership and support. Learn about the groups, institutions, and funders partnering with COS to equip their researchers with open infrastructure, methods, and training.2.1.1Leadership & Support_9ac80d9c-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_13Provide community leadership and support to researchersStatus2020-01-012020-12-31[To be determined]2020-01-012020-12-31IncentivesReengineer incentives to support openness and transparency_ee558886-2e8c-11eb-a1d2-7209fa82ea003We shift the research culture toward greater reproducibility by partnering with research stakeholders to realign the norms, incentives, and policies that shape researcher behavior.JournalsRate the degree of openness and transparency of journal policies_ee558962-2e8c-11eb-a1d2-7209fa82ea003.1Taylor & FrancisVisibilityCOS acknowledges publishers like Taylor & Francis raising the visibility of open practices through efforts like open science badging, an initiative to incentivize and reward transparent behaviors such as data sharing, making materials publicly available, and preregistering research designs.JournalsRepresentationOver 400 journals are now represented on TOP Factor since its launch this year, with more than 200 to be added soon.TOP Factor -- TOP Factor rates the degree of openness and transparency of journal policies to provide authors with information and to encourage progression toward more assertive transparency policies by journal editors. COS recognizes the journals working to implement policies promoting transparency and reproducibility and alter dysfunctional publishing incentives that reward novelty over rigor.3.1.1ListingsNumber_9ac81058-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_14Unspecified2020-01-012020-12-31413 Journals listed at topfactor.org4132020-01-012020-12-313.1.2ImprovementsPercentage_9ac81260-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_15Unspecified2020-01-012020-12-31In a pilot test, being assessed via TOP Factor prompted 24% of journals listed to improve their score within 6 weeks. 242020-01-012020-12-31PreparednessAssess the preparedness of research communities for culture change_ee558a52-2e8c-11eb-a1d2-7209fa82ea003.2Bodo SternHHMIEndorsement“In 2019 the Center for Open Science (COS) conducted an open science survey of HHMI scientists on behalf of HHMI. This close collaboration was important to HHMI, because the survey results influenced the design and communication of HHMI’s new open access policy. The COS survey gave HHMI scientists a voice in shaping this important policy and contributed to a remarkably positive reception when the policy was announced on October 1, 2020.”The OSS measures opinions, behaviors, and perceived incentives around open practices such as open access, preprints, preregistration, replication and sharing data, materials, and code. The OSS is applicable across disciplines, regions, and career stage, enabling stakeholders to receive comparative data about the state of their community versus peer institutions or disciplines.3.2.1Survey_9ac8147c-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_16UnspecifiedStatus2020-01-012020-12-31COS unveiled the Open Scholarship Survey (OSS) to help funders, institutions, and other stakeholders assess the preparedness of their research communities for culture change. Unveiled2020-01-012020-12-31Rigor & TransparencyPursue culture change toward rigor and transparency_ee558b38-2e8c-11eb-a1d2-7209fa82ea003.3Julie SchaferFlu LabEndorsement“Flu Lab believes that addressing a challenge as formidable as influenza requires greater availability of information and collaboration. As funders, increasing the publication of null and negative results and replication studies is an important part of our efforts to promote transparency and open science principles in the influenza research community. We are pleased to partner with Center for Open Science, which shares our values and a willingness to experiment with new programs and approaches.”PublishersUsageUse COS Research Transparency Initiative (RTI) consulting services as they pursue culture change toward rigor and transparency. FundersUsageUse COS Research Transparency Initiative (RTI) consulting services as they pursue culture change toward rigor and transparency. Research GroupsUsageUse COS Research Transparency Initiative (RTI) consulting services as they pursue culture change toward rigor and transparency. NSFFundingAnd, with a recent award from NSF, COS is evaluating the effectiveness of these initiatives in spurring culture change.Research Transparency Initiatives3.3.1Design & Administration_9ac83858-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_17StatusThe RTI project assists stakeholders in designing and administering culture change initiatives, particularly partnerships between funders and journals to reduce publication bias, promote rigor and replication, and increase credibility.2020-01-012020-12-313.3.2Effectiveness_9ac83ab0-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_18Evaluate the effectiveness of these initiatives in spurring culture change.Status2020-01-012020-12-31[To be determined]2020-01-012020-12-31ResearchConduct evaluations and interventions to improve research _ee558c28-2e8c-11eb-a1d2-7209fa82ea004LabsEvaluationEvaluate science and the scientific processFundersEvaluationEvaluate science and the scientific processResearch GroupsEvaluationEvaluate science and the scientific processWe collaborate with labs, funders, and research groups to evaluate science and the scientific process. Studying how research is done surfaces deficiencies and spurs methodological innovation.ReplicabilityProvide evidence that transparent and rigorous procedures are associated with high replicability_ee558d68-2e8c-11eb-a1d2-7209fa82ea004.1High Replicability of Newly-Discovered Social-behavioral Findings is Achievable4.1.1Replicability_9ac83cd6-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_19UnspecifiedStatus2020-01-012020-12-31A prospective replication study of 16 novel experimental findings released as a preprint in September 2020 provides evidence that transparent and rigorous procedures are associated with high replicability.High2020-09-012020-09-30PreprintsIdentify signals servers can implement to increase the credibility of preprints_ee558e62-2e8c-11eb-a1d2-7209fa82ea004.2Credibility of preprints: an interdisciplinary survey of researchers -- Preprints services can accelerate scholarly communication and improve accessibility, but do researchers find them credible enough to use?4.2.1Signals_9ac83efc-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_20UnspecifiedStatus2020-01-012020-12-31A survey of 3,759 researchers across disciplines identified the signals preprint servers can implement—such as cues for available data, materials, and code—to help researchers better assess credibility and trust in preprints. Identified2020-01-012020-12-31MetascienceMature metascience as a disciplinary community and produce new mechanisms for collaboration and innovation_ee558f5c-2e8c-11eb-a1d2-7209fa82ea004.3Metascience CommunityCollaborationInnovationMetascience: the emerging field of research on the scientific process4.3.1Metascience_9ac842bc-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_21UnspecifiedStatus2020-01-012020-12-31Metascience continued to mature as a disciplinary community in 2020, and produced new mechanisms for collaboration and innovation, including the Metascience research collection and Metascience registry.Maturing2020-01-012020-12-31CultureEnact change in the research culture_ee55906a-2e8c-11eb-a1d2-7209fa82ea005ResearchersPromotionAll researchers and stakeholders play a part in maximizing scientific impact and changing research culture when they engage with and promote open and transparent behaviors, norms, and incentives across the research lifecycle.InstitutionsPolicymakersPublishersFundersDonorsFundingCOS could not accomplish its mission without the collective support of individual donors, philanthropic foundations, and research funders. Visit cos.io/supporters to learn more about the financial partners that furthered COS’s open science initiatives in 2020.Stewards of science for a better worldStrategiesFoster the creation of strategies for greater reproducibility_ee559236-2e8c-11eb-a1d2-7209fa82ea005.1Shifting the research culture toward greater reproducibility requires a comprehensive and progressive strategy at every tier.5.1.1Strategies_9ac8450a-2e96-11eb-b298-cd0afc82ea00PLACEHOLDER_22Create research reproducibility strategiesStatus2020-01-012020-12-31[To be determined]2020-01-012020-12-31_9ac84b40-2e96-11eb-b298-cd0afc82ea002020-11-24https://f.hubspotusercontent00.net/hubfs/6723653/Center%20for%20Open%20Science%202020%20Impact%20Report.pdf_9ac84f46-2e96-11eb-b298-cd0afc82ea00OwenAmburOwen.Ambur@verizon.net