PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0225883 PONE-D-19-18102 Research Article Research and analysis methodsResearch assessmentCitation analysis Research and analysis methodsScientific publishing Social sciencesPolitical science Science policy Science policyScience policy and economics Research and analysis methodsMathematical and statistical techniquesStatistical methodsInstrumental variable analysis Physical sciencesMathematicsStatisticsStatistical methodsInstrumental variable analysis Physical sciencesMathematicsStatisticsStatistical data Science policyScience and technology workforceCareers in researchScientists People and placesPopulation groupingsProfessionsScientists A study of the impact of data sharing on article citations using journal policies as a natural experiment The impact of data sharing on article citations http://orcid.org/0000-0003-2558-1609 Christensen Garret Data curation Formal analysis Funding acquisition Investigation Methodology Project administration Software Supervision Validation Visualization Writing – original draft Writing – review & editing 1 * Dafoe Allan Conceptualization Formal analysis Writing – review & editing 2 Miguel Edward Conceptualization Data curation Formal analysis Funding acquisition Methodology Project administration Writing – review & editing 3 Moore Don A. Conceptualization Data curation Formal analysis Investigation Methodology Project administration Resources Supervision Writing – original draft Writing – review & editing 3 Rose Andrew K. Conceptualization Formal analysis Investigation Methodology Validation Visualization Writing – original draft Writing – review & editing 3 U.S. Census Bureau, Washington, DC, United States of America University of Oxford, Oxford, England, United Kingdom University of California, Berkeley, California, United States of America Naudet Florian Editor University of Rennes 1, FRANCE

The authors have declared that no competing interests exist.

* E-mail: garret@berkeley.edu
18 12 2019 2019 14 12 e0225883 27 6 2019 14 11 2019 This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

This study estimates the effect of data sharing on the citations of academic articles, using journal policies as a natural experiment. We begin by examining 17 high-impact journals that have adopted the requirement that data from published articles be publicly posted. We match these 17 journals to 13 journals without policy changes and find that empirical articles published just before their change in editorial policy have citation rates with no statistically significant difference from those published shortly after the shift. We then ask whether this null result stems from poor compliance with data sharing policies, and use the data sharing policy changes as instrumental variables to examine more closely two leading journals in economics and political science with relatively strong enforcement of new data policies. We find that articles that make their data available receive 97 additional citations (estimate standard error of 34). We conclude that: a) authors who share data may be rewarded eventually with additional scholarly citations, and b) data-posting policies alone do not increase the impact of articles published in a journal unless those policies are enforced.

GC, EM Laura and John Arnold Foundation, grant number 040951. http://www.arnoldventures.org Publication made possible in part by support from the Berkeley Research Impact Initiative (BRII) sponsored by the UC Berkeley Library. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability All data files are available on the Open Science Framework: https://osf.io/pxdch/ and https://osf.io/cdt8y/.
Introduction

Verifiability and replicability are fundamental to science. The Royal Society’s motto “nullius in verba" (“take nobody’s word for it”) encourages scientists to verify the claims of others. By sharing data, scientists can increase the verifiability and credibility of their claims. Most academic journals and professional societies encourage researchers to share their data, but these are often informal recommendations; until recently, few journals required it.

The ease of posting data on the internet has lowered the cost of data sharing; accordingly, advocates of open science have argued that data posting should be standard practice [1], and a growing number of scientific journals have started requiring that authors publicly post their data. However, this requirement remains more the exception than the rule in many fields, and researchers have not routinely posted their data unless journals require them to do so [24].

Researchers give several reasons for their failure to post data. Some highlight costs to the individual, including the effort required, the potential for being scooped, and the risk of being shown to be in error. But there are also benefits from posting data. If research with posted data is more persuasive or believable, it might have greater impact. Moreover, data sets are often useful for analyses that the original author(s) did not think of or chose not to conduct; researchers typically cite an article if they utilize its posted data. It seems plausible that sharing the data used in an article would increase its citations. In fact, several papers across disciplines over the last decade consistently show a positive association between data sharing and citations: in disciplines as varied as cancer microarray trials [5], gene expression microarrays [6], astrophysics [7, 8], paleoceanography [9], and peace and conflict studies [10] data sharing has been shown positively associated with citations, and computational code sharing has been shown to be positively associated with citations in the image processing literature [11]. These studies range in size (from N = 85 to N>10,000) and find a range of estimates of the increase (from a low of 9% to a high of 69% with most between 20% to 40%). They typically focus on one discipline or subject area. To our knowledge none attempt to explicitly take advantage of a change in journal policy to estimate a treatment effect.

The objective of this paper is to determine whether sharing data for a research article results in more citations for the article, using changes to journal policies as a natural experiment. If sharing data does result in increased citations, then the private benefits to data sharing could popularize the practice and improve science. We are able to study this issue across a wide variety of disciplines, enhancing the generalizability of our findings. However, we are limited since we cannot ascertain why publicly posted data would or would not lead to additional citations, merely whether articles garner more citations. We are also limited to observational rather than experimental data; we discuss the methods employed to deal with this below.

Methods

A simple comparison of citations between articles published with and without posted data is difficult to interpret. Authors who post their data may be systematically different from those who do not, as they may choose to publish in different journals. We try to minimize this problem by focusing on journals which began, in principle, to require data posting. This natural experiment enables us to compare articles published before and after the change, exploiting plausibly exogenous variation in data availability caused by shifts in editorial policy.

Two separate and independent teams of researchers, both based at the University of California, Berkeley, serendipitously learned of each other’s plans to exploit the natural experiment caused by shifts in journal policy. One team (Moore and Rose, hereafter MR) collected a broad sample of articles and use within-journal variation in citations to keep the journal (and thus the research community) constant, focusing attention on the effect of the change in data sharing policy. The other team (Christensen, Dafoe, and Miguel, hereafter CDM) collected deeper, more detailed data on a smaller subset of articles, without knowing the results from the MR sample. The data collection processes of both teams appear in Fig 1, and the appendix explains the timing of the researchers’ interactions.

10.1371/journal.pone.0225883.g001 Flow diagram of sample selection.

Separately our two teams selected journals, then used all empirical articles in those journals as treated observations.

Broad analysis

To exploit the change in journal policies, MR systematically searched the top 250 scientific journals, as identified by SCImago (http://scimagojr.com), and identified all those that changed their policies to require data posting for published articles.

Following the MR pre-analysis plan (https://osf.io/pxdch/), we collected citation count data for empirical articles–those that analyze quantitative data–published immediately after a change in data posting policy, as well as analogous citation counts for articles published in the period before the regime change. The MR analysis examined, for each journal, 200 empirical articles or two years’ worth of articles (whichever was less) on either side of a policy change. Research assistants recorded the annual flow of new Web of Science citations received one, two, three, four, and five years post-publication for each of these articles. This enables us to compare the difference in citations for articles published in the same journal shortly before and after a new data sharing requirement.

To account for the possibility of events that influenced both the change in journal policy and citation rates that could bias estimates, we collected comparable data for two natural comparison sets. We gathered data on theoretical articles published in the same journals; since these do not use empirical data, their citations should be largely unaffected by any change in data posting policy. We also matched the 17 treatment journals (which began to require data sharing) to control journals (which did not) and collected comparable citation data for empirical articles published in these control journals.

We selected control journals using conventional one-to-one propensity score matching with replacement [12,13] from top-ranked journals that most closely match the treatment journals on SCImago criteria. Our objective was to identify control journals that did not require data posting, but that were otherwise as similar as possible to the treatment journals in terms of observable characteristics. Accordingly, we began with non-treatment journals—that never required data posting–from the same SCImago “Top 250” list of journals where we identified our treatment journals. We matched treatment to control journals using the six indicators used to create the SCImago list itself. These criteria appear on the Scimago website. The six variables are: a) the journal’s h-index; b) the total number of citable documents published in the journal’s last three years; c) citations per document over the last two years; d) references per document; e) the country where the journal was published; and f) the category of the journal. Since there are only two countries where our treatment journals were published, we created a binary variable for journals published in the UK, leaving US journals as the default. And since we only have a limited number of treatment journals, we consolidated journal category into eight areas: a) Biology; b) Ecology; c) Economics; d) Medicine; e) Molecular Biology; f) Multidisciplinary; g) Sociology and Political Science; and h) Miscellaneous.

After creating the data set, we created a binary variable, coded 1 for our treatment journals and 0 for all remaining journals (potential control journals). We then estimated a cross-sectional probit equation; the results are tabulated in our online supplement. After estimating the probit model, we then matched each of the treatment journals to a single control journal, using the closest possible journal, by predicted probit score within journal category. Sometimes this resulted in more than one treatment journal being matched to the same control journal. Both the control journals and the probit regression estimates themselves are freely available online (see https://osf.io/67c5z/). We are left with a set of 13 unique control journals along with 17 treatment journals; thus the MR analysis included data from a total of 30 distinct scientific journals. Appendix A provides the list of journals and more details on the data construction procedure.

We divide our data into citations for three types of research articles, as per above: a) empirical articles from the 17 treatment journals, our chief interest; b) theoretical articles from the treatment journals; and c) empirical articles from the 13 control journals. For each set of articles, we further split the data into articles published before and after the imposition of the data sharing policy. Control journals, by construction, do not experience any policy shift; we use the corresponding dates for the matched treatment journals.

Deep analysis

The CDM analysis focused on two of the 17 treated journals, namely, the American Economic Review (AER) and the American Journal of Political Science (AJPS), along with comparable control journals, the Quarterly Journal of Economics (QJE) and the American Political Science Review (APSR). The CDM team pre-registered their analysis in 2015 (see https://osf.io/hv97m/).

The results that follow are consistent with our pre-registered analysis plan, with the following three exceptions: First, we did not think to exclude articles that do not use data. (Results on the full sample, described in Appendix Tables B21 to B23, are generally weaker). Second, we planned to control for time using cubic functions of months since publication. The results presented in the paper instead use year-discipline fixed effects to more flexibly control for time, as can be seen in the equations below. (Models with the cubic function yield nearly identical results). Third, in September 2017 CDM expanded their analysis from the pre-registered field of political science (from which they had seen results) to include the economics portion of the project, which had been previously discussed, but not pre-registered; this was before seeing any results from the broad analysis conducted by MR.

Fig 1 shows the sample selection procedure. We chose AJPS and AER as treatment journals because of their prominence in their fields, and because they are known for relatively strong enforcement of data-posting requirements. We examined each article published in these four journals around the time of the change in journal policy in order to determine the availability of data (as well as the code) required to reproduce the article’s results.

Specifically, CDM collected data on articles published in the AER from 2001 to 2009, centering +/- four years around the 2005 change in data policy; articles from the QJE (which had no comparable policy change during this period) serve as a comparison group. We collected data from AJPS articles from 2006 through 2014, since the journal experienced enhanced data posting policy changes in both 2010 and 2012; APSR serves as the control journal to AJPS for identifying potentially confounding temporal effects.

CDM regressed total accumulated citations from Elsevier’s Scopus database as of November 2017 (unlike the MR analysis which used flow of citations) on article characteristics and journal policy using ordinary least squares linear regression (OLS). Because it remains possible that articles sharing data are different from those that do not in other dimensions, thus biasing a simple comparison, we also attempt to identify and control for other plausible predictors. We classified articles by research method (specifically, experimental, observational, and theoretical), by subject matter, and by the top institutional ranking of the authors (online Appendix B provides details on the data construction). Observations with missing data are excluded.

The OLS estimates are still liable to suffer from omitted variable bias (for example, if article quality is correlated with both data sharing and citations). To address this potential bias, CDM also use the changes in journal policies as instrumental variables for data availability, and examine the relationship between (instrumented) data availability and citations using two stage least squares (2SLS) which produces estimates which are biased but consistent [14,15]. The 2SLS model relies on two major assumptions: relevance (the data sharing policy is a strong predictor of data sharing) and the exclusion restriction (the data sharing policy only affects citations through data sharing, and not through any other channel). We explicitly test the relevance assumption and show evidence regarding the second below. We focus on models run with only those articles that use data, since articles without data are not subject to the policy. Estimation uses 2SLS regressions, per the equations below: availabilityi=α1+β1AERi+β2AJPSi+β3APSRi+β4Post2005i+β5Post2010i+β6Post2012i+β7AER*Post2005i+β8AJPS*Post2010i+β9AJPS*Post2012i+dtγtd+νi citationsi=α2+η1AERi+η2AJPSi+η3APSRi+η4Post2005i+η5Post2010i+η6Post2012i+η7availability^i+dtγtd+ui

Eq 1 refers to the first stage, which predicts availability of data based on journal and publication date, while Eq 2 uses predicted data availability to attempt to estimate the impact on number of total citations under assumptions we discuss below. We control for the journal (coefficient estimates β1, β2, β3), date of publication (whether it is after the policy: β4, β5, β6), and our instrumental variables are the interaction terms (with coefficient estimates β7, β8, β9). We also flexibly account for time since publication (and the fact that total citations mechanically increase over time) with fixed effects for each discipline-year combination (γtd).

To test the key assumption of this instrumental variables model (referred to as the exclusion restriction), we examine the observable characteristics of articles and assess whether these were affected by the editorial changes. If not, we can be more confident that unobservable characteristics are also largely uncorrelated with editorial changes. Accordingly, we classified articles by research method (experimental, observational, theoretical), by subject matter (American politics, public policy, international relations, comparative politics, political methodology, and political theory for political science, and Journal of Economic Literature topic codes for economics), and by the top institutional ranking of authors. We then employ the specification in Eq 1, but with these variables as the outcome.

Results Broad analysis

Fig 2 presents six event studies using the data from the MR sample, two for each of our three combinations of articles/journals. First, consider Panel A of Fig 2, which portrays the average citations for the 4,403 empirical articles published in our treatment journals. At the left of the graph, the mean flow of new citations (≈4.1) garnered in the first year after publication, for articles published before the policy change, is portrayed with a thin line along with a corresponding (+/- 2 standard deviation or 95%) confidence interval. Average annual citations rise with time; the right side of each panel shows new citations garnered in the fifth year after publication (again, only for articles published before the regime change). Panel B depicts the analogue for the 322 theory articles published in treatment journals, while Panel C on the right portrays that for the 1,024 empirical articles in control journals.

10.1371/journal.pone.0225883.g002 Citations per article appear in the top row, for articles published before treated journals’ adoption of a data-posting policy.

The bottom row shows the difference in citations per article following the policy change. Error bars show 95% confidence intervals.

Panel D presents the difference between average citations for empirical articles published before and after the regime change in the treatment journals, along with the associated 95% confidence interval. Our first main finding is that empirical articles published in the 17 treatment journals just before required data sharing receive citations comparable to those published just afterwards.

The divergence between average article citations before and after the policy shift is small, and is neither substantively nor statistically significant. These small differences characterize the main sample of empirical articles in the treatment journals (Panel D) as well as for the other two comparison sets of articles (Panels E and F), where null effects are expected (as placebo checks). Most strikingly, as Table 1 shows, there is no evidence that data sharing policies raise citations counts. However, this may be because such policies are not enforced; the existence of a data sharing policy does not necessarily require that data is actually shared.

10.1371/journal.pone.0225883.t001 Effect of policy switch, treating years after publication simultaneously.
Years since Publication Empirical Papers, Treatment Journals Theoretical Papers, Treatment Journals Empirical Papers, Control Journals
Estimator LS LS Poisson LS LS Poisson LS LS Poisson
Log(1+Cites) N Y N N Y N N Y N
1–5 -.20 (.42) -.03 (.05) -.02 (.06) .11 (.88) -.04 (.11) -.04 (.20) -1.98 (2.15) -.06 (.08) -.21 (.16)
1 -.08 (.59) -.00 (.07) -.00 (.10) .09 (.98) -.01 (.14) .03 (.22) -2.14 (2.51) -.05 (.08) -.20 (.20)
2 -.17 (.23) -.03 (.03) -.02 (.03) .18 (.48) -.02 (.05) .00 (.10) -.93 (1.00) -.03 (.04) -.10 (.07)
3 -.14 (.16) -.02 (.02) -.02 (.02) .10 (.27) -.00 (.03) -.01 (.06) -.64 (.68) -.02 (.02) -.07 (.05)
4 .04 (.14) .01 (.01) .01 (.02) .01 (.22) -.01 (.03) -.01 (.05) -.45 (.54) -.01 (.02) -.05 (.04)
5 -.06 (.10) -.01 (.01) -.01 (.01) -.06 (.17) -.02 (.02) -.03 (.04) -.43 (.41) -.02 (.02) -.05 (.03)

The table presents coefficients for a dummy variable (1 for after data posting required, 0 otherwise) estimated with least squares (LS) or Poisson; each column represents a separate regression. Robust standard error (clustered by journal) recorded in parentheses. Regression is annual citations/article. Controls included but not recorded: intercepts for each year elapsed since publication, fixed effects for journal and publication year, and log number of co-authors.

Deep analysis

We checked the availability of data and code for each of the 1660 empirical articles published in regular issues (i.e., excluding conference proceedings, anniversary retrospectives, and the like) of the four CDM journals. We find that data are posted for less than a quarter of articles published prior to the policy changes; this increases to approximately 70 percent for articles published immediately after the policy changes at AER and AJPS, as shown in Panels A and B respectively of Fig 3. Both journals experienced abrupt increases in data availability following the policy change, and it is this variation that we exploit in the analysis below. We analyzed sharing of both data and the statistical code necessary to replicate the analysis, as well as sharing of at least data; rates are similar for both, as are impacts on citations; for simplicity, we focus on data sharing alone (for more information, see Appendix B).

10.1371/journal.pone.0225883.g003 Percentage of articles with posted data appear in the top row.

The middle row shows the cumulative citation advantage for articles with posted data as of November 2017, and the bottom row shows citations in year five post-publication. AER = American Economic Review, QJE = Quarterly Journal of Economics, AJPS = American Journal of Political Science, and APSR = American Political Science Review.

Our second main finding, based on the CDM analysis, is that articles published after the data posting policy change in these two journals enjoy approximately 40 percent higher total citations than those published before, when using total accumulated citations after a relatively long time period (as of November 2017). This is visible in panels C and D of Fig 3, which present the average difference in total citations between the treatment and control journals for articles published in a given year, and the associated confidence intervals. Differences are regularized to zero in the year before the policy was first implemented in each treatment journal. The apparent trend in the pre-treatment period in economics is a potential concern. Some of the variation in the AER-QJE pre-treatment trend can be explained by a single paper [16], published in the AER in 2001, which has been cited two orders of magnitude more than most AER articles. We also confirm that citations in years one through five after article publication do not increase in the treatment journals (see Fig 3 panels E and F), mirroring the findings in Fig 2. The delay in the effect seems reasonable given that economics and political science articles receive less than half of their 15-year citation total in the first five years after publication [17], and in what follows we thus focus on Panels C and D.

The findings of Fig 3 indicate a potentially positive association between journal data posting policies and subsequent article citations. Our regression analysis (both OLS and 2SLS) leads to similar conclusions. CDM collected total citations for all articles in these four journals in November 2017. The time horizon varies from three to sixteen years, since articles are published between 2001 and 2014. The main OLS regression is shown in Table 2, with strong positive associations between data sharing and citations, after controlling for journal, and year-discipline fixed effects as described above. Note that in this table, the sample descriptions for each column listed as “Data-Only” imply using only articles that use data. “Data-NoPP” implies a further limitation of removing the Papers & Proceedings conference and similar anniversary-style issues. “Data-Econ” or “Data-NoPP-Econ” imply limiting to only data articles from economics. For the specifications with both disciplines in the first and second columns, the coefficient for data sharing ranges from 52.96 (standard error 9.38) to 64.15 (standard error 7.61) additional citations.

10.1371/journal.pone.0225883.t002 OLS estimates of citations and data sharing.
VARIABLES (1)Citations (2)Citations (3)Citations (4)Citations
Data Available 64.15***(7.61) 52.96***(9.38) 93.47***(11.42) 87.23***(16.75)
AER -130.05***(10) -100.36***(12.08) -130.59***(11.93) -107.77***(15.96)
AJPS -189.75***(21.59) -160.18***(25.36)
APSR -166.55***(22.31) -138.07***(26.1)
Observations 2,210 1,660 1,467 920
R-squared 0.18 0.2 0.15 0.12
Year-Discipline FE Yes Yes Yes Yes
Sample Data-Only Data-NoPP Data-Econ Data-NoPP-Econ
Mean Dep. Var. 99.03 116.1 122.8 167.4

Standard errors in parentheses

*** p<0.01,

** p<0.05,

* p<0.1

Table features heteroskedasticity-robust linear regressions of article citations on whether an article has shared data. All regressions included year-discipline fixed effects. The sample for each column is defined as follows: “Data-Only” uses only articles that use data. “Data-NoPP” further removes the Papers & Proceedings conference and similar anniversary-style issues. “Data-Econ” or “Data-NoPP-Econ” limits to only data articles from economics.

Results from the 2SLS regression also show large positive effects from data sharing, as shown in Table 3. F-statistics from the first stage of these regressions are over 20 in the specifications that include both disciplines. The table shows published articles with posted data enjoyed an increase of 97.04 (standard error 34.12) to 109.28 (standard error 41.15) total citations over a mean of approximately 100, suggesting nearly a doubling.

10.1371/journal.pone.0225883.t003 2SLS estimate of data sharing and citations.
VARIABLES (1)Citations (2)Citations (3)Citations (4)Citations
Data Available 97.04***(34.12) 109.28***(41.15) 672.53***(259.61) 234.55***(86.28)
AER -130.33***(15.91) -112.57***(23.59) -142.14***(23.51) -139.67***(32.07)
AJPS -204.19***(26.1) -181.21***(26.71)
APSR -173.91***(20.57) -147.74***(20.72)
Post-Mar 2005 -24.78(29.22) -16.52(32.28) 72.03(78.05) -28.14(36.52)
Post-Oct 2010 -32.34(22.12) -35.64(24.36)
Post-July 2012 -8.54(10.19) -8.86(11.31)
Observations 2,210 1,660 1,467 920
R-squared 0.17 0.18 -1.34 0.04
Year-Discipline FE Yes Yes Yes Yes
Sample Data-Only Data-NoPP Data-Econ Data-NoPP-Econ
Mean Dep. Var. 99.03 116.1 122.8 167.4
F Stat 20.14 32.94 9.738 48.03

Robust standard errors in parentheses

*** p<0.01,

** p<0.05,

* p<0.1

Table features 2SLS regressions of article citations on whether an article has shared data, with journal policy requirements as an instrument for data sharing. All regressions included year-discipline fixed effects. The sample for each column is defined as follows: “Data-Only” uses only articles that use data. “Data-NoPP” further removes the Papers & Proceedings conference and similar anniversary-style issues. “Data-Econ” or “Data-NoPP-Econ” limits to only data articles from economics and drops political science.

However, when we test the exclusion restriction assumption of our 2SLS model, the timing of the change in editorial policy regarding data posting does appear to have been associated with a modest change in the types of articles published in the AER (relative to the QJE). Experimental articles increased by 9 percentage points and observational empirical studies decreased by 12 percentage points around the time of the policy change; both estimates have standard errors of 5 percentage points and are thus significant at the 10 percent and 5 percent significance level, respectively. Submissions from the most prestigious institutions increased relative to those from only slightly less prestigious universities by as much as 24 percentage points, statistically significant at the one percent level. We observe no such changes in article or author characteristics at AJPS at the time of its data policy change.

Discussion

Our main results seem to indicate that for most journals, data sharing policies do not lead to increased citations in the five years following publication. The same is true of theory articles published in the treatment journals, and empirical articles in the control journals, as shown in Panels E and F of Fig 1 respectively. However, sharing data does appear to lead to increased citations over a longer time period at the American Economic Review, as shown in Tables 2 and 3. Given the results of our tests of the exclusion restriction in Table 4, however, we cannot completely rule out the possibility that changes in the types of authors or articles contributed to higher rates of data sharing following the new data policies at the AER.

10.1371/journal.pone.0225883.t004 Tests of the 2SLS exclusion restriction.
VARIABLES (1)Experimental (2)Observational (3)Top 1 (4)Top 10 (5)Top 20 (6)Top 50
AER post-2005 Policy 0.09*(0.05) -0.12**(0.05) 0.24***(0.06) -0.13***(0.05) -0.06(0.05) -0.01(0.05)
AJPS post-2010 Policy 0.13*(0.07) -0.12(0.08) 0.03(0.09) 0.05(0.07) 0.12(0.08) 0.07(0.08)
AJPS post-2012 Policy -0.13*(0.08) 0.11(0.08) -0.04(0.09) 0.03(0.07) -0.02(0.08) 0.06(0.09)
Observations 1,658 1,658 1,653 1,653 1,653 1,653
R-squared 0.02 0.03 0.11 0.03 0.01 0.09
Year-Discipline FE Yes Yes Yes Yes Yes Yes
Sample Data-NoPP Data-NoPP Data-NoPP Data-NoPP Data-NoPP Data-NoPP
Mean Dep. Var. 0.139 0.842 0.266 0.117 0.155 0.195

Standard errors in parentheses

*** p<0.01,

** p<0.05,

* p<0.1

Table features OLS regressions of article characteristics on whether the article was in a journal after a data sharing policy requirement was enacted. The sample in all specifications (described as “Data-NoPP”) uses only articles with data and further removes the Papers & Proceedings conference and similar anniversary-style issues.

What can explain the null results for most journals? One obvious concern is the consistency with which journals actually enforced data-posting policies. There is ample evidence that journal data posting policies are enforced weakly, if at all. We found that rates of data availability for empirical articles published after journals adopted data-sharing policies differ widely between journals, from 0% to 83%, with a mean of 35% (see Table A17 of supplemental online materials). Examining the effect of actual data availability at the level of the individual article clarified the results. As shown by the increase in the fraction of articles that share data (Panels A and B in Fig 3), the policies seem to be at least partially enforced for these journals: the first stage in the 2SLS regression is strong, with F-statistics generally over 20 (Table 2). Confirming the presence of posted data is difficult, requiring systematic internet searches across many possible locations. However, it appears that journals with strictly enforced policies may see increased citations.

One limitation of our study is that the analysis does not reveal precisely why publicly posting data might help an article garner citations. Future research could examine whether shared data is an indicator of the reliability of results, or an input that is used in subsequent research. Still, regardless of the underlying mechanisms, sharing data represents the provision of a public good that benefits the scientific enterprise. It allows scientists to enact the Royal Society’s motto and examine evidence for themselves, moving closer to the ideal of open and collaborative scientific inquiry [18].

The two main results, taken together, indicate that it is not sufficient for scientific journals merely to announce a data sharing requirement. This appears generalizable across a wide group of disciplines, given the range of fields covered by the MR analysis. Without diligent enforcement, a toothless journal data policy appears to produce the same result as no policy at all; few authors post their data. But even if de jure journal policy does not guarantee data sharing, our results indicate that public data sharing can eventually yield private benefits for scholars, in the form of enhanced citations, which provide meaningful de facto incentives to share scientific data.

Supporting information STROBE statement—Checklist of items that should be included in reports of observational studies.

(DOCX)

<p>(DOCX)</p> </caption> </supplementary-material> </sec> </body> <back> <ack> <p>The authors thank the editor Florian Naudet, and two referees for useful suggestions. Rose thanks the Central Bank of Barbados and the National University of Singapore for hospitality during the course of this research. Any opinions and conclusions expressed herein are those of the authors and do not necessarily reflect the views of the U.S. Census Bureau.</p> </ack> <ref-list> <title>References Miguel E, Camerer C, Casey K, Cohen J, Esterling KM, Gerber A, et al. Promoting transparency in social science research. Science. 2014;343(6166):301. doi: 10.1126/science.1245317 24385620 Tenopir C, Allard S, Douglass K, Aydinoglu AU, Wu L, Read E, et al. Data sharing by scientists: practices and perceptions. PloS One. 2011;6(6):e21101. doi: 10.1371/journal.pone.0021101 21738610 Vines TH, Albert AYK, Andrew RL, Débarre F, Bock DG, Franklin MT, et al. The availability of research data declines rapidly with article age. Curr Biol. 2014;24(1):947. doi: 10.1016/j.cub.2013.11.014 24361065 Christensen G, Miguel E. Transparency, reproducibility, and the credibility of economics research. J Econ Lit. 2018;56(3):92080. Piwowar HA, Day RS, Fridsma DB. Sharing detailed research data is associated with increased citation rate. PloS One. 2007;2(3):e308. doi: 10.1371/journal.pone.0000308 17375194 Piwowar HA, Vision TJ. Data reuse and the open data citation advantage. PeerJ. 2013;1:e175. doi: 10.7717/peerj.175 24109559 Henneken EA, Accomazzi A. Linking to data-effect on citation rates in astronomy. ArXiv Prepr ArXiv11113618. 2011; Drachen T, Ellegaard O, Larsen A, Dorch S. Sharing data increases citations. Liber Quarterly. 2016 Aug 15;26(2). Sears, JRL (2011), Data sharing effect on article citation rate in paleoceanography, Abstract IN53B-1628, presented at 2011 Fall Meeting, AGU, San Francisco, Calif., 5–9 Dec. Gleditsch NP, Metelits C, Strand H. Posting your data: Will you be scooped or will you be famous? International Studies Perspectives. 2003;4(1):8997. Vandewalle P. Code sharing is associated with research impact in image processing. Computing in Science & Engineering. 2012 Jul;14(4):427. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):4155. Dehejia RH, Wahba S. Propensity score-matching methods for nonexperimental causal studies. Rev Econ Stat. 2002;84(1):15161. Imbens GW, Angrist JD. Identification and Estimation of Local Average Treatment Effects. Econometrica. 1994 Mar 1;62(2):467. Angrist JD, Imbens GW, Rubin DB. Identification of causal effects using instrumental variables. Journal of the American statistical Association. 1996 Jun 1;91(434):44455. Acemoglu D, Johnson S, Robinson JA. The colonial origins of comparative development: An empirical investigation. Am Econ Rev. 2001;91(5):1369401. Galiani S, Gálvez RH. The life cycle of scholarly articles across fields of research. NBER Work Pap. 2017;23447. Merton RK. A note on science and democracy. J Leg Pol Soc. 1942;1:115. 10.1371/journal.pone.0225883.r001 Decision Letter 0 Naudet Florian Academic Editor 2019 Florian Naudet This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

26 Jul 2019

PONE-D-19-18102

Does Data Sharing Increase Citations?

PLOS ONE

Dear Dr. Christensen,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I would like to thank here the two reviewers who assessed your manuscript as fast as possible.

One reviewer was somewhat enthusiast and asked for major revisions. He made very precise comments. Your answers to these comments will be all the more important.

The second reviewer recommended to reject the paper because the overall presentation of your manuscript was inadequate and unclear. He asked for an extensive revision following current reporting guidelines (please select the more appropriate for your paper from guidelines from the EQUATOR network). And I must say that I agree with him. He proposed to reject the paper and suggested a resubmission of an extensively amended version. I rather think that asking for “major revisions” would be a better compromise. Reporting research at the highest standards is an important aspect of research reproducibility and this point should be consequently improved before any decision on your paper. I recognize that there are no perfect guidelines for your study design but some may be relevant (PRISMA for a flow diagram of identified papers / STROBE for the handling of observational data etc.). However, you must be informed that a decision of major revision does not imply any commitment to accept your manuscript unless your alterations meet the standards of the review. In the present manuscript, revisions are expected to be extensive.

We would appreciate receiving your revised manuscript by Sep 08 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1) Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2) Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a carefully planned and performed study dedicated to the impact of data sharing on citations, which should be published. Looking at three types of research articles (empirical papers from 17 treatment journals), theoretical papers from treatment journals and empirical papers from 13 control journals, the first part of the analysis shows that empirical papers published in the 17 treatment journals just before required data sharing receive citations comparable to those published afterwards. The results are explained by the inconsistency with which journals actually enforced data posting policies. Because data availability is difficult to assess, analysis was focussed on 2 treatment journals in part 2 of the study. Here again, citations in year 1 through 5 after article publication do not increase in the treatment journals. With the argument of delay of citations for economics and political science articles, the authors then focus on total citations accumulated as of November 2017, covering a time horizon between 3 and 16 years. The authors found, using 2 stage least squares regression, that there was nearly a doubling of citations. In order to interpret this positive result correctly, several questions should be answered:

Pre-analysis plan

a) In the data sharing and citations pre-analysis plan, which was published in 2015, data are used from all articles from 2006 through 2014 in AJPS and from APSR as control group. In the second part of the manuscript the authors focused instead on 2 treated journals (AER, AJPS) along with comparable control journals (QJE, APSR), The authors should explain why this extension of focus was performed.

b) Table A14 in the supplement presents individual results when β is estimated using data on empirical papers from a single journal. Only 2 journals (AER, JLE) deliver consistently positive estimates of beta at conventional significance levels. AER was selected for the second part of the study. Could it be that the decision for having a look at AER was made after this first analysis was performed?

c) In the first part of the study citations 1-5 years after publication are considered and in the second part 5-year citation advantage both with negative statistical results. In part 2, the analysis is extended by using total citations accumulated at November 2017, which may cover longer periods. The reviewer could not find information whether the analysis period was specified in advance or not (1-5 years following publication or total citations accumulated at November 2017).

In summary, it is not totally clear to the reviewer, whether the 2nd part of the study followed the pre-analysis plan and if not, where it deviates. This may have influence on the interpretation of statistical significance (alpha error).

Regression model and bias

Citations in year 1 through 5 after article publication do not increase in the treatment journals (AER, AJP); see figure 2, panel E, F). If the cumulative citation advantage (Nov. 2017) is considered, the 2 journals enjoy approximately 40% higher total citations (see figure 2, panel C,D).

a) In figure 2, panel D, the 95% CI is extremely small for 2014 and AJPS Cumulative Citation Advantage (Nov 2017). Apart from this value, the AJPS cumulative citation advantage seems to be small. The authors should explain this.

b) The apparent trend in the pre-treatment period in economics is a potential concern for the authors and explained by a highly cited AER paper. The reviewer does not understand this argument because AER – QJE is negative in the pre-treatment period.

c) From the 2-stage least squares regression it is concluded that published articles with posted data enjoyed an increase of 90 to 125 citations over a mean of approximately 100, suggesting nearly a doubling of citations. Here the question is how good the goodness-of-fit of the model is. The R-squared values are not very impressive. Residual plots to determine whether the coefficient estimates are biased could not be found in the supplementary material. The authors should demonstrate the adequacy of the model fit.

Reviewer #2: Dear authors,

below you can find comments that might improve your manuscript in different perspectives. Please enface my comments as a way to improve your piece of research and I will be more than happy to re-appraise it in a more suitable version.

a) This is a methodological paper and reporting guidelines for methodological research is under discussion. However, you may borrow some items (or even in an intuitive manner) from reporting guidelines from other designs to better guide your writing. Your manuscript in the current form is poorly written and unidentifiable, what is even more crucial for a paper that deals with a question related, at least in some level, to science reproducibility. For example, a simple item like the title does is flawed - it does not indicate if is it an original experiment, a letter, an editorial etc. Standards of reproducible reporting should be addressed (please check www.equator-network.org) and some items might be included accordingly to allow the reproducibility of your paper - title, structured abstract, primary and secondary outcomes, eligibility criteria, search strategy (or the deep description of the approach to address conveniently the journals) and etc. You've also waste too much words with non-informative sentences. Please go straight to the point. Avoid also the use of rhetorical sentences and the excess of latim expressions (although suitable at some points).

b) Structure your manuscript. Methods, Results and Discussion are completely unorganized. I don't care about the inversion of results and discussion in terms of the length (so, the fact of your discussion was tiny was not a problem for me), if the journal permits. However, they should be informative and address the point to why they were created originally - and you've left a lot of discussion informations in the results section. As for the methods part, this is critical. I must confess I went lost several times when reading your methods due to the way you presented this. It does not favour you neither the reader. Please think careful about it. We need to acknowledge sharply the waste in research nowadays and I don't think your research is waste.

c) In your methods, I'm very concerned in the way you've designed your experiment. Propensity scores may balance non-randomized studies at baseline, however, caveats are widely known. Beyond the fact the description of your control group and your propensity score model (I don't know for what variables you've adjusted) are poorly written, I'm hesitant if it is the best method to match groups for your experiments, if is it possible. Given the data on citations related to data-sharing is very naive, I don't know if I would rather prefer to first describe the rate of citations instead of making a comparison. Also, the rate of citation for papers, doesn't matter the data-sharing regimen, is usually low within the first years when there is a salient amount of citations; and, if linear, we would have a decent number only after years. Therefore, you may incur in a comparison with a low frequency of events (with or without inferential analysis) and thus I re-affirm I'm not convinced this is the time to compare citations, but yes for a description. In a future, I would say a comparison within the same journal might be more appropriate, once one will always need to deal with confounding factors that would be difficult to input in a model like the merit of a paper (which makes it more cited), publication bias, gender bias, geographic bias and others. As a last mention, please be very detailed with your control group. I couldn't follow it.

d) The rationale seems weak to me. Do you really think the perception of integrity might improve citations, at least nowadays? I mention the "thinking" because we don't have empirical evidence to support this. I know for a given topic when still under investigated, we have little evidence to support our kick-offs, but I don't know if is it the critical point, or even if data-sharing might incur in the augment of citations (unless for the use of the data, as you acknowledged in your paper). Citations are a multidimensional approach and a multivariable-explained phenomenon. Anecdotally, I'm not convinced data-sharing may play a role in citations within the myriad of variables - if researchers nowadays didn't perceive the value of data-sharing in various scenarios, I would say that an association is still harder (although not implausible by nature). Even, as a data-sharing researcher, I think data sharing should be observed as a way to reduce the waste in research; to improve the reproducibility and verifiability of a given finding; to augment the utility of a finding; and an ethical obligation of every single researcher regardless of the experimental addressed designed - what you brilliantly did. In time, congratulations by the way you shared the data. I didn't re-analyze your findings, but it is clear your compliance with reproducibility standards in that way.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Lucas Helal

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

10.1371/journal.pone.0225883.r002 Author response to Decision Letter 0 Submission Version 1

31 Aug 2019

Reviewers,

We thank you for your careful review and provide specific responses below. Reviewer comments are repeated with our responses below. This information is also included (with formatting) in our uploaded response letter to the editor.

Reviewer #1: This is a carefully planned and performed study dedicated to the impact of data sharing on citations, which should be published. Looking at three types of research articles (empirical papers from 17 treatment journals), theoretical papers from treatment journals and empirical papers from 13 control journals, the first part of the analysis shows that empirical papers published in the 17 treatment journals just before required data sharing receive citations comparable to those published afterwards. The results are explained by the inconsistency with which journals actually enforced data posting policies. Because data availability is difficult to assess, analysis was focussed on 2 treatment journals in part 2 of the study. Here again, citations in year 1 through 5 after article publication do not increase in the treatment journals. With the argument of delay of citations for economics and political science articles, the authors then focus on total citations accumulated as of November 2017, covering a time horizon between 3 and 16 years. The authors found, using 2 stage least squares regression, that there was nearly a doubling of citations. In order to interpret this positive result correctly, several questions should be answered:

Pre-analysis plan

a) In the data sharing and citations pre-analysis plan, which was published in 2015, data are used from all articles from 2006 through 2014 in AJPS and from APSR as control group. In the second part of the manuscript the authors focused instead on 2 treated journals (AER, AJPS) along with comparable control journals (QJE, APSR), The authors should explain why this extension of focus was performed.

There was internal debate from the beginning amongst the authors involved in this piece of the project (Christensen, Dafoe, Miguel) about whether to add the additional journals (AER and QJE). The desire to collect the AER/QJE data was always present, but it was initially unclear how feasible the data gathering process would be, especially because it was Christensen’s first time managing a team of undergraduate researchers. The AER/QJE data was not gathered until after authors viewed preliminary results from the AJPS/APSR sample. Conservative readers should be aware that the addition of the AER/QJE sample was not pre-specified. We now explicitly mention this point in the revised version of the paper.

b) Table A14 in the supplement presents individual results when β is estimated using data on empirical papers from a single journal. Only 2 journals (AER, JLE) deliver consistently positive estimates of beta at conventional significance levels. AER was selected for the second part of the study. Could it be that the decision for having a look at AER was made after this first analysis was performed?

This portion of the analysis was conducted independently by separate teams (Moore, Rose responsible for A14; Christensen, Dafoe, Miguel, responsible for adding the AER/QJE sample after having seen AJPS/APSR results, but before having seen any results of any sort from the Moore, Rose team.) We thoroughly agree that the timing of our studies is both sub-optimal and confusing to the reader, but the research projects truly did stumble upon each other serendipitously mid-stream, so a perfect combination was impossible. We hope that we are now fully transparent on this issue, and that our re-written Introduction and Methods sections clearly explain this.

c) In the first part of the study citations 1-5 years after publication are considered and in the second part 5-year citation advantage both with negative statistical results. In part 2, the analysis is extended by using total citations accumulated at November 2017, which may cover longer periods. The reviewer could not find information whether the analysis period was specified in advance or not (1-5 years following publication or total citations accumulated at November 2017).

As described above, the two separate teams began the project independently, and the data they collected differed in some important ways, including the timeframe over which citations are considered. Christensen, Dafoe, and Miguel always intended to analyze citations accumulated at a single point in time. Though this was not explicitly stated in our (admittedly too-brief) pre-analysis plan, the regression functional form we proposed in that 2015 document called for controlling flexibly for time since publication, which is important for data collected at one point in time (for articles published in varying years) but would be superfluous if one uses a flow measure of citations.

In summary, it is not totally clear to the reviewer, whether the 2nd part of the study followed the pre-analysis plan and if not, where it deviates. This may have influence on the interpretation of statistical significance (alpha error).

As noted above, we now clarify at various points of the revised paper where the analysis goes beyond the pre-analysis plan, in the interest of transparency. We hope that this helps address the referee’s concern.

Regression model and bias

Citations in year 1 through 5 after article publication do not increase in the treatment journals (AER, AJP); see figure 2, panel E, F). If the cumulative citation advantage (Nov. 2017) is considered, the 2 journals enjoy approximately 40% higher total citations (see figure 2, panel C,D).

a) In figure 2, panel D, the 95% CI is extremely small for 2014 and AJPS Cumulative Citation Advantage (Nov 2017). Apart from this value, the AJPS cumulative citation advantage seems to be small. The authors should explain this.

Thank you for bringing up this issue. The explanation has to do with the amount of variation in total citations in the AJPS articles by 2017, given that they were published only in 2014. As the overall mean and standard deviation of this outcome measure are small overall (as there has been less time for articles to accumulate citations), this leads to tighter confidence intervals. (In the limit, if we had examined citations a mere month, say, after publication of articles, few would have any citations and the confidence interval would be even smaller.) We hope that this provides a logical explanation for the pattern you observed.

b) The apparent trend in the pre-treatment period in economics is a potential concern for the authors and explained by a highly cited AER paper. The reviewer does not understand this argument because AER – QJE is negative in the pre-treatment period.

This has to do with the observation for papers published in 2001, which includes one extremely cited AER paper. As you can see, in 2001 the AER – QJE difference appears to be much more positive relative to the difference in 2002 and 2003, say. Had it not been for the outlier paper published in 2001 in the AER, the AER – QJE difference would have been more stably negative in the pre-period (at least during 2001 through 2003) than it currently appears to be. Of course, there is some further unexplained variation, including the apparent rise in AER citations for papers published in 2004 (compared to 2002 and 2003), which we admittedly cannot fully account for, although some of the year-to-year variation is also presumably being driven by random sampling variation, given the relatively small number of articles published each year in a journal.

c) From the 2-stage least squares regression it is concluded that published articles with posted data enjoyed an increase of 90 to 125 citations over a mean of approximately 100, suggesting nearly a doubling of citations. Here the question is how good the goodness-of-fit of the model is. The R-squared values are not very impressive. Residual plots to determine whether the coefficient estimates are biased could not be found in the supplementary material. The authors should demonstrate the adequacy of the model fit.

Thank you for raising this issue. We have two responses. Our first is that we generated the goodness-of-fit residual plot figure that you propose, and it is included here below. As you can see, and as you guessed, the overall goodness-of-fit is not very high for this model. This figure also appears in the supplementary material as Figure B6.

Our second response is that this approach is not particularly meaningful for the main instrumental variable (IV) two-stage least squares method that we employ in this analysis. It has been shown in econometrics that standard goodness-of-fit measures, like the R-squared and associated metrics, are not meaningful for assessing goodness of fit in IV models. For a well-known discussion, please see: Pesaran, M., & Smith, R. (1994). “A Generalized R^2 Criterion for Regression Models Estimated by the Instrumental Variables Method.” Econometrica, 62(3), 705-710. doi:10.2307/2951666. More broadly, in our view model fit is not central to the exercise we are carrying out: our goal is to obtain unbiased estimates of a particular coefficient, not to create a highly predictive overall model. Goodness-of-fit would matter more for scholars with a different goal than what motivates our paper.

Reviewer #2: Dear authors,

below you can find comments that might improve your manuscript in different perspectives. Please enface my comments as a way to improve your piece of research and I will be more than happy to re-appraise it in a more suitable version.

a) This is a methodological paper and reporting guidelines for methodological research is under discussion. However, you may borrow some items (or even in an intuitive manner) from reporting guidelines from other designs to better guide your writing. Your manuscript in the current form is poorly written and unidentifiable, what is even more crucial for a paper that deals with a question related, at least in some level, to science reproducibility. For example, a simple item like the title does is flawed - it does not indicate if is it an original experiment, a letter, an editorial etc. Standards of reproducible reporting should be addressed (please check www.equator-network.org) and some items might be included accordingly to allow the reproducibility of your paper - title, structured abstract, primary and secondary outcomes, eligibility criteria, search strategy (or the deep description of the approach to address conveniently the journals) and etc.

Thank you for this important suggestion. As background on the original submission, reporting guidelines are rarely used in our home disciplines in the social sciences (namely, economics, political science, psychology) but we agree that the proposed re-organization will add value and make the article more accessible to scholars across fields, which is our intention. As a result, in the revised version of the paper we have changed the title (along the lines you suggest), have reformatted the paper, and added a flow diagram and text explaining sample determination, per the STROBE guidelines (which appear relevant, if not perfect, for our observational analysis; more information on the STROBE guidelines can be found here: http://www.equator-network.org/reporting-guidelines/strobe/). As you can see, the revised title is: “A Study of the Impact of Data Sharing on Article Citations using Journal Policies as a Natural Experiment”.

You've also waste too much words with non-informative sentences. Please go straight to the point. Avoid also the use of rhetorical sentences and the excess of latin expressions (although suitable at some points).

Thank you for your guidance. We have substantially reduced the word count in the Introduction. The Latin phrases (three in total) are famous quotes and common legal phrases that we feel are the most direct way to express our point; if you see alternatives to some or all of them, we are very open to considering them. If there are other specific wasteful or non-informative sentences that you think remain in the revised version, we would be delighted to modify or cut them in the interest of making the article more direct and parsimonious.

b) Structure your manuscript. Methods, Results and Discussion are completely unorganized. I don't care about the inversion of results and discussion in terms of the length (so, the fact of your discussion was tiny was not a problem for me), if the journal permits. However, they should be informative and address the point to why they were created originally - and you've left a lot of discussion informations in the results section. As for the methods part, this is critical. I must confess I went lost several times when reading your methods due to the way you presented this. It does not favour you neither the reader. Please think careful about it. We need to acknowledge sharply the waste in research nowadays and I don't think your research is waste.

We have sought to clearly demarcate the paper into the four traditional sections, consistent with the reviewer’s requests. This includes major changes in the Methods and Results sections, to make sure that methodological issues are focused in the appropriate section. We have also moved some additional text from Results to Discussion, where this seems appropriate. If our changes are insufficient, we remain open to doing more to improve the structure of the paper, and in particular to make the Methods section clearer for the reader.

c) In your methods, I'm very concerned in the way you've designed your experiment. Propensity scores may balance non-randomized studies at baseline, however, caveats are widely known. Beyond the fact the description of your control group and your propensity score model (I don't know for what variables you've adjusted) are poorly written, I'm hesitant if it is the best method to match groups for your experiments, if is it possible. Given the data on citations related to data-sharing is very naive, I don't know if I would rather prefer to first describe the rate of citations instead of making a comparison. Also, the rate of citation for papers, doesn't matter the data-sharing regimen, is usually low within the first years when there is a salient amount of citations; and, if linear, we would have a decent number only after years. Therefore, you may incur in a comparison with a low frequency of events (with or without inferential analysis) and thus I re-affirm I'm not convinced this is the time to compare citations, but yes for a description. In a future, I would say a comparison within the same journal might be more appropriate, once one will always need to deal with confounding factors that would be difficult to input in a model like the merit of a paper (which makes it more cited), publication bias, gender bias, geographic bias and others. As a last mention, please be very detailed with your control group. I couldn't follow it.

Thank you for pushing us to clarify our approach. We have added the details you requested. We have also added arguments and citations (the well-known Rosenbaum and Rubin 1983 and Dehejia and Wahba 2002 articles) to justify our use of propensity score matching to identify a control group of journals. One important issue to keep in mind is that the statistical analysis depends crucially on the existence of data over time for both the treated and comparison groups of journals; unlike most propensity score analyses, which simply rely on analysis at a single time point (in cross-sectional analysis), our analysis uses propensity score to match journals but then relies on the longitudinal dimension of the data for estimation, along the lines of a difference-in-differences approach. The use of data over time for the same journals is, we believe, an important strength of the analysis since addresses many common concerns associated with propensity score matching.

We regard as appropriate the reviewer’s questions about the right time to measure the effect we examine. We debated this issue prior to formulating our pre-analysis plan. Given the difficulties of examining all possible time periods, we formulated our research plans balancing importance and feasibility. We worry that any time period we could have chosen would have been vulnerable to questions about whether it was the right one. While it is possible our results would have been different had we chosen to examine a different time period, we did not choose our approach endogenously—that is, after having seen the results – and this should help address some concerns about the validity of the findings. We also note that Christensen, Dafoe, and Miguel’s analysis, focusing on citations at a single point in time (in 2017), does help to address some of these concerns about the timeframe of the analysis.

d) The rationale seems weak to me. Do you really think the perception of integrity might improve citations, at least nowadays? I mention the "thinking" because we don't have empirical evidence to support this. I know for a given topic when still under investigated, we have little evidence to support our kick-offs, but I don't know if is it the critical point, or even if data-sharing might incur in the augment of citations (unless for the use of the data, as you acknowledged in your paper). Citations are a multidimensional approach and a multivariable-explained phenomenon. Anecdotally, I'm not convinced data-sharing may play a role in citations within the myriad of variables - if researchers nowadays didn't perceive the value of data-sharing in various scenarios, I would say that an association is still harder (although not implausible by nature). Even, as a data-sharing researcher, I think data sharing should be observed as a way to reduce the waste in research; to improve the reproducibility and verifiability of a given finding; to augment the utility of a finding; and an ethical obligation of every single researcher regardless of the experimental addressed designed - what you brilliantly did. In time, congratulations by the way you shared the data. I didn't re-analyze your findings, but it is clear your compliance with reproducibility standards in that way.

Thank you for this comment, which is useful. You are correct that a journal’s or an article’s impact is determined by many more things than just whether the data happen to be shared. Even though this is one among many different factors, we see it as increasingly important in the era of open science, and designed our study to detect its influence. Moreover, there is an influential body of research (which we cite in our article) arguing that it is exactly the case that articles that share data are cited more often; however, we believe that this existing research has many methodological limitations that we are able to address in the current paper.

Nevertheless, we recognize the limitations of the evidence regarding how sharing data might affect perceptions of integrity, and have greatly reduced the discussion of whether sharing data should result in improved researcher reputation.

10.1371/journal.pone.0225883.r003 Decision Letter 1 Naudet Florian Academic Editor 2019 Florian Naudet This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

2 Oct 2019

PONE-D-19-18102R1

A study of the impact of data sharing on article citations using journal policies as a natural experiment

PLOS ONE

Dear Dr. Christensen,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The first referee found the manuscript improved (I agree) and he made two important comments. Please follow these comments.

In addition, in your abstract, please provide explicitly more details about your methods and results, expressed in a quantitative way using point estimates and 95 % confidence interval for your main outcomes. Please also make a short statement of your main limitations (see below) in the introduction. 

The second reviewer still thinks that the reporting must be improved. And I still agree with him because this is a core part of reproducible research.

To be more concrete here are some important suggestions for improvement : 

- In the method section, please provide all the details necessary to reproduce your selection of journals. Please try to adapt the PICOS formulation of the PRISMA statement for this. Please provide more details concerning the matching of journals explaining specifically why 17 journals were matched with 13 journals (this should be also addressed in the discussion section). Please make a specific paragraph "changes to the initial protocol". Please use subheadings to differentiate the different parts of the project (e.g. broad analysis / deep analysis / and a specific part statistical analysis). Please do not use the word "causal" in the manuscript, as you are only exploring association and not causal relationships : e.g. when you state : "equation 2 uses predicted data availability to estimate the causal...") ;

- In the result section, please provide all quantitative results of the model described in the method section. Importantly, I would expect your to report point estimates and 95 % CI. Table(s), or results written in the text must appear in the paper. As it is now, there are only qualitative descriptions/informations about the results. This section must report data, be factual and should not present a discussion of the results. Please move any interpretation to the discussion section. 

- In the discussion section, please start with a description of the principal finding and here please interpret your findings. Please add an important emphasis on risk of bias and especially confounding. Please make it very clear that this study is about associations and not causal relationships. Please also discuss the rational and the previous literature about the expected association between citations and data sharing practices. Please try to be systematic and exhaustive in your appraisal of the literature.

In my opinion, the manuscript therefore still needs major revision, on the form, before formal acceptance. 

We would appreciate receiving your revised manuscript by Nov 16 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (if provided):

None

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The comments of the reviewers were adressed and the manuscript has been considerably improved. There are two minor issues, which need to be adressed:

a) The abstract should be structured (introduction, methods, results, ...)

b) The authors have responded to the question about goodness of fit of 2-stage least squares regression in the "response to reviewers".In the manuscript, This point is not really tackled in the manuscript (only a comment that the first stage in the two-stage least squares regression is strong). The authors should include a statement concerning potential bias of estimates of coefficients in the text (along the lines of their argumentation in the "response to reviewers").

Reviewer #2: Dear authors,

thank your for addressing the points and checking my comments. They are partially addressed and I have more several comments on the format, however, I don't they are worthy now given the timing - you will need to re-write the manuscript, what I don't want. So, for now, I'm only on the merit of the content. Nothing more to say, please add/expand your limitation paragraph for future directions and the ICMJE Statement for Data-Sharing (your conditions).

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Lucas Helal

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

10.1371/journal.pone.0225883.r004 Author response to Decision Letter 1 Submission Version 2

11 Nov 2019

November 3, 2019

Response to Reviewers

Thank you to our reviewers for their continued careful review of our paper. We respond point by point to the comments that were highlighted by the editor.

In addition, in your abstract, please provide explicitly more details about your methods and results, expressed in a quantitative way using point estimates and 95 % confidence interval for your main outcomes.

Thank you for this suggestion. We now specifically mention both of our main methods (matching, instrumental variables) and describe the results quantitatively, reporting the standard error of our main estimate, as is reported in the tables and as is customary in our disciplines of economics and political science.

Please also make a short statement of your main limitations (see below) in the introduction.

The last sentences of the introduction now discuss our generalizability and limitations (namely, that we cannot address why articles are cited more, and we use observational not experimental data.)

- In the method section, please provide all the details necessary to reproduce your selection of journals. Please try to adapt the PICOS formulation of the PRISMA statement for this. Please provide more details concerning the matching of journals explaining specifically why 17 journals were matched with 13 journals (this should be also addressed in the discussion section).

We now clearly address this in the methods section with the following language:

“To account for the possibility of events that influenced both the change in journal policy and citation rates that could bias estimates, we collected comparable data for two natural comparison sets. We gathered data on theoretical articles published in the same journals; since these do not use empirical data, their citations should be largely unaffected by any change in data posting policy. We also matched the 17 treatment journals (which began to require data sharing) to control journals (which did not), and collected comparable citation data for empirical articles published in these control journals.

Our control journals are derived using conventional one-to-one propensity score matching with replacement from top-ranked journals that most closely match the treatment journals on six SCImago criteria (8,9). Our objective was to identify control journals that did not require data posting, but that were otherwise as similar as possible to the treatment journals in terms of observable characteristics. Accordingly, we began with non-treatment journals -- that never required data posting – from the same SCImago “Top 250” list of journals where we searched for our treatment journals. We matched treatment to control journals using the six indicators used to create the SCImago list itself. These criteria vary by journal and are posted by Scimago on its website. The six variables are: a) the journal’s h-index; b) the total number of citable documents published in the journal’s last three years; c) citations per document over the last two years; d) references per document; e) the country where the journal was published; and f) the category of the journal. Since there are only two countries where our treatment journals were published, we created a binary variable for journals published in the UK, leaving US journals as the default. And since we only have a limited number of treatment journals, we consolidated journal category into eight areas: a) Biology; b) Ecology; c) Economics; d) Medicine; e) Molecular Biology; f) Multidisciplinary; g) Sociology and Political Science; and h) Miscellaneous.

After creating the data set, we created a binary variable, with unity for our treatment journals and zero for all remaining journals (potential control journals). We then estimated a cross-sectional probit equation; the results are tabulated in our online supplement. After estimating the probit model, we then matched each of the treatment journals to a single control journal, using the closest possible journal, by predicted probit score within journal category. Sometimes this resulted in more than one treatment journal being matched to the same control journal. Both the control journals and the probit regression estimates themselves are freely available online. We are left with a set of 13 unique control journals along with 17 treatment journals; thus the MR analysis included data from a total of 30 distinct scientific journals. Appendix A provides the list of journals and more details on the data construction procedure.”

Please make a specific paragraph "changes to the initial protocol".

This information was already explicitly included in the appendix, but we now include it in the main methods section as well, as follows:

“In 2015, the CDM team pre-registered their analysis (https://osf.io/hv97m/). We deviate from it here in the following ways: First, we did not think to exclude articles that do not use data. (Results on the full sample, described in Appendix Tables B21 to B23, are generally weaker.) We also planned to control for time using cubic functions of months since publication. The results presented in the paper instead use year-discipline fixed effects to more flexibly control for time, as can be seen in the equations below. (Models with the cubic function yield nearly identical results.) In September 2017 CDM, expanded their analysis from the pre-registered field of political science (from which they had seen results) to include the economics portion of the project; this was before seeing any results from the broad analysis conducted by DM.”

Please use subheadings to differentiate the different parts of the project (e.g. broad analysis / deep analysis / and a specific part statistical analysis).

This is a good suggestion that will help make our paper clearer. We have added subsections to both methods and results for both the broad analysis and deep analysis. As both of these analyses are clearly statistical in nature, we do not specifically label either as statistical.

Please do not use the word "causal" in the manuscript, as you are only exploring association and not causal relationships : e.g. when you state : "equation 2 uses predicted data availability to estimate the causal...") ;

This was the only instance of the word “causal” in our paper. We are happy to modify the language here to say that “equation 2 uses predicted data availability to attempt to estimate the impact on number of total citations under assumptions we discuss below,” which we believe is completely accurate. We have also added citations to the 1996 Angrist, Imbens, and Rubin JASA paper “Identification of Causal Effects Using Instrumental Variables” (which has been cited over 5,000 times) and the 1994 Imbens and Angrist Econometrica paper “Identification and Estimation of Local Average Treatment Effects” (cited over 4,400 times) which both strongly make the case that instrumental variables can be used to estimate causal effects under certain assumptions. This is in addition to the Dehijia and Wahba (2002) REStat paper with 4,600 citations and the Rosenbaum and Rubin (1985) paper with 23,800 citations that discuss propensity score matching and causality, both of which we already cited. We discuss and test these assumptions in our paper to the extent possible, and are forthright about the evidence that we may violate one of these assumptions. We thus feel strongly that it is appropriate to state that we attempted to estimate causal effects, because that is, in fact, what we attempted to do.

- In the result section, please provide all quantitative results of the model described in the method section.

Importantly, I would expect you to report point estimates and 95 % CI. Table(s), or results written in the text must appear in the paper. As it is now, there are only qualitative descriptions/informations about the results. This section must report data, be factual and should not present a discussion of the results. Please move any interpretation to the discussion section.

Thank you for this suggestion. We have checked that every numerical result in the paper is either:

(a) directly from a table that appears in the paper itself, if a main result, with standard errors included

(b) from a discussion of a figure, and thus not an exact statistical result, and thus not possessing a standard error

(c) from a table that appears in the appendix, if not a main result and instead the result of a robustness check, with the exact appendix table number specified.

We have moved additional results tables into the main paper, but in order to be transparent and exhaustive, the appendix includes 40 tables, and certainly not all of these belong in the main paper.

- In the discussion section, please start with a description of the principal finding and here please interpret your findings. Please add an important emphasis on risk of bias and especially confounding. Please make it very clear that this study is about associations and not causal relationships. Please also discuss the rational and the previous literature about the expected association between citations and data sharing practices. Please try to be systematic and exhaustive in your appraisal of the literature.

Thank you for these suggestions. We now begin the discussion section with a discussion of our main results. We discuss bias in both of the methods sections, and have lengthened the discussion regarding both of the main methods, their assumptions, and potential biases.

Though this article is not intended as a review piece, an updated search did provide a few extra citations, which we discuss in more detail. If you are aware of others that we have not identified, we are open to suggestions. Regarding the interpretation of our results, and in particular whether we are estimating causal impacts, please refer to our discussion above of your earlier point.

Sincerely,

Garret Christensen, Allan Dafoe, Edward Miguel, Don A. Moore, and Andrew K. Rose

Submitted filename: 191107 ResponsetoReviewers.docx

10.1371/journal.pone.0225883.r005 Decision Letter 2 Naudet Florian Academic Editor 2019 Florian Naudet This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 2

15 Nov 2019

A study of the impact of data sharing on article citations using journal policies as a natural experiment

PONE-D-19-18102R2

Dear Dr. Christensen,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements. Thank you for all your edits and thanks to the two reviewers for their careful reading and comments. 

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

10.1371/journal.pone.0225883.r006 Acceptance letter Naudet Florian Academic Editor 2019 Florian Naudet This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

21 Nov 2019

PONE-D-19-18102R2

A study of the impact of data sharing on article citations using journal policies as a natural experiment

Dear Dr. Christensen:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Florian Naudet

Academic Editor

PLOS ONE
https://storage.googleapis.com/plos-corpus-prod/10.1371/journal.pone.0225883/1/pone.0225883.xml?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=wombat-sa%40plos-prod.iam.gserviceaccount.com%2F20220926%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220926T021823Z&X-Goog-Expires=86400&X-Goog-SignedHeaders=host&X-Goog-Signature=5fcee41164046694b4c12f816e673c3aab938ff363b3073a9b3365cc5a4e586771ab4ccf81434b8b5867126053e848191639ea4636d87b6ed24559ab045d0308a42654ad14e605f2b0cea2f8410cd94d65b79e76c2ffb0bda9d4586de6070700640c7d8d64e5a30ec3b6679fcefc63ddb9ad2583e998ffd395d232b67e8cf7ef2e0b7e7be429ba73776173635ae8bea3dc69deb1b12adb5378ebe8df7c44af2e2e5f841cdb6ab23263b5706e402b85b0c2170646c3986a960a8e367fce0f723539efd81e8e6febf33b13923bafd9cb43926d58d32d1d3cbcc8d28aebfba8b1c6dce18f727844fa8a13bae07290013fa0729b06ae10d8dbac11590c20518cf975
storage.googleapis.com
crawl=CC-MAIN-2022-40
152650
application/http; msgtype=response
sha1:VQL4PEZTJP46MMV6MDKDJWVUOBYBCVHX
<urn:uuid:45727002-9f08-4973-b3cd-9d4360d3a941>
2022-09-26T02:18:24Z
142.251.111.128
application/xml
sha1:PHUQTEZC7UFBGFMSL3TWHWEXHTGQLYEZ
<urn:uuid:b796b5af-2a8b-4e13-80d2-7002e49ed262>
https://storage.googleapis.com/plos-corpus-prod/10.1371/journal.pone.0225883/1/pone.0225883.xml?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=wombat-sa%40plos-prod.iam.gserviceaccount.com%2F20220926%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220926T021823Z&X-Goog-Expires=86400&X-Goog-SignedHeaders=host&X-Goog-Signature=5fcee41164046694b4c12f816e673c3aab938ff363b3073a9b3365cc5a4e586771ab4ccf81434b8b5867126053e848191639ea4636d87b6ed24559ab045d0308a42654ad14e605f2b0cea2f8410cd94d65b79e76c2ffb0bda9d4586de6070700640c7d8d64e5a30ec3b6679fcefc63ddb9ad2583e998ffd395d232b67e8cf7ef2e0b7e7be429ba73776173635ae8bea3dc69deb1b12adb5378ebe8df7c44af2e2e5f841cdb6ab23263b5706e402b85b0c2170646c3986a960a8e367fce0f723539efd81e8e6febf33b13923bafd9cb43926d58d32d1d3cbcc8d28aebfba8b1c6dce18f727844fa8a13bae07290013fa0729b06ae10d8dbac11590c20518cf975
response
<urn:uuid:8694b7d1-bd68-4471-b5b6-a66cc7267423>
isPartOf: CC-MAIN-2022-40
publisher: Common Crawl
description: Wide crawl of the web for September/October 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-62
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.4-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/