Abstract

In July 2017, Economics: The Open Access, Open Assessment E-Journal issued a call for papers for a special issue on “The Practice of Replication.” In that call, the journal explained that there was no generally accepted procedure for how to do a replication. Likewise, there was no generally accepted standard for determining whether a replication “confirms” or “disconfirms” an original study. Accordingly, the journal called for papers to identify principles for how to do a replication and how to interpret its results; and to apply those principles in crafting a replication plan for a study of the author’s choosing. The hope was that this exercise would produce some progress on “the practice of replication”. The special issue is now complete with a total of eight journal articles. This commentary places the respective articles within a common framework and identifies observations and lessons learned from the respective studies.

JEL Classification:

C10, C18, C50

Cite As

[Please cite the corresponding journal article] W. Robert Reed (2019). Takeaways from the Special Issue on The Practice of Replication. Economics Discussion Papers, No 2019-5, Kiel Institute for the World Economy. http://www.economics-ejournal.org/economics/discussionpapers/2019-5

Comments and Questions

Anonymous - Referee report 1

January 17, 2019 - 02:37

See attached file

[ Download attached file ]

W. Robert Reed - Response to reviewer 1

January 23, 2019 - 11:49

I thank the reviewer for the thoughtful review and comments. In terms of revising my manuscript, I will make the insertion suggested by the reviewer under “Minor points”. I also will also include a mention of Lei, Gelman and Ghitza (2017) in the revision.

Anonymous - Referee report 2

January 22, 2019 - 10:22

Overall, I found this article to be a fairly brief overview of the special issue. As much of it was summary of other papers, I do not have many comments to provide. Here are my thoughts/suggestions: 1) As you mention push button reproducibility/replication a few times, I suggest you cite the recent PLOS ONE paper, in case your readers would like to know more about that replication method: https://doi.org/10.1371/journal.pone.0209416.2) Would you please provide a few more sentences on your “significant sameness” suggestion? Why do you think this is an important approach to replication? From a replication perspective, I’ve had original authors complain when replication researchers “extend” their research that the study is not a replication. Pushing the boundaries of replication may lead original authors to argue that replication researchers are “scooping” their research. 3) Several replication critics have previously mentioned that replication plans (or pre-analysis plans for replication studies) don’t guarantee that researchers haven’t run their replication analysis before formulating a plan. Given that these plans feature prominently in your article, what would you say to these critics? Is there any method you can propose in your discussion of these plans to increase their validity/robustness? One possibility is mentioning that 3ie required replication researchers to discuss their level of interaction with the data to date when posting their plans.4) Given your extensive background with replication, I’m wondering if any of your views on replication changed from before this special issue to now. 5) Do you consider this special issue a “success”? Were you surprised by any of the papers/suggestions in these papers in this special issue? If you were going to re-run this special issue idea, would you want to change anything?6) I would argue that replication continues to be an under-utilized tool in research. Based on this special issue, do you have any thoughts on how replication should be made more mainstream?

W. Robert Reed - Response to reviewer 2

January 23, 2019 - 11:54

see attached file

[ Download attached file ]

Abstract

JEL Classification:

Links

Cite As

Comments and Questions