Replications Guidelines

The following is a list of guidelines/principles that apply to replications, from the perspective of the e-journal Economics. These guidelines are intended to provide systematic guidance as to how to encourage high-quality replications. This is because replications are an important public good to the economics community, and as such, they tend to be under-valued. By providing these guidelines and by opening a replications section in our open access, open assessment journal, we hope to make it easier for authors to provide this important public good.

These guidelines are provisional, in that they may be changed in the future. However, these changes are not likely to be large, and at the present time, these guidelines serve as a guide for authors who wish to submit replication studies to Economics.

Definition of a replication

A replication is a study whose primary purpose is to confirm the validity of a previously published study in a peer-reviewed economics journal, or of research from other sources (e.g., books, government publications, etc.). A replication will generally consist of two parts. The first part is an attempt to reproduce key findings from the original study. In this exercise, the replicating author will ideally attempt to exactly reproduce the original study’s findings using the identical data set. If the replicating author is initially unsuccessful in this exercise, it is expected that they will contact the original author to gain their assistance in this exercise. In the event that a replication study is successful at reproducing the original study, the next part should consist of a robustness check, if possible. This can be done through a variety of means such as estimating alternative variable specifications, using new data, and/or applying appropriate, alternative estimation procedures. The journal is committed to publishing both confirming and disconfirming replications. The only criterion is that the replication be done to a high standard of professional competency.

Overview of procedure

1. The procedure mirrors the procedure followed for regular submissions, with some exceptions where noted.

First, the article is submitted to the replication section of the journal. The journal sends out the replication study to the original author(s) asking for feedback within 60 days. At the end of 60 days, either with or without feedback from the original author, the journal decides whether the replication study has sufficient merit to be published as a discussion paper. All replication studies published as a discussion paper shall contain a note from the journal, placed at the front of the discussion paper, emphasizing that their findings should be viewed as tentative until the paper has completed the review process and been published as an article.

The motivation for the 60-day waiting period (which does not apply to regular submissions) is to provide due process for replicated authors. As such, if the replication study is judged by the editors at any time within the 60-day waiting period to be without substantial merit, then the submission will be rejected without being accepted as a discussion paper.

2. If the paper is accepted as a discussion paper, the journal shall invite at least two reviewers (other than the original author) to evaluate the discussion paper, and the discussion paper will be open for comments. This is the same refereeing process as for a regular submission.

3. Upon receiving reviews and comments, and then upon receiving a reply from the replicating author, the journal makes a decision whether or not to publish as an article.

4. If the decision is to publish as an article, the original author is invited to submit a new reply in response to the final revised version, or to modify a previously submitted reply.

5. When the replication study is ready to be published as a journal article, then it is published immediately. However, a reply can be received up to 6 months after publication of the replication study and still be posted alongside the replication study.

Economics may, at its discretion, choose to amend these guidelines at any time to impose a second 60-day waiting period, analogous to the first 60-day waiting period. This waiting period would be intended to give the original author a second chance to reply.

6. The final reply by the original author is reviewed by editors and not sent out for external review.

7. Upon publication as articles, both the replication study and the final reply are published alongside each other.

Editorial responsibilities

1. The responsibility for handling replications will be shared across several co-editors. The journal has appointed initial Co-editors for Replications with expertise in empirical macro, empirical micro, econometrics, and experimental economics.

2. The lead Co-Editor for Replications is initially Bob Reed. He assigns the paper to one of the Co-Editors depending on area. The journal will coordinate with that Co-editor in deciding on appropriate reviewers.

3. The procedure for making a decision on the paper requires the approval of both members of a two-person committee. This committee consists of the main Co-Editor for Replications (Bob Reed) and one other Co-Editor (generally the Co-Editor handling the paper, if that is not the main Co-Editor for Replications), with disagreements mediated by the Assistant Editor for Replications (Claire Boeing-Reicher).

Guidelines for replicating and replicated authors

1. The journal will have a special category for replications, as it already has for surveys and overviews and for policy papers.

2. Submitted manuscripts should be clearly identified in the online submission process as a “Replication Study”. Such studies are to be screened by the Assistant Editor for Replications (Claire Boeing-Reicher) to see if the study in fact fits this definition of a replication study and may be of substantial merit. If this is the case, the Assistant Editor for Replications will forward this replication on to the relevant Co-Editor for Replications.

3. The original research will generally be an article in a peer-reviewed economics journal, though research from other sources (e.g., books, government publications, etc.) may be considered if that research is judged to have had sufficient impact.

4. Replication studies may come from any area of economics (e.g., macroeconomics, microeconomics, econometrics, experimental).

5. Researchers may inquire in advance with a designated Co-editor for guidance on the suitability of a particular study for replication. Specifically, potential replicators may contact the e-journal in advance to clarify whether a particular study has “sufficient impact” (point 3), although the journal does not commission replication studies or actively suggest topics. The journal does suggest, however, that replicators choose topics that are related to their own research programs, in order to maximize the chance of a replication being of substantial merit.

In addition, the Göttingen Replication Wiki is a good source of information on which articles need to be replicated: http://replication.uni-goettingen.de/wiki/index.php/Main_Page.

6. Replication studies should be standard, full-length manuscripts, although shorter manuscripts will be considered. Brevity is good.

7. The replicating researcher must first attempt to exactly reproduce the original findings. If the original results cannot be exactly reproduced, then the replicating researcher is required to contact the original author to reconcile any differences. The journal will require evidence that the replicating author has attempted to contact the original author before publishing the paper as a discussion paper. Such evidence may consist of a trail of correspondence (e.g. e-mail) which indicates that the replicating researcher has given the original author a good-faith chance to respond and/or clarify the situation.

8. The standard format for a replication study consists of 4 parts:

  • An introduction in which the importance of the original study is discussed, including metrics where possible (e.g., citations from Web of Science, Scopus).
  • A reproduction of key results from the original study, with the original study’s results and the replicating study’s corresponding estimates placed side-by-side in a table.
  • If necessary, robustness checks in which one or more dimensions of the original study are extended or modified (e.g., the sample is updated with more recent data, alternative econometric procedures are employed, etc.) NOTE: It will not be necessary to do robustness checks if it is determined that the original study’s results cannot be reproduced using the original author’s code and data. Also, robustness checks are to be judged by their relevance to the original research question—if a replication study moves sufficiently beyond the original research question, it ceases to be a replication study but instead becomes a regular submission, which is subject to a different set of guidelines.
  • A conclusion that summarizes which results from the original study have been confirmed/disconfirmed.

9. The journal will publish both confirmations and disconfirmations of original studies. The only consideration will be quality of the replicating study.

10. All replication studies will use the following title template: “A Replication of ‘Title of original paper here’ (Name of journal of original study here, Year of publication of original study here)”. If a second replication on the same paper gets done, then the title will be: "A Second Replication of 'Name of Article Here' (Name of Journal Here, Year Here)", or a third, or so on. This has the added advantage of letting the reader immediately know that more than one replication of the same article has been done.

11. All submissions should include a financial disclosure form. Authors should follow the instructions listed in the AER’s disclosure policy (https://www.aeaweb.org/journals/policies/disclosure-policy).  This financial disclosure form will be published with the discussion paper. 

12. The replicating author will be required to make their data and code available.  These must be provided before the paper is published as a discussion paper.  The data and code must allow exact reproduction of the results of the replication study.

13. The original author will be given the opportunity to respond to the replication. The response will be published alongside the replication study. Any further correspondence will then take place in the comment section for the publication. We do this in order to promote conversation.

14. The journal will publish replies that contain clarification of data, programs, and/or procedures that should have been supplied to the replicating author, with the note (where appropriate) that they were not originally supplied in the first place.

15. The journal is willing to publish multiple replications of an original article if the studies are carried out by different replicating authors.

16. It is not the responsibility of the journal to adjudicate disputes between the replicating and original authors. The journal’s task is merely to ensure that both the replication study and corresponding reply have been done to a high standard of competence.