Comments and Questions

Summary of the Paper’s Findings/Contributions: The paper does not have empirical or theoretical findings, per se. It is a think piece, arguing the general merits of preanalysis plans for the case of replication studies and commenting on how to construct them. It raises several provocative issues and, in my opinion, has several thoughtful insights on the subject. However, the author has crafted a preanalysis plan that tailors itself very closely to his own current situation, dramatically limiting the usefulness of this discussion to the field at large. I had hoped to find some sort of scientific approach to how one might select and execute a replication study – and the current write up reflects something far different from that. If the currently submitted paper is not useful for laying out a defendable scientific scheme for others to follow, again because of the way situational information is called upon so heavily, then this manuscript becomes a simple addendum to the actual replication study of interest (Haurin and Rosenthal, 2007) that was conducted by the author, such that an abbreviated version of this discussion belongs paired with that work, rather than independently published in Economics: The Open-Access, Open-Assessment E-Journal. This becomes even more obvious once one considers the way the selection criteria were specifically crafted around the mentioned special issue’s call for replications. [As a side note, the special issue’s call was referred to multiple times but never actually pinned down. What Journal? Perhaps a link to that call for papers could be included. It is hard to evaluate the author’s choices in the preanalysis plan without seeing the specific constraints of the call.] Most of my comments fall along these lines. Comments: 1. I fail to see how the statements made about replication attempts do not apply more generally to essentially all scholarly research projects. Consider the use of statements like “Without prespecification, the amount of flowtime and budget that you could invest in a replication could grow uncontrollably” and “Your budget for a ‘successful’ replication is, most likely, less than that of the Bill & Melinda Gates Foundation”. Replace the word ‘replication’ with ‘study’ and you still have equally true statements. 2. Several arbitrary and/or unnecessary criteria are included in the selection method. For example: a. The statement about not selecting from the authors own previous replication work is a pure redundancy from the second (more general) criteria, which stated that previously replicated papers were to be selected. By definition, if the author had previously replicated a study then someone had previously replicated it.b. To exclude work by those with a connection to the current place of employment and those with a personal correspondence history seems completely arbitrary. When I started into the description of how the paper to target was “selected”, I suspected some sort of broad/general criteria that would produce a large population of potential choices that was not subject to obvious biases, and then with that pool some sort of random selection procedure would be applied. [For example, alphabetize the entire list of articles then use a random number generator to select one by position on the list.]c. The criteria of “A paper that I read within a year prior to the special issue’s call” illustrates the arbitrary nature of this discussion even more clearly than points a and b above. I do not actually know how to interpret this. Is the author really suggesting others follow this approach? How does this ‘selection criteria’ not boil down in the end to picking a paper the individual author is comfortable with and interested in for independent reasons? Again, I am not challenging the idea that this ‘comfortable’ method is a fine way to proceed with picking a study to replicate, but my point is that any ‘comfortable’ and/or ‘clearly non-scientific’ selection method does not merit independent publication as a stand-alone academic contribution. 3. Several arbitrary and/or unnecessary criteria are included in the definition of “success”. For example: a. Point 4 is quite vague. The author would wait a “prespecified amount of time” (later quantifying as a ‘few’ weeks) and would engage in a “prespecified number of attempts” (never quantifying).b. Similarly, point 5 mentions a “flowtime of around two months” to do the replication. How is this different from a normal research not just thinking to themselves of their research goals? Would the author’s own life/circumstances that played out during the time in question not be accounted for in a reasonable way? Again, I just don’t see a rigorous scientific contribution here that others could benefit from.c. Most importantly, point 9 indicates that “If the data that I downloaded was obviously flawed, then I would give up and work on another research paper.” The mind (ok, MY mind) reels upon reading this. I trust the authors of the original Haurin and Rosenthal paper because I have no reason at all not to. I trust the author of this paper for the same reason. Setting that aside, it seems that in the case of a purely doctored research endeavor, this step in the process would actually cause the research conducting a replication to give up when they should not. Assume the original study misrepresented the steps they took to include/exclude observations. When the replicator checked simple things like raw observation counts they would get something quite different and ‘give up’ on that replication. 4. While I understand the researcher, and in turn the journal to which the paper is submitted to, carries a direct interest in the field of Economics. However, Economics has direct connections to other disciplines including Finance, Accounting, Marketing, Political Science, Geography, Sociology, Urban Planning, and many others. [A simple review of the 750+ journals indexed by Econlit substantiates this point.] In every way, the paper was written as if other fields do not exist. For some topics, I could see that an acceptable choice, but in this case is difficult to defend given the position of the hard sciences, medicine, and psychology as being so dramatically ahead of Economics in terms of replication protocol. Since the paper does not shy away from making normative assertions, I’ll make one of my own. Academic research should seek to reflect the current state of knowledge on the investigated topic, regardless of what field has produced the key insights that determine that knowledge. For example, relevant classic research from psychology that considers the key differences between replications and the scientific method in physical sciences relative to social/behavioral sciences is ignored.