Discussion Paper
No. 2017-75 | September 28, 2017
Raymond Hubbard
A proposal for replicating Evanschitzky, Baumgarth, Hubbard, and Armstrong’s ‘Replication research’s disturbing trend’ (Journal of Business Research, 2007)
(Published in The practice of replication)

Abstract

This paper is about how the author proposes to replicate Evanschitzky, Baumgarth, Hubbard, and Armstrong’s ‘Replication research’s disturbing trend’ (Journal of Business Research, 2007). This is because estimating the incidence of published replication research and its outcomes must be continued.

JEL Classification:

A14, B23, B41, C18

Cite As

Raymond Hubbard (2017). A proposal for replicating Evanschitzky, Baumgarth, Hubbard, and Armstrong’s ‘Replication research’s disturbing trend’ (Journal of Business Research, 2007). Economics Discussion Papers, No 2017-75, Kiel Institute for the World Economy. http://www.economics-ejournal.org/economics/discussionpapers/2017-75


Comments and Questions



Anonymous - Referee report 1
October 30, 2017 - 11:20
see attached file
[ Download attached file ]

Raymond Hubbard - Response to reviewer 1
December 14, 2017 - 11:26
Responses to Reviewer 1 on “A Proposal for Replicating EBHA(2007).” I would like to thank Reviewer 1 for his/her constructive comments. I would also like to apologize for my delay in responding to you. This reviewer notes that more attention should have been addressed “to the definition of various types of replication” found in T and K’s (1999) six-fold replication typology in the TABLE. Fair enough. Let me respond by saying that, while extremely important, I do not consider (1) and (2) to be replications at all (they don’t involve collecting new data from the same or different populations). Meanwhile (6) is so broad as to constitute a different/original paper. See Hubbard, Corrupt Research: The Case for Reconceptualizing Empirical Management and Social Science, 2016, SAGE, pps. 105-124, for detailed explanations for the above. This leaves (3), (4), and (5). I noted in my proposal that there is no such thing as an exact replication (3), which is why I said I prefer the expression “replication with extension,” which involves modifications to measurement instruments (4) or target populations (5). Reviewer 1 then asks “how does one determine whether the population is the same?” Again, fair enough. Except that, as I emphasized, the passage of time alone means that no two replications/populations will ever be exactly alike. And I completely agree with this reviewer that “Some truths just turn out to be time-dependent. “ Which is precisely why we need to find out. This is an empirical question to be addressed. And replication research answers this question. Finally, Reviewer 1 notes that if I find a 20% published replication rate in the follow-up, it would not be a “successful” replication because the confidence intervals do not overlap, and asks “But what would be the meaning of this?” To which I would respond, this is wonderful news! Sometimes we don’t want “successful” replications if the initial result is woeful. Refer again to the Hubbard reference in paragraph 2. Again, my thanks to this reviewer for his/her comments. They’ve helped reinforce the value of replication research.

Anonymous - Referee report 2
November 15, 2017 - 14:25
This is a great proposal. I cannot agree more with everything said. Here are just a few minor remarks: 1. Management seems to (suddenly?) move quickly, as is clear from the recent editorials in top field journal such as JIBS (Meyer et al. 2017) and SMJ (Bettis et al., 2016) regarding new reporting guidelines. 2. Is what you propose here regarding double-blind coding an exact copy of what EH and EBHA did? I guess that would be a good thing. 3. You have a nice anecdote about EH’a submission journey. What about EBHA’s? Similar, given that this piece ended up in the JBR? (If so, that would be telling.) Good luck.

Raymond Hubbard - Response to reviewer 2
December 14, 2017 - 11:27
I thank this reviewer for his/her very constructive comments, and apologize for my delay in replying to them. The reviewer made “a few minor remarks.” (1) Referred to recent changes in management journals aimed at encouraging replications. On the surface, this is obviously good news. But it must be vigorously reinforced by actively encouraging the submission and publication of replications on an ongoing basis. Too often this is not done. Or a new editor has different ideas, etc. So I’m not holding my breath. (2) Re. double-blind coding being used in HA (only for a 10% subsample) and EBHA (yes). (3) About HA’s “submission journey.” EBHA is somewhat similar. Thanks for your good wishes.

Anonymous - Referee report 3
November 18, 2017 - 11:05
see attached file
[ Download attached file ]

Raymond Hubbard - Response to reviewer 3
December 14, 2017 - 11:28
I thank reviewer 3 for his/her appreciation of the EBHA Proposal, and the constructive comments, addressed below. (i) General Discussion of Principles …… “First, I am not sure about the author’s comments on the impossibility of conducting an exact replication.” The reviewer then appears to confuse Tsang and Kwan’s type 1 replication -- checking the analysis -- with exact replication. But I do not consider T and K’s type 1 replication to be a replication at all. An exact replication involves taking a new sample from the same population (almost always for a different time period than the original study). “Third, what is a difference between a replication (esp. … type 6) and a follow-up study?” Type 6 of T and K’s also is not a replication. To see why this is so see Hubbard, Corrupt Research: The Case for Reconceptualizing Empirical Management and Social Science, 2016, SAGE, p. 124. “The main goal of replication … is to reproduce findings>” This is an important, but much too narrow a definition. See Hubbard (2016, pps. 105-124) for a discussion of other roles for replication, e.g., developing empirical generalizations. (iii) Replication Plan Some of the Reviewers recommendations for comparisons in Step 1 are not practical, e.g., comparisons between sample 1 and sample 2 for 1990-2004, because no journal would consider publishing them. For 2005-2015 comparisons, I’ve already suggested this. Some of these arguments extend to Step 2, also. I consider only T and K’s types 3, 4, and 5 to be true replications See the above Hubbard 2016, pps. 105-124, for a thorough discussion of why. (iv) Discussion of How to Interpret ……. Confidence intervals would not be used on populations, only when comparing samples. I extend my thanks to this Reviewer.

W. Robert Reed - Decision letter
January 20, 2018 - 20:00
see attached file
[ Download attached file ]