Discussion Paper

No. 2019-65 | November 18, 2019
Inference in economic experiments

Abstract

Replication crisis and debates about p-values have raised doubts about what we can statistically infer from research findings, both in experimental and observational studies. With a view to the present debate on inferential errors, this paper systematizes and discusses experimental designs with regard to the inferences that can and – perhaps more important – that cannot be made from particular designs.

JEL Classification:

B41, C18, C90

Assessment

  • Downloads: 276

Links

Cite As

Norbert Hirschauer, Sven Grüner, Oliver Mußhoff, and Claudia Becker (2019). Inference in economic experiments. Economics Discussion Papers, No 2019-65, Kiel Institute for the World Economy. http://www.economics-ejournal.org/economics/discussionpapers/2019-65


Comments and Questions


Anonymous - Referee Report 1
January 06, 2020 - 08:21

This is a very useful paper, reminding the reader what the requirements are for establishing causation in experiments. The paper also contains a good discussion of external validity. This paper is timely, given the awarding of the 2019 Nobel Prize in Economic Sciences to three pioneers of using randomised controlled ...[more]

... trials (RCTs) in development economics. I have only a couple of minor comments on the paper.

The discussion of randomised controlled trials versus non-randomised controlled trials, on pages 2 and 3, could give the impression that RCTs do not involve before and after comparisons, when typically they do.

The paper correctly makes the point that a low p-value tells us nothing about the external validity of the results. This is presumably also true of studies based on observational data. It would have been useful to include a discussion of why external validity is more problematic (assuming it is) for experimental studies than for studies using observational data.


Sven Grüner - Response
January 09, 2020 - 14:01

We very much appreciate the reviewer’s thorough reading of our manuscript as well as his very thoughtful comments. In the following, we have listed the reviewer’s comments along with our replies to the changes implemented in the manuscript.


Anonymous - Referee Report 2
January 06, 2020 - 08:31

see attached file


Sv en Grüner - Reply
January 09, 2020 - 14:02

We very much appreciate the reviewer’s thorough reading of our manuscript as well as his very thoughtful comments. In the following, we have listed the reviewer’s comments along with our replies to the changes implemented in the manuscript.


Bob Reed - Decision Letter
January 27, 2020 - 08:32

see attached file