Discussion Paper

No. 2015-3 | January 20, 2015
Meta-Analysis in a Nutshell: Techniques and General Findings
(Published in Special Issue Meta-Analysis in Theory and Practice)
(Submitted for Survey and Overview)

Abstract

The purpose of this note is to introduce the technique and main findings of meta-analysis to the reader, who is unfamiliar with the field and has the usual objections. A meta-analysis is a quantitative survey of a literature reporting estimates of the same parameter. The funnel showing the distribution of the results is normally amazingly wide given their t-ratios. Little of the variation can be explained by the quality of the journal or by the estimator used. The funnel has often asymmetries consistent with the most likely priors of the researchers, giving a publication bias.

JEL Classification:

B4, C2

Assessment

  • Downloads: 2063

Links

Cite As

Martin Paldam (2015). Meta-Analysis in a Nutshell: Techniques and General Findings. Economics Discussion Papers, No 2015-3, Kiel Institute for the World Economy. http://www.economics-ejournal.org/economics/discussionpapers/2015-3


Comments and Questions


Anonymous - Referee report
January 28, 2015 - 12:58

see attached file


Tom D. Stanley, Hendrix College - Invited reader comment 1
February 03, 2015 - 11:35

see attached file


Jon P. Nelson, Pennsylvania State University - Invited reader comment 2
February 11, 2015 - 09:05

A more a general comment on the method of meta-analysis (see attached file)


Anonymous - Comments
February 26, 2015 - 10:54

Paldam provides a nice, sharp and informative introduction to meta-analysis. Overall, the introduction is very well written. I found some statements in the paper that could be reassessed or qualified.

On page 2, Paldam states that regressions are cheap in economics. They are certainly cheaper than running clinical trials ...[more]

... but more and more economists are conducting field and other experiments, some of which are rather expensive. And even though medical trials are more expensive, the analysis of the trials is also vulnerable to various research biases. Paldam’s overall point, however, is valid.

Also on page 2 Paldam refers to simulation studies that have been conducted. Some of these are actually problematic in the way they have been constructed. There remains a need for a comprehensive assessment of these studies. This is an area that warrants further attention by meta-analysts.

Page 4, the second point in table 1 was a bit unclear: what is meant by the expression “it is actually true”? Further on this page the term cp-controls is unclear (I assume it refers to ceteris paribus controls). There are other such terms used in the paper, and they may not all be helpful to readers new to meta-analysis: though this comment is probably more of an issue of style.

Paldam devotes the paper to the FAT-PET MRA. This is actually only one strand/framework/methodology. There are others. While I share Paldam’s enthusiasm for the FAT-PET, readers should be aware that there are other approaches to conducting MRA.

On page 8 there is an important discussion on ‘top journals’. My own suspicion is that ‘top’ journals will tend to publish larger effects – they not only innovate, they inflate!