In a post on this thread on August 13, our friend Herman Aquinis made an assertion that we consider counter-factual. This assertion has been made by Herman in the past and has yet again been asserted. Specifically, Herman stated: "Related to your request, there is empirical evidence that so-called file drawer "problem" may not pose the threat to meta-analytic conclusions." He then provided a reference to the Dalton et al. (2012) paper on which he is second author.
In general, we have no objection to the self-promotion of one's work. It is a practice that many academics use to enhance their reputations in their fields. However, it is problematic when an assertion based on one's publication is incorrect and damaging to the conduct of science.
The reference to the "file drawer problem" concerns sample-level publication bias. That is, a reviewer of the literature does not have access to all the papers on a relation of interest. This raises a concern that the papers available to the reviewer are not representative of the literature on this relation and when synthesized yield inaccurate conclusions. One could also have a non-representative set of findings due to outcome-level suppression of results, such as when a paper is available but the author deleted results associated with non-significant hypotheses (see O'Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. (in press). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management. This paper is available on the JOM website).
Whether we are correct or Herman is correct is best addressed by reviewing the relevant literature. Thus, we have provided an annotated bibliography of research that can inform the debate so that readers can draw their own conclusions. Below, we introduce several lines of evidence in bold and larger font and after each line of evidence provide citations to the relevant source material.
Several papers have critiqued the Dalton et al. (2012) work. My interpretation of this body of work is that the Dalton et al. (2012) paper provides no useful information on the frequency in which publication bias occurs or the extent of damage of such publication bias when specific bivariate relations are examined. The papers are:
Kepes, S., Bennett, A., & McDaniel, M.A. (in press). Evidence-based management and the trustworthiness of our cumulative scientific knowledge: Implications for teaching, research, and practice. Academy of Management Learning and Education. doi: 10.5465/amle.2013.0193
Banks, G.C., Kepes, S., McDaniel, M.A. (in press). Publication bias: Understanding the myths concerning threats to the advancement of science. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity, and fable in organizational and social sciences. New York, NY: Routledge.
Kepes, S. & McDaniel, M.A. (2014, August). Publication bias: Causes, detection, & remediation. PDW presented at the annual meeting of the Academy of Management. Philadelphia.
Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6: 252-268.
Kepes, S., McDaniel, M. A., Brannick, M. T., & Banks, G. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the Meta-analytic Reporting Standards). Journal of Business and Psychology, 28: 123-143.
McDaniel, M.A. (2012, November). Publication bias and other sensitivity analyses in meta-analysis. Invited presentation to Center for Advanced Research Methods and Analysis (CARMA), November 9, 2012.
Ferguson and Brannick (2012) examined meta-analyses in the psychological literature. Their conclusions: (1) Publication bias exists in 40% of published meta-analyses; (2) publication bias is "worrisome" in about 25% of meta-analyses:
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. doi:10.1037/a0024445
Publication bias has been inferred in journal-published mean racial differences in job performance (McDaniel, McKay, & Rothstein, 2006):
McDaniel, M. A., McKay, P. & Rothstein, H. (2006, May). Publication bias and racial effects on job performance: The elephant in the room. Paper presented at the 21st Annual Conference of the Society for Industrial and Organizational Psychology. Dallas.
Publication bias has been inferred in employment test vendor validity data (McDaniel, Rothstein, Whetzel, 2006; Pollack & McDaniel, 2008):
McDaniel, M. A., Rothstein, H. R. & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927-953. doi: 10.1111/j.1744-6570.2006.00059.x
Pollack, J. M. & McDaniel, M. A. (2008, April). An examination of the PreVisor Employment Inventory for publication bias. Paper presented at the 23rd Annual Conference of the Society for Industrial and Organizational Psychology. San Francisco.
Publication bias has been inferred in journal-published research on the validity of structured interviews (Kepes, Banks, McDaniel, & Whetzel (2012):
Kepes, Banks, McDaniel, & Whetzel (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624-662. doi: 10.1177/1094428112452760
Publication bias has been inferred in literature on mean racial differences in personality (Tate & McDaniel, 2008):
Tate, B. W. & McDaniel, M. A. (2008, August). Race differences in personality: an evaluation of moderators and publication bias. Paper presented at the Annual meeting of the Academy of Management, Anaheim CA.
Publication bias has been inferred in the judgment and decision making literature (Renkewitz, Fuchs, & Fiedler, 2011):
Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870-881.
Publication bias has been inferred in the Big 5 validity literature (Kepes, McDaniel, Banks, Hurtz, & Donovan, 2011):
Kepes, S., McDaniel, M. A., Banks, C., Hurtz, G., & Donovan, J. (2011, April). Publication bias and the validity of the Big Five. Paper presented at the 26th Annual Conference of the Society for Industrial and Organizational Psychology. Chicago, IL.
Publication bias has been inferred in the reactions to training literature (Kepes, Banks, McDaniel, & Sitzmann, 2012):
Kepes, S., Banks, G. C., McDaniel, M. A., & Sitzmann, T. (2012, August). Assessing the robustness of meta-analytic results and conclusions. Paper presented at the annual meeting of the Academy of Management, Boston, MA.
Publication bias has been inferred in the literatures concerning the relation between work experience and performance, gender differences on transformational leadership, and Pygmalion interventions (Kepes, Banks, & Oh, 2014):
Kepes, S., Banks, G., & Oh, I.-S. (2014). Avoiding bias in publication bias research: The value of "null" findings. Journal of Business and Psychology, 29: 183-203.
Publication bias has been inferred in the literature concerning Conditional Reasoning Test validity (Banks, Kepes, & McDaniel, 2012):
Banks, G.C., Kepes, S., & McDaniel, M.A. (2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20, 182-196. doi: 10.1111/j.1468-2389.2012.00591.x
Publication bias has been inferred in the literatures concerning strategy and entrepreneurship:
Harrison, J. S., Banks, G. C., Pollack, J. M., O'Boyle, E. H., & Short, J. (in press). Publication bias in strategic management research. Journal of Management. doi: 10.1177/0149206314535438
O'Boyle, E. H., Banks, G. C., & Rutherford, M., W. (in press). Publication bias in entrepreneurship research: An examination of dominant relations to performance. Journal of Business Venturing. doi: 10.1016/j.jbusvent.2013.10.001
APA's Meta-Analysis Reporting Standards encourage the inclusion of publication bias analyses (see the section on data censoring):
American Psychological Association. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839–851. doi:810.1037/0003-1066X.1063.1039.1839.
American Psychological Association. (2010). Publication manual of the American psychological association (6th ed.). Washington, DC: American Psychological Association.
Concluding remarks:
Some of the above publication bias analyses are based on small amounts of data and/or use limited publication bias analyses. Other publication bias analyses are based on substantial amounts of data and use multiple methods to investigate publication bias.
Some investigations of publication bias draw the inference that publication bias is not present in a literature. Other investigations infer that the effects of publication bias are modest. Still other investigations infer that worrisome levels of publication bias exist in the literature examined.
When judging the prevalence and impact of publication bias, we suggest it would be useful to consider all the evidence.
Best wishes,
Mike McDaniel & Sven Kepes