Hi – really enjoying this discussion. I just came across this which I think more or less does exactly what you mention below Robert.
http://www.psychologicalscience.org/index.php/replication
Replicability is a cornerstone of science. Yet replication studies rarely appear in psychology journals. The new Registered Replication Reports article type in Perspectives on Psychological Science fortifies the foundation of psychological science by publishing collections of replications based on a shared and vetted protocol. It is motivated by the following principles:
• Psychological science should emphasize findings that are robust, replicable, and generalizable.
• Direct replications are necessary to estimate the true size of an effect.
• Well-designed replication studies should be published regardless of the size of the effect or statistical significance of the result.
Also, there is a two-stage submission process in some journals which gets around the problem of publication bias such as Attention, Perception, & Psychophysics: http://www.springer.com/psychology/cognitive+psychology/journal/13414
The RRR submission is a two-stage process. Authors submit a proposed study. If it passes initial review, APP will commit to publishing the results, regardless of the outcome, so long as the final study conforms with the initially approved proposal. That is, for example, we will publish a null result as a Registered Report. The Registered Report format is appropriate for studies that seek to test clearly articulated, theoretically significant hypotheses (e.g., Theory A predicts X whereas Theory B predicts Y). Replications should be precise replications (with possible extensions) of theoretically important findings. The initial submission would approximate the background and method sections of a relatively short standard research article.
Cheers
Rob
Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath
Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Robert Steinbauer (rstnbuer)
Sent: 19 February 2014 01:38
To: OB@AOMLISTS.PACE.EDU
Subject: Re: [OB-LIST] Journal article retractions
Kim raised a valid point by stressing that editors as well as reviewers frequently associate supported hypotheses with a contribution to the field; well-reasoned but unsupported hypotheses are too often automatically dismissed. The pressure to publish, the lack of a proper control system, and the lack of consequences might motivate some enough to misrepresent their data in an attempt to keep their job and get tenure. While making data and scripts available is an honorable idea, I suspect that individuals who are willing to misrepresent results will probably misrepresent data too. So it would only shift the problem to a different level.
Not reporting the result of a study to the reviewers might be one way to resolve the issue. However, I would like to take this idea a step further. A little while ago, I was discussing the same topic with Frank Bosco. He argued that similar to a dissertation, scholars should only submit a proposal to a Journal. If the editor and reviewers find the proposed study interesting, the Journal agrees to publish it – regardless of whether the hypotheses are supported in the end. This would shift the focus from data driven manuscripts that are also vulnerable to manipulation to well-reasoned studies that are grounded in solid-theory. After all, most of us are Doctors of Philosophy and not Statisticians whose job it is to make data work.
Robert Steinbauer
I don't know if this is true, but I think many journals have a bias towards only publishing articles in which the majority of the hypotheses are supported. I wonder what would be the effects if Reviewers and Editors eliminated this bias. That is, we only judge the paper on whether or not we think the topic is relevant to the readers and the logic of the initial hypotheses is sound. Then we leave it up to the authors to discuss why their hypotheses were not supported. Would this lessen the likelihood of unethical practices by authors? I don't know. But it should, at least, prevent authors from practicing the file drawer problem.
The article may still be rejected for other reasons, but it wouldn't be because the initial hypotheses were rejected. This, in the long run, might lead to better theorizing, theory trimming, and partial replication.
Kim Boal
I would also like to point your attention to the upcoming issue of the Psychology of Aesthetics, Creativity in the Arts (Feb 2014) which will have a special section on replications and how to address this issue including a focal article and responses from a number of reviewers and notable figures. While some of the content is specific to creativity, the reasoning and issues raised are not!
Roni
Roni Reiter-Palmon, Ph.D.
Isaacson Professor of I/O Psychology
Editor, The Psychology of Aesthetics, Creativity and the Arts
Director, I/O Psychology Graduate Program
Director of Innovation, Center for Collaboration Science
University of Nebraska at Omaha
Office location: ASH 347K
Phone: 402-554-4810
Hi there
This paper (attached and forthcoming in JoM) has just been bought to my attention. It seems very relevant to this discussion.
O'Boyle, E. H. & Banks, G.C., & Gonzalez-Mule, E. (in press). The Chrysalis Effect: How ugly initial results metamorphosize into beautiful articles.
ABSTRACT: The issue of a published literature not representative of the population of research is most often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resulting journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically non-significant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label, the Chrysalis Effect.
Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath
Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)
Hi Michael and everyone
I'm personally really pleased this is being discussed. Because of limited retractions, an ever-increasing proportion of hypotheses that are supported in published papers and the lack of replications it's hard to exaggerate how distorted the scientific process has become.
This a recent OB-ish example: http://retractionwatch.com/2013/09/19/fredrickson-losada-positivity-ratio-paper-partially-withdrawn/
A comment about (the lack of) retractions specifically in economics and business: http://retractionwatch.com/2012/12/12/why-arent-there-more-retractions-in-business-and-economics-journals/
Even The Economist (October 2013) has picked up on this: http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble
Best wishes
Rob
Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath
Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)
Steve and Herman,
It should come as no surprise that there are a growing number of retractions, given the evidence my co-authors and I gathered in a paper we published in AMLE
Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010). Management Science on the Credibility Bubble: Cardinal Sins and Various Misdemeanors. Academy of Management Learning & Education, 9(4), 715-725.
Regards,
Alan
Alan N. Miller, Ph.D.
Professor of Management and Chairman
Department of Management, Entrepreneurship and Technology
Lee Business School
University of Nevada, Las Vegas
"Aguinis, Herman" ---02/13/2014 07:22:59 AM---Steve, The answer to your question is yes. There is a site called "Retraction Watch," and they keep
From: "Aguinis, Herman" <haguinis@INDIANA.EDU>
To: OB@AOMLISTS.PACE.EDU
Date: 02/13/2014 07:22 AM
Subject: Re: [OB-LIST] Journal article retractions
Sent by: Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU>
Steve,
The answer to your question is yes. There is a site called "Retraction Watch," and they keep track of retractions across many scientific fields. See: http://retractionwatch.com/
I hope this helps,
Regards,
--Herman.
Herman Aguinis, Ph.D.
John F. Mee Chair of Management
Professor of Organizational Behavior and Human Resources
Founding Director, Institute for Global Organizational Effectiveness
Department of Management and Entrepreneurship
http://mypage.iu.edu/~haguinis/
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Kelman, Steven
Sent: Tuesday, February 11, 2014 11:16 AM
To: OB@AOMLISTS.PACE.EDU
Subject: Re: [OB-LIST] Journal article retractions
Have there been retractions of OB papers?
Steve Kelman
Albert J. Weatherhead III and Richard W.
Weatherhead Professor of Public Management
Editor, International Public Management Journal
Tel: 617-496-6302
Personal Homepage: http://www.ksg.harvard.edu/fs/skelman
"The Lectern," my blog on FCW.com: http://fcw.com/blogs/lectern/list/blog-list.aspx
IPMJ Homepage: http://www.tandfonline.com/toc/upmj20/current
Like IPMJ on Facebook:
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
Sent: Tuesday, February 11, 2014 9:00 AM
To: OB@AOMLISTS.PACE.EDU
Subject: [OB-LIST] Journal article retractions
A modest proposal: Given recent journal article retractions, it would seem prudent for journals to require all data sets and syntax/scripts to made available to all on journal websites.
Mike
--
Michael A. McDaniel, Ph.D.
Professor - Human Resources and Organizational Behavior,
Department of Management,
Research Professor, Department of Psychology
Virginia Commonwealth University
301 West Main Street, PO Box 844000
Richmond, VA 23284-4000
http://www.people.vcu.edu/~mamcdani/
Voice: 804-827-0209
E-mail: mamcdani@VCU.edu