One way of reducing the questionable research practices in our field is for authors to submit data sets and syntax/scripts that are made available to all on the journal website. Perhaps that information would be made available immediately or perhaps after a time lag. Supplemental information can also include all that good stuff that the editor made you take out of the paper, like the five ways that you operationalized a construct and only reported the results for one operationalization (i.e., the one that gave you the largest effect size). Separate from reducing the frequency of the gerrymandering of our science, data sharing relieves authors of the burden of digging through their old data sets when some wants the data or an analysis run (Chad, I have not forgotten your data request).
My understanding is that Journal of Management is the sole management/applied psychology journal that permits the submission of supplemental materials. Could someone post the policies of the other management/applied psychology journals?
APA journals have a policy of permitting the submission of supplementary materials but left the decision up to each journal editor. As far as I can tell, the outgoing editor of Journal of Applied Psychology declined to do so. The current editor of Personnel Psychology told me he would try to do this but I do not know that status of that effort.
It would be beneficial to science if the current editors of our major journals started to accept supplemental information. I am hoping that the editors of our major journals can make a public statement about their intentions in this regard and post it on this lisetserv.
Thank you for considering this request.
Mike
On Sat, Feb 15, 2014 at 10:31 AM, Lori Kendall <lorikendall@gmail.com> wrote:
As a 2nd year doctoral student (mid-career switch), this thread has been eye-opening and extremely helpful. It appears that the pressure for tenure and the narrowness of windows for what journals are considered acceptable creates a lot of pressure to produce. I will keep this thread close to heart - cutting corners of rigor and relevance just "ain't worth it" if I read the takeaways of this commentary correctly.
Lori D. Kendall
At Journal of Business and Psychology we are trying to address the concerns in a small way
Our special feature on Null Results is coming out in a few months. This special feature presents excellent research that basically found "nothing". Ron Landis, Larry James, Chuck Pierce, and Chuck Lance led this effort.
Our special feature on inductive research is coming out this year. The special feature presents excellent work that captures interesting empirical insights while seeking to contribute to the building of theory, rather than testing theory. Ann Marie Ryan, Shelly Zedeck, Paul Spector, and Neal Schmitt led this effort.
In both cases, we learned that welcoming a broader spectrum of research not only serves to enhance the quality of our science, but logistically was highly feasible. Recognizing that scholarship advances come in many different sizes and shapes serves to allay (not eliminate sadly) many of the problems we have noted on this listserv from post hoc alterations of hypotheses, altering of data, HARKing, etc. These are not one-offs for us. These experiences have shaped our editorial perspectives going forward.
So many additional institutional steps are needed (e.g., not making tenure decisions dependent upon publishing in a very narrow set of "elite" journals), but small steps compounded across journals start to add up.
Steven
---------------------------------------------------------------------------------
Steven G. Rogelberg, PhD
University of North Carolina Charlotte University Professor
Professor, Organizational Science, Psychology, and Management
Director, Organizational Science | Editor, Journal of Business and Psychology
UNC Charlotte | Colvard 4025 | Friday 249
9201 University City Blvd. | Charlotte, NC 28223
Phone: 704-687-1351 | Fax: 704-687-1317
sgrogelb@uncc.edu/">sgrogelb@uncc.edu | http://www.orgscience.uncc.edu/sgrogelb/
Twitter: @stevenrogelberg
---------------------------------------------------------------------------------
From: Roni Reiter-Palmon <rreiter-palmon@UNOMAHA.EDU>
Reply-To: Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU>
Date: Friday, February 14, 2014 11:36 AM
To: "OB@AOMLISTS.PACE.EDU" <OB@AOMLISTS.PACE.EDU>
Subject: Re: Journal article retractions
I would also like to point your attention to the upcoming issue of the Psychology of Aesthetics, Creativity in the Arts (Feb 2014) which will have a special section on replications and how to address this issue including a focal article and responses from a number of reviewers and notable figures. While some of the content is specific to creativity, the reasoning and issues raised are not!
Roni
Roni Reiter-Palmon, Ph.D.
Isaacson Professor of I/O Psychology
Editor, The Psychology of Aesthetics, Creativity and the Arts
Director, I/O Psychology Graduate Program
Director of Innovation, Center for Collaboration Science
University of Nebraska at Omaha
Office location: ASH 347K
Phone: 402-554-4810
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
Sent: Friday, February 14, 2014 9:34 AM
To: OB@AOMLISTS.PACE.EDU
Subject: Re: [OB-LIST] Journal article retractions
Hi there
This paper (attached and forthcoming in JoM) has just been bought to my attention. It seems very relevant to this discussion.
O'Boyle, E. H. & Banks, G.C., & Gonzalez-Mule, E. (in press). The Chrysalis Effect: How ugly initial results metamorphosize into beautiful articles.
ABSTRACT: The issue of a published literature not representative of the population of research is most often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resulting journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically non-significant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label, the Chrysalis Effect.
Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath
Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
Sent: 13 February 2014 19:36
To: OB@AOMLISTS.PACE.EDU
Subject: Re: [OB-LIST] Journal article retractions
Hi Michael and everyone
I'm personally really pleased this is being discussed. Because of limited retractions, an ever-increasing proportion of hypotheses that are supported in published papers and the lack of replications it's hard to exaggerate how distorted the scientific process has become.
This a recent OB-ish example: http://retractionwatch.com/2013/09/19/fredrickson-losada-positivity-ratio-paper-partially-withdrawn/
A comment about (the lack of) retractions specifically in economics and business: http://retractionwatch.com/2012/12/12/why-arent-there-more-retractions-in-business-and-economics-journals/
Even The Economist (October 2013) has picked up on this: http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble
Best wishes
Rob
Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath
Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of alan.miller@UNLV.EDU
Sent: 13 February 2014 19:04
To: OB@AOMLISTS.PACE.EDU
Subject: [OB-LIST] Journal article retractions
Steve and Herman,
It should come as no surprise that there are a growing number of retractions, given the evidence my co-authors and I gathered in a paper we published in AMLE
Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010). Management Science on the Credibility Bubble: Cardinal Sins and Various Misdemeanors. Academy of Management Learning & Education, 9(4), 715-725.
Regards,
Alan
Alan N. Miller, Ph.D.
Professor of Management and Chairman
Department of Management, Entrepreneurship and Technology
Lee Business School
University of Nevada, Las Vegas
<image001.gif>"Aguinis, Herman" ---02/13/2014 07:22:59 AM---Steve, The answer to your question is yes. There is a site called "Retraction Watch," and they keep
From: "Aguinis, Herman" <haguinis@INDIANA.EDU>
To: OB@AOMLISTS.PACE.EDU
Date: 02/13/2014 07:22 AM
Subject: Re: [OB-LIST] Journal article retractions
Sent by: Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU>
Steve,
The answer to your question is yes. There is a site called "Retraction Watch," and they keep track of retractions across many scientific fields. See: http://retractionwatch.com/
I hope this helps,
Regards,
--Herman.
<image002.jpg>
Herman Aguinis, Ph.D.
John F. Mee Chair of Management
Professor of Organizational Behavior and Human Resources
Founding Director, Institute for Global Organizational Effectiveness
Department of Management and Entrepreneurship
http://mypage.iu.edu/~haguinis/
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Kelman, Steven
Sent: Tuesday, February 11, 2014 11:16 AM
To: OB@AOMLISTS.PACE.EDU
Subject: Re: [OB-LIST] Journal article retractions
Have there been retractions of OB papers?
Steve Kelman
Albert J. Weatherhead III and Richard W.
Weatherhead Professor of Public Management
Editor, International Public Management Journal
Tel: 617-496-6302
Personal Homepage: http://www.ksg.harvard.edu/fs/skelman
"The Lectern," my blog on FCW.com: http://fcw.com/blogs/lectern/list/blog-list.aspx
IPMJ Homepage: http://www.tandfonline.com/toc/upmj20/current
Like IPMJ on Facebook: <image003.jpg>
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
Sent: Tuesday, February 11, 2014 9:00 AM
To: OB@AOMLISTS.PACE.EDU
Subject: [OB-LIST] Journal article retractions
A modest proposal: Given recent journal article retractions, it would seem prudent for journals to require all data sets and syntax/scripts to made available to all on journal websites.
Mike
--
Michael A. McDaniel, Ph.D.
Professor - Human Resources and Organizational Behavior,
Department of Management,
Research Professor, Department of Psychology
Virginia Commonwealth University
301 West Main Street, PO Box 844000
Richmond, VA 23284-4000
http://www.people.vcu.edu/~mamcdani/
Voice: 804-827-0209
E-mail: mamcdani@VCU.edu
<image001.gif>
<image002.jpg>
<image003.jpg>