Discussion: View Thread

  • 1.  Simmer data on low heat until statistically significant

    Posted 03-20-2014 18:28

    One just has to like a guy who can credibly argue that 82% of the analyzed articles in an "A"  journal are likely based on questionable research practices or data suppression. Maybe he can do a management journal study next.


    http://www1.psych.purdue.edu/~gfrancis/Publications/Francis2014PBR.pdf

     

    Francis, G. (in press). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review.


    "Recent controversies have questioned the quality of scientific practice in the field of  psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009-2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings were suppressed, the experiments or analyses were improper, or that the theory does not properly account for the data. The analyses conclude problems for 82% (36 out of 44) of the articles in Psychological Science that have four or more experiments and could be analyzed."

    Mike
    --
    Michael A. McDaniel, Ph.D. - Professor, Department of Management, 
    Virginia Commonwealth University 301 West Main Street, PO Box 844000
    Richmond, VA 23284-4000
    voice: 804.827.0209
    Research Professor, VCU Department of Psychology


  • 2.  Simmer data on low heat until statistically significant

    Posted 03-21-2014 04:33

    Thanks Mike

     

    Very interesting study of publication bias.  Add to this all the other things we've discussed here including NHST, honorary authorships, the dropping of participants, hypothesizing after the results are known, and so on, and so on, it seems that we as a 'community' have a some profound ethical and scientific problems.  As Ernest O'Boyle put it in his Wall Street Journal interview about his JoM paper:  ""If practitioners can't trust what's coming out of academia, we don't have a reason to exist."

     

    I know some people are optimistic that things are happing to improve this – and perhaps it is happening in other fields – but in management and OB it just seems like (fishy) business as usual.  I'm not sure why.  Do most people think all this stuff is fine and just disagree that these problems really are problems?  Or is more that we can't think of easy ways of dealing with these problems so it's better to not do anything?

     

    I'd be really interested to know what others think.

     

    Cheers

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: 20 March 2014 22:28
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] Simmer data on low heat until statistically significant

     

    One just has to like a guy who can credibly argue that 82% of the analyzed articles in an "A"  journal are likely based on questionable research practices or data suppression. Maybe he can do a management journal study next.

     

    http://www1.psych.purdue.edu/~gfrancis/Publications/Francis2014PBR.pdf

     

    Francis, G. (in press). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review.

     

    "Recent controversies have questioned the quality of scientific practice in the field of  psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009-2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings were suppressed, the experiments or analyses were improper, or that the theory does not properly account for the data. The analyses conclude problems for 82% (36 out of 44) of the articles in Psychological Science that have four or more experiments and could be analyzed."

     

    Mike

    --

    Michael A. McDaniel, Ph.D. - Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000

    Richmond, VA 23284-4000

    voice: 804.827.0209

    Research Professor, VCU Department of Psychology



  • 3.  Simmer data on low heat until statistically significant

    Posted 03-22-2014 09:42

    For what it's worth, Uri Simonsohn – who is himself one of the leading figures trying to change how science is done/reported to be more transparent – has published a few papers disputing Francis' claims and techniques; Simonsohn's  articles are worth reading just so you can find out the story behind "Francis' tests are like a young Swedish couple", which Simonsohn uses as a section header!  The article titles (copied from his website) and links to the articles on his website are below. 

     

    My take on Simonsohn's responses is that he seems to be of the opinion "yes, there's problems in the lit; no, Francis' analyses are not accurate." I'll state up front that I haven't read Francis' 2014 PBR paper in depth, so it's possible (hopefully probable) his new analysis addresses these previous critiques of Simonsohn's. 

     

     

    Simonsohn (2012) "It Does Not Follow: Evaluating the One-Off Publication Bias Critiques by Francis (2012a,b,c,d,e,f), Perspectives on Psychological Science, V7(6),  597-599 - http://opim.wharton.upenn.edu/~uws/papers/follow.pdf

     

    Simonsohn (2013)  "It Really Just Does Not Follow, Comments on Francis (2013)", invited commentary for the Journal of Mathematical Psychology, V57(5) p.174-176 - http://opim.wharton.upenn.edu/~uws/papers/not_follow_2.pdf

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
    Sent: Friday, March 21, 2014 4:33 AM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Simmer data on low heat until statistically significant

     

    Thanks Mike

     

    Very interesting study of publication bias.  Add to this all the other things we've discussed here including NHST, honorary authorships, the dropping of participants, hypothesizing after the results are known, and so on, and so on, it seems that we as a 'community' have a some profound ethical and scientific problems.  As Ernest O'Boyle put it in his Wall Street Journal interview about his JoM paper:  ""If practitioners can't trust what's coming out of academia, we don't have a reason to exist."

     

    I know some people are optimistic that things are happing to improve this – and perhaps it is happening in other fields – but in management and OB it just seems like (fishy) business as usual.  I'm not sure why.  Do most people think all this stuff is fine and just disagree that these problems really are problems?  Or is more that we can't think of easy ways of dealing with these problems so it's better to not do anything?

     

    I'd be really interested to know what others think.

     

    Cheers

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: 20 March 2014 22:28
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] Simmer data on low heat until statistically significant

     

    One just has to like a guy who can credibly argue that 82% of the analyzed articles in an "A"  journal are likely based on questionable research practices or data suppression. Maybe he can do a management journal study next.

     

    http://www1.psych.purdue.edu/~gfrancis/Publications/Francis2014PBR.pdf

     

    Francis, G. (in press). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review.

     

    "Recent controversies have questioned the quality of scientific practice in the field of  psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009-2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings were suppressed, the experiments or analyses were improper, or that the theory does not properly account for the data. The analyses conclude problems for 82% (36 out of 44) of the articles in Psychological Science that have four or more experiments and could be analyzed."

     

    Mike

    --

    Michael A. McDaniel, Ph.D. - Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000

    Richmond, VA 23284-4000

    voice: 804.827.0209

    Research Professor, VCU Department of Psychology



  • 4.  Simmer data on low heat until statistically significant

    Posted 03-22-2014 17:38

    Here are links to papers by Francis that address critics:

    http://pps.sagepub.com/content/7/6/597.short/reply#sppps_el_74

    Francis, G. (2013). We should focus on the biases that matter: A reply to commentaries. Journal of Mathematical Psychology, 57, 190-195.
    http://www.sciencedirect.com/science/article/pii/S0022249613000527?_rdoc=1&_fmt=high&_origin=gateway&_docanchor=&md5=b8429449ccfc9c30159a5f9aeaa92ffb

    http://www.sciencedirect.com/science/article/pii/S002224961300014X

    --
    Michael A. McDaniel, Ph.D.
    Professor,  - Department of Management, 
    Virginia Commonwealth University
    301 West Main Street, PO Box 844000
    Richmond, VA 23284-4000
    http://www.people.vcu.edu/~mamcdani/
    Voice: 804-827-0209
    E-mail: mamcdani@VCU.edu

    Research Professor, Department of Psychology
    President-Elect, International Society of Intelligence Research

    On Mar 22, 2014 10:00 AM, "Lance Ferris" <duf14@smeal.psu.edu> wrote:

    For what it’s worth, Uri Simonsohn – who is himself one of the leading figures trying to change how science is done/reported to be more transparent – has published a few papers disputing Francis’ claims and techniques; Simonsohn’s  articles are worth reading just so you can find out the story behind “Francis’ tests are like a young Swedish couple”, which Simonsohn uses as a section header!  The article titles (copied from his website) and links to the articles on his website are below. 

     

    My take on Simonsohn’s responses is that he seems to be of the opinion “yes, there’s problems in the lit; no, Francis’ analyses are not accurate.” I’ll state up front that I haven’t read Francis’ 2014 PBR paper in depth, so it’s possible (hopefully probable) his new analysis addresses these previous critiques of Simonsohn’s. 

     

     

    Simonsohn (2012) "It Does Not Follow: Evaluating the One-Off Publication Bias Critiques by Francis (2012a,b,c,d,e,f), Perspectives on Psychological Science, V7(6),  597-599 - http://opim.wharton.upenn.edu/~uws/papers/follow.pdf

     

    Simonsohn (2013)  "It Really Just Does Not Follow, Comments on Francis (2013)", invited commentary for the Journal of Mathematical Psychology, V57(5) p.174-176 - http://opim.wharton.upenn.edu/~uws/papers/not_follow_2.pdf

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
    Sent: Friday, March 21, 2014 4:33 AM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Simmer data on low heat until statistically significant

     

    Thanks Mike

     

    Very interesting study of publication bias.  Add to this all the other things we’ve discussed here including NHST, honorary authorships, the dropping of participants, hypothesizing after the results are known, and so on, and so on, it seems that we as a ‘community’ have a some profound ethical and scientific problems.  As Ernest O’Boyle put it in his Wall Street Journal interview about his JoM paper:  ““If practitioners can’t trust what’s coming out of academia, we don’t have a reason to exist.”

     

    I know some people are optimistic that things are happing to improve this – and perhaps it is happening in other fields – but in management and OB it just seems like (fishy) business as usual.  I’m not sure why.  Do most people think all this stuff is fine and just disagree that these problems really are problems?  Or is more that we can’t think of easy ways of dealing with these problems so it’s better to not do anything?

     

    I’d be really interested to know what others think.

     

    Cheers

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: 20 March 2014 22:28
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] Simmer data on low heat until statistically significant

     

    One just has to like a guy who can credibly argue that 82% of the analyzed articles in an "A"  journal are likely based on questionable research practices or data suppression. Maybe he can do a management journal study next.

     

    http://www1.psych.purdue.edu/~gfrancis/Publications/Francis2014PBR.pdf

     

    Francis, G. (in press). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review.

     

    "Recent controversies have questioned the quality of scientific practice in the field of  psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009-2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings were suppressed, the experiments or analyses were improper, or that the theory does not properly account for the data. The analyses conclude problems for 82% (36 out of 44) of the articles in Psychological Science that have four or more experiments and could be analyzed."

     

    Mike

    --

    Michael A. McDaniel, Ph.D. - Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000

    Richmond, VA 23284-4000

    voice: 804.827.0209

    Research Professor, VCU Department of Psychology



  • 5.  Simmer data on low heat until statistically significant

    Posted 03-26-2014 08:10

    Another perspective:

    http://www.epjournal.net/blog/2014/03/excess-success-research-practices-in-psychology/

    --
    Michael A. McDaniel, Ph.D.
    Professor,  - Department of Management, 
    Virginia Commonwealth University
    301 West Main Street, PO Box 844000
    Richmond, VA 23284-4000
    http://www.people.vcu.edu/~mamcdani/
    Voice: 804-827-0209
    E-mail: mamcdani@VCU.edu

    Research Professor, Department of Psychology
    President-Elect, International Society of Intelligence Research

    On Mar 22, 2014 10:00 AM, "Lance Ferris" <duf14@smeal.psu.edu> wrote:

    For what it’s worth, Uri Simonsohn – who is himself one of the leading figures trying to change how science is done/reported to be more transparent – has published a few papers disputing Francis’ claims and techniques; Simonsohn’s  articles are worth reading just so you can find out the story behind “Francis’ tests are like a young Swedish couple”, which Simonsohn uses as a section header!  The article titles (copied from his website) and links to the articles on his website are below. 

     

    My take on Simonsohn’s responses is that he seems to be of the opinion “yes, there’s problems in the lit; no, Francis’ analyses are not accurate.” I’ll state up front that I haven’t read Francis’ 2014 PBR paper in depth, so it’s possible (hopefully probable) his new analysis addresses these previous critiques of Simonsohn’s. 

     

     

    Simonsohn (2012) "It Does Not Follow: Evaluating the One-Off Publication Bias Critiques by Francis (2012a,b,c,d,e,f), Perspectives on Psychological Science, V7(6),  597-599 - http://opim.wharton.upenn.edu/~uws/papers/follow.pdf

     

    Simonsohn (2013)  "It Really Just Does Not Follow, Comments on Francis (2013)", invited commentary for the Journal of Mathematical Psychology, V57(5) p.174-176 - http://opim.wharton.upenn.edu/~uws/papers/not_follow_2.pdf

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
    Sent: Friday, March 21, 2014 4:33 AM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Simmer data on low heat until statistically significant

     

    Thanks Mike

     

    Very interesting study of publication bias.  Add to this all the other things we’ve discussed here including NHST, honorary authorships, the dropping of participants, hypothesizing after the results are known, and so on, and so on, it seems that we as a ‘community’ have a some profound ethical and scientific problems.  As Ernest O’Boyle put it in his Wall Street Journal interview about his JoM paper:  ““If practitioners can’t trust what’s coming out of academia, we don’t have a reason to exist.”

     

    I know some people are optimistic that things are happing to improve this – and perhaps it is happening in other fields – but in management and OB it just seems like (fishy) business as usual.  I’m not sure why.  Do most people think all this stuff is fine and just disagree that these problems really are problems?  Or is more that we can’t think of easy ways of dealing with these problems so it’s better to not do anything?

     

    I’d be really interested to know what others think.

     

    Cheers

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    Vice-Chair Academic Council | Center for Evidence-Based Management (www.cebma.org)

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: 20 March 2014 22:28
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] Simmer data on low heat until statistically significant

     

    One just has to like a guy who can credibly argue that 82% of the analyzed articles in an "A"  journal are likely based on questionable research practices or data suppression. Maybe he can do a management journal study next.

     

    http://www1.psych.purdue.edu/~gfrancis/Publications/Francis2014PBR.pdf

     

    Francis, G. (in press). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review.

     

    "Recent controversies have questioned the quality of scientific practice in the field of  psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009-2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings were suppressed, the experiments or analyses were improper, or that the theory does not properly account for the data. The analyses conclude problems for 82% (36 out of 44) of the articles in Psychological Science that have four or more experiments and could be analyzed."

     

    Mike

    --

    Michael A. McDaniel, Ph.D. - Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000

    Richmond, VA 23284-4000

    voice: 804.827.0209

    Research Professor, VCU Department of Psychology