Discussion: View Thread

Expand all | Collapse all

Comfortable in our delusions

  • 1.  Comfortable in our delusions

    Posted 10-02-2011 12:01
    Although organizational science researchers remain comfortable in their likely delusional beliefs concerning the absence of publication bias in our literature, I thought this TED talk may be of interest to the less delusional: http://www.ted.com/talks/ben_goldacre_battling_bad_science.html

    Best wishes,

    Mike
    Michael A. McDaniel, Ph.D.
    Professor - Human Resources and
    Organizational Behavior
    Department of Management, Research Professor, Department of Psychology
    Virginia Commonwealth University
    301 West Main Street, , PO Box 844000
    Richmond, VA 23284-4000

    http://www.people.vcu.edu/~mamcdani/
    voice: 804.827.0209
    e-mail:     MAMcDani@vcu.edu
    skype: MichaelAMcDaniel

    		Doctoral Program in Management

    The Management Department of the VCU School of Business offers a Ph.D. in Business. Participating faculty with research interests in OB and HR include: Ron Humphrey, Sven Kepes, Michael McDaniel, In-Sue Oh, Doug Pugh, & Anson Seers.

    Students with interest in the doctoral program, should contact Anson Seers.




  • 2.  Comfortable in our delusions

    Posted 10-02-2011 21:58
     Very funny, Michael.  But you know something?  We're not medical researchers.  And we don't shield real results from "dayllight" in the way that the "comedian" suggested.  And finally, we don't do the same type of research.  So what was the point?  If I'm ever funded by a drug company, I'll be very careful.  But what else?  I'm not really getting it.  --  Gayle
     
     
    Gayle Baugh
    Associate Professor
    Co-Editor, Research in Careers Series
      published by Information Age Publishing
    Associate Editor, Group & Organization
      Management
    Department of Management & MIS
    University of West Florida
    11000 University Parkway
    Pensacola, Florida  32514-5752
    (850) 474-2206  (Office)
    (850) 474-2314  (FAX)
    gbaugh@uwf.edu



  • 3.  Comfortable in our delusions

    Posted 10-03-2011 02:17
    Don't publications fetch us, the academicians, some pecuniary benefits such as salary hike?
     
    Regarding the publication bias: many times the manuscript submitted to a journal is available on websites in the form of conference presentations/proceedings or working papers or..This means reviewers can easily know or find who the authors are. Is it time we should question the "blindness" in our blind review process?

    George



    On Mon, Oct 3, 2011 at 7:28 AM, Gayle Baugh <gbaugh@uwf.edu> wrote:
     Very funny, Michael.  But you know something?  We're not medical researchers.  And we don't shield real results from "dayllight" in the way that the "comedian" suggested.  And finally, we don't do the same type of research.  So what was the point?  If I'm ever funded by a drug company, I'll be very careful.  But what else?  I'm not really getting it.  --  Gayle
     
     
    Gayle Baugh
    Associate Professor
    Co-Editor, Research in Careers Series
      published by Information Age Publishing
    Associate Editor, Group & Organization
      Management
    Department of Management & MIS
    University of West Florida
    11000 University Parkway
    Pensacola, Florida  32514-5752
    (850) 474-2206  (Office)
    (850) 474-2314  (FAX)
    gbaugh@uwf.edu




  • 4.  Comfortable in our delusions

    Posted 10-03-2011 08:45
    But we do stop null results from being published. A lot. I think that was the point, and it's a valid one.

    Sent from my iPad

    On Oct 2, 2011, at 10:05 PM, "Gayle Baugh" <gbaugh@UWF.EDU<mailto:gbaugh@UWF.EDU>> wrote:

    Very funny, Michael. But you know something? We're not medical researchers. And we don't shield real results from "dayllight" in the way that the "comedian" suggested. And finally, we don't do the same type of research. So what was the point? If I'm ever funded by a drug company, I'll be very careful. But what else? I'm not really getting it. -- Gayle


    Gayle Baugh
    Associate Professor
    Co-Editor, Research in Careers Series
    published by Information Age Publishing
    Associate Editor, Group & Organization
    Management
    Department of Management & MIS
    University of West Florida
    11000 University Parkway
    Pensacola, Florida 32514-5752
    (850) 474-2206 (Office)
    (850) 474-2314 (FAX)
    <mailto:gbaugh@uwf.edu>gbaugh@uwf.edu<mailto:gbaugh@uwf.edu>


  • 5.  Comfortable in our delusions

    Posted 10-03-2011 12:28

    Hi Gayle

     

    No, I guess we don't shield it in the same way as drug companies might but we do have many other collective and institutional ways of ensuring some or most of our data never sees the light of day.  In particular, it's pretty difficult to publish single studies with statistically non-significant results* or results which seem to be the reverse of those previously found.  It seems likely that in many (most?) cases the complete body of data around a particular question (like having complete data from all clinical trials of a drug) would not support the prevailing view given how much the need to publish certain kinds of papers (rather than the need to report all results found) drives researcher behaviour.

     

    Not sure if that helps you to "get it" or to push it even further out of reach!

     

    Cheers

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

     

    *Not to mention the issues with null hypothesis significance testing

     

    From: Rob Briner [mailto:r.briner@bbk.ac.uk]
    Sent: 03 October 2011 04:05
    To: Rob Briner
    Subject: FW: [OB-LIST] Comfortable in our delusions

     


    -------------------------------------------
    From:     Organizational Behavior Division Listserv on behalf of Gayle Baugh[SMTP:GBAUGH@UWF.EDU]
    Sent:     Monday, October 03, 2011 2:58:16 AM
    To:     OB@AOMLISTS.PACE.EDU
    Subject:     Re: [OB-LIST] Comfortable in our delusions
    Auto forwarded by a Rule

     

     Very funny, Michael.  But you know something?  We're not medical researchers.  And we don't shield real results from "dayllight" in the way that the "comedian" suggested.  And finally, we don't do the same type of research.  So what was the point?  If I'm ever funded by a drug company, I'll be very careful.  But what else?  I'm not really getting it.  --  Gayle

     

     

    Gayle Baugh
    Associate Professor
    Co-Editor, Research in Careers Series
      published by Information Age Publishing
    Associate Editor, Group & Organization
      Management
    Department of Management & MIS
    University of West Florida
    11000 University Parkway
    Pensacola, Florida  32514-5752
    (850) 474-2206  (Office)
    (850) 474-2314  (FAX)
    gbaugh@uwf.edu



  • 6.  Comfortable in our delusions

    Posted 10-03-2011 14:19

    Gayle,

    Well, this what I took away from it, and it was interesting because I've been having the same thoughts recently while working on a broad literature review.  The main issue is of the talk was **bad science.** Bad science leads to scientific claims made without providing full or even accurate information.  But bad science  doesn't just mean not reporting data, it also means ignoring (hiding) data that might be in plain sight when it doesn't agree with one's agenda. It also means creatively citing or presenting data that in the end leads to an inaccurate portrayal of the evidence.  Various issues can include:

    1) Author bias:  Conduct a lab study and fail to support a hypothesis or find the opposite. Try again and get support for the hypothesis.  What will be reported? Study 2 only? Or both study 1 and 2?  Test a predictor in relation to five outcomes and one is outcome is related and the other 4 are not.  What will be reported?

    2) Publication Bias: Journals only publish that which is statistically significant.  It's rare to see a paper that completely fails to support some general hypothesis get published even when there might be good reason to publish it.

    3) Cherry picking citations in manuscripts or reviews.  Citing a paper or two that agree with one's putative hypothesis or perspective and ignoring the preponderance of studies that fail to support it (nonsignificant findings do get published if embedded in a larger paper where some key relation may be  significant and certain disciplines are more likely to publish null or contrary effects).  This is especially problematic given the great reliance on convenience samples.  This is especially true when vested interests are involved--i.e., there exists a product, service, or ideology to hawk.  

    4) In the absence of papers to support some tenet in developing the rationale for an hypothesis, cite any paper in the area that by its title could be plausible support.  I've seen my own papers cited to support a relation when my study did not assess any of the constructs, sometimes after the citing paper was published.  Because I know a few literature very well, I've also found this to be true of other articles, not written by myself, cited to support a relation when they did not assess the variables in question.  Accident?  Carelessness? Perhaps sometimes, but not likely always.  I've seen single manuscripts that had many such accidents.  Moreover, none of the other reviewers picked up on these citation errors.  Then the next researcher who is a bit lazy cites the study to support a relation that it cannot support and the error proliferates.

    5) Findings are cited that can't be traced to any actual study.  They are cited because it supports one's perspective and they have been cited by others many times.  Or present data in such a way that is very misleading.  These fictitious data or distorted presentations often start in trade magazines and find their way into "scientific" papers because one can cite the trade article.

    6) Make a big deal over a statistically significant effect even though it may have absolutely no practical value.  For example, one might read in a review that X has been shown to lead to impaired psychomotor performance.  It is this general claims that develops a life of its own.  But go back and look at the literature, and one might find how incredibly small the effects really are, and this might even have been pointed out by the primary researchers. But statistical significance can serve an agenda, even when effects size does not.   So the latter fall by the roadside.

    And the list could go on.  #1 is hard to ever know.  The rest I've seen during years of reading, manuscript reviewing, and presently while working on a broad review of the workforce and workplace substance use literature.  Given the topic of the review, the level of data and evidence massaging in scientific and nonscientific outlets isn't surprising because there are strong ideologies and billions of dollars in consulting and product sales at stake.  And for the nonscientists, much information comes from the internet, which can be a dangerous source of information fro the naive.

    Most research is honest and any errors are honest.  But to suggest that what is published and not published in a general area, including OB. HRM, or I/O, and which studies are funded and which are not funded is not touched by politics, vested interests, ideology, or reward systems is just not congruent with reality. I guess this is the delusion we often would like to hang on to.

    Mike Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office:    716-887-2519
    Fax:        716-887-2477
    E-mail:     frone@ria.buffalo.edu
    Internet:     http://www.ria.buffalo.edu/profiles/frone.html
    ***************************************************************

     

    >  Very funny, Michael.  But you know something?  We're not medical
    > researchers.  And we don't shield real results from "dayllight" in
    > the way that the "comedian" suggested.  And finally, we don't do the
    > same type of research.  So what was the point?  If I'm ever funded
    > by a drug company, I'll be very careful.  But what else?  I'm not
    > really getting it.  --  Gayle
    >  
    >  
    > Gayle Baugh
    > Associate Professor
    > Co-Editor, Research in Careers Series
    >   published by Information Age Publishing
    > Associate Editor, Group & Organization
    >   Management
    > Department of Management & MIS
    > University of West Florida
    > 11000 University Parkway
    > Pensacola, Florida  32514-5752
    > (850) 474-2206  (Office)
    > (850) 474-2314  (FAX)
    > gbaugh@uwf.edu


  • 7.  Comfortable in our delusions

    Posted 10-03-2011 15:52

    Hi Mike

     

    A great, though depressing, list.  I keep wondering why we don't view this collectively as an ethical failure and do more, also collectively to address and start to resolve such issues.

     

    I was interested that you'd come across these issues while doing a broad literature review.  I think you get quite a different perspective about claims made both in individual papers and by communities of scholars when standing back and conducting reviews (including systematic reviews).  Received wisdom often turns out to be not so wise.

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    -------------------------------------------
    From:     Organizational Behavior Division Listserv on behalf of Michael Frone[SMTP:FRONE@RIA.BUFFALO.EDU]
    Sent:     Monday, October 03, 2011 7:18:41 PM
    To:     OB@AOMLISTS.PACE.EDU
    Subject:     Re: [OB-LIST] Comfortable in our delusions
    Auto forwarded by a Rule



    Gayle,

    Well, this what I took away from it, and it was interesting because I've been having the same thoughts recently while working on a broad literature review.  The main issue is of the talk was **bad science.** Bad science leads to scientific claims made without providing full or even accurate information.  But bad science  doesn't just mean not reporting data, it also means ignoring (hiding) data that might be in plain sight when it doesn't agree with one's agenda. It also means creatively citing or presenting data that in the end leads to an inaccurate portrayal of the evidence.  Various issues can include:

    1) Author bias:  Conduct a lab study and fail to support a hypothesis or find the opposite. Try again and get support for the hypothesis.  What will be reported? Study 2 only? Or both study 1 and 2?  Test a predictor in relation to five outcomes and one is outcome is related and the other 4 are not.  What will be reported?

    2) Publication Bias: Journals only publish that which is statistically significant.  It's rare to see a paper that completely fails to support some general hypothesis get published even when there might be good reason to publish it.

    3) Cherry picking citations in manuscripts or reviews.  Citing a paper or two that agree with one's putative hypothesis or perspective and ignoring the preponderance of studies that fail to support it (nonsignificant findings do get published if embedded in a larger paper where some key relation may be  significant and certain disciplines are more likely to publish null or contrary effects).  This is especially problematic given the great reliance on convenience samples.  This is especially true when vested interests are involved--i.e., there exists a product, service, or ideology to hawk.  

    4) In the absence of papers to support some tenet in developing the rationale for an hypothesis, cite any paper in the area that by its title could be plausible support.  I've seen my own papers cited to support a relation when my study did not assess any of the constructs, sometimes after the citing paper was published.  Because I know a few literature very well, I've also found this to be true of other articles, not written by myself, cited to support a relation when they did not assess the variables in question.  Accident?  Carelessness? Perhaps sometimes, but not likely always.  I've seen single manuscripts that had many such accidents.  Moreover, none of the other reviewers picked up on these citation errors.  Then the next researcher who is a bit lazy cites the study to support a relation that it cannot support and the error proliferates.

    5) Findings are cited that can't be traced to any actual study.  They are cited because it supports one's perspective and they have been cited by others many times.  Or present data in such a way that is very misleading.  These fictitious data or distorted presentations often start in trade magazines and find their way into "scientific" papers because one can cite the trade article.

    6) Make a big deal over a statistically significant effect even though it may have absolutely no practical value.  For example, one might read in a review that X has been shown to lead to impaired psychomotor performance.  It is this general claims that develops a life of its own.  But go back and look at the literature, and one might find how incredibly small the effects really are, and this might even have been pointed out by the primary researchers. But statistical significance can serve an agenda, even when effects size does not.   So the latter fall by the roadside.

    And the list could go on.  #1 is hard to ever know.  The rest I've seen during years of reading, manuscript reviewing, and presently while working on a broad review of the workforce and workplace substance use literature.  Given the topic of the review, the level of data and evidence massaging in scientific and nonscientific outlets isn't surprising because there are strong ideologies and billions of dollars in consulting and product sales at stake.  And for the nonscientists, much information comes from the internet, which can be a dangerous source of information fro the naive.

    Most research is honest and any errors are honest.  But to suggest that what is published and not published in a general area, including OB. HRM, or I/O, and which studies are funded and which are not funded is not touched by politics, vested interests, ideology, or reward systems is just not congruent with reality. I guess this is the delusion we often would like to hang on to.

    Mike Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office:    716-887-2519
    Fax:        716-887-2477
    E-mail:     frone@ria.buffalo.edu
    Internet:     http://www.ria.buffalo.edu/profiles/frone.html
    ***************************************************************

     

    >  Very funny, Michael.  But you know something?  We're not medical
    > researchers.  And we don't shield real results from "dayllight" in
    > the way that the "comedian" suggested.  And finally, we don't do the
    > same type of research.  So what was the point?  If I'm ever funded
    > by a drug company, I'll be very careful.  But what else?  I'm not
    > really getting it.  --  Gayle
    >  
    >  
    > Gayle Baugh
    > Associate Professor
    > Co-Editor, Research in Careers Series
    >   published by Information Age Publishing
    > Associate Editor, Group & Organization
    >   Management
    > Department of Management & MIS
    > University of West Florida
    > 11000 University Parkway
    > Pensacola, Florida  32514-5752
    > (850) 474-2206  (Office)
    > (850) 474-2314  (FAX)
    > gbaugh@uwf.edu



  • 8.  Comfortable in our delusions

    Posted 10-03-2011 18:56

    Hi all,

     

    I happen to be reading this guy's book at the moment so I'll throw in what I know from the book, not from the video.  This guy is a fan of evidence-based medicine, and he likes to expose the quacks.  He exposes dubious claims from homeopathy, the cosmetics industry, nutritionists, etc.  They use science-like language but not rigorous scientific methods. 

     

    Our methods are by no means perfect but we aren't quacks either.  If the quality control is so lax in our journals, why are we not publishing more?  

     

    I know that not all non-findings are published but we do publish papers with unsupported Hypotheses (admittedly as long as something else is supported). 

     

    Thanks for reading,

     

    Laura

     

     

    Dr. Laura Guerrero

    Assistant Professor of Management

    Marketing & Management Department

    College of Business Administration

    University of Texas at El Paso

    500 W. University Avenue

    El Paso, TX 79968-0539

    915-747-5014

    lguerrero5@utep.edu

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
    Sent: Monday, October 03, 2011 1:52 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions

     

    Hi Mike

     

    A great, though depressing, list.  I keep wondering why we don't view this collectively as an ethical failure and do more, also collectively to address and start to resolve such issues.

     

    I was interested that you'd come across these issues while doing a broad literature review.  I think you get quite a different perspective about claims made both in individual papers and by communities of scholars when standing back and conducting reviews (including systematic reviews).  Received wisdom often turns out to be not so wise.

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    -------------------------------------------
    From:     Organizational Behavior Division Listserv on behalf of Michael Frone[SMTP:FRONE@RIA.BUFFALO.EDU]
    Sent:     Monday, October 03, 2011 7:18:41 PM
    To:     OB@AOMLISTS.PACE.EDU
    Subject:     Re: [OB-LIST] Comfortable in our delusions
    Auto forwarded by a Rule



    Gayle,

    Well, this what I took away from it, and it was interesting because I've been having the same thoughts recently while working on a broad literature review.  The main issue is of the talk was **bad science.** Bad science leads to scientific claims made without providing full or even accurate information.  But bad science  doesn't just mean not reporting data, it also means ignoring (hiding) data that might be in plain sight when it doesn't agree with one's agenda. It also means creatively citing or presenting data that in the end leads to an inaccurate portrayal of the evidence.  Various issues can include:

    1) Author bias:  Conduct a lab study and fail to support a hypothesis or find the opposite. Try again and get support for the hypothesis.  What will be reported? Study 2 only? Or both study 1 and 2?  Test a predictor in relation to five outcomes and one is outcome is related and the other 4 are not.  What will be reported?

    2) Publication Bias: Journals only publish that which is statistically significant.  It's rare to see a paper that completely fails to support some general hypothesis get published even when there might be good reason to publish it.

    3) Cherry picking citations in manuscripts or reviews.  Citing a paper or two that agree with one's putative hypothesis or perspective and ignoring the preponderance of studies that fail to support it (nonsignificant findings do get published if embedded in a larger paper where some key relation may be  significant and certain disciplines are more likely to publish null or contrary effects).  This is especially problematic given the great reliance on convenience samples.  This is especially true when vested interests are involved--i.e., there exists a product, service, or ideology to hawk.  

    4) In the absence of papers to support some tenet in developing the rationale for an hypothesis, cite any paper in the area that by its title could be plausible support.  I've seen my own papers cited to support a relation when my study did not assess any of the constructs, sometimes after the citing paper was published.  Because I know a few literature very well, I've also found this to be true of other articles, not written by myself, cited to support a relation when they did not assess the variables in question.  Accident?  Carelessness? Perhaps sometimes, but not likely always.  I've seen single manuscripts that had many such accidents.  Moreover, none of the other reviewers picked up on these citation errors.  Then the next researcher who is a bit lazy cites the study to support a relation that it cannot support and the error proliferates.

    5) Findings are cited that can't be traced to any actual study.  They are cited because it supports one's perspective and they have been cited by others many times.  Or present data in such a way that is very misleading.  These fictitious data or distorted presentations often start in trade magazines and find their way into "scientific" papers because one can cite the trade article.

    6) Make a big deal over a statistically significant effect even though it may have absolutely no practical value.  For example, one might read in a review that X has been shown to lead to impaired psychomotor performance.  It is this general claims that develops a life of its own.  But go back and look at the literature, and one might find how incredibly small the effects really are, and this might even have been pointed out by the primary researchers. But statistical significance can serve an agenda, even when effects size does not.   So the latter fall by the roadside.

    And the list could go on.  #1 is hard to ever know.  The rest I've seen during years of reading, manuscript reviewing, and presently while working on a broad review of the workforce and workplace substance use literature.  Given the topic of the review, the level of data and evidence massaging in scientific and nonscientific outlets isn't surprising because there are strong ideologies and billions of dollars in consulting and product sales at stake.  And for the nonscientists, much information comes from the internet, which can be a dangerous source of information fro the naive.

    Most research is honest and any errors are honest.  But to suggest that what is published and not published in a general area, including OB. HRM, or I/O, and which studies are funded and which are not funded is not touched by politics, vested interests, ideology, or reward systems is just not congruent with reality. I guess this is the delusion we often would like to hang on to.

    Mike Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office:    716-887-2519
    Fax:        716-887-2477
    E-mail:     frone@ria.buffalo.edu
    Internet:     http://www.ria.buffalo.edu/profiles/frone.html
    ***************************************************************

     

    >  Very funny, Michael.  But you know something?  We're not medical
    > researchers.  And we don't shield real results from "dayllight" in
    > the way that the "comedian" suggested.  And finally, we don't do the
    > same type of research.  So what was the point?  If I'm ever funded
    > by a drug company, I'll be very careful.  But what else?  I'm not
    > really getting it.  --  Gayle
    >  
    >  
    > Gayle Baugh
    > Associate Professor
    > Co-Editor, Research in Careers Series
    >   published by Information Age Publishing
    > Associate Editor, Group & Organization
    >   Management
    > Department of Management & MIS
    > University of West Florida
    > 11000 University Parkway
    > Pensacola, Florida  32514-5752
    > (850) 474-2206  (Office)
    > (850) 474-2314  (FAX)
    > gbaugh@uwf.edu



  • 9.  Comfortable in our delusions

    Posted 10-03-2011 22:06

    Orson Welles is reputed to have said: There is no passion so keen as that of one person to edit another's draft".  That may be the reason we are not publishing more.

     

    Best,

     

    G.

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Guerrero, Laura
    Sent: Monday, October 03, 2011 3:56 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions

     

    Hi all,

     

    I happen to be reading this guy's book at the moment so I'll throw in what I know from the book, not from the video.  This guy is a fan of evidence-based medicine, and he likes to expose the quacks.  He exposes dubious claims from homeopathy, the cosmetics industry, nutritionists, etc.  They use science-like language but not rigorous scientific methods. 

     

    Our methods are by no means perfect but we aren't quacks either.  If the quality control is so lax in our journals, why are we not publishing more?  

     

    I know that not all non-findings are published but we do publish papers with unsupported Hypotheses (admittedly as long as something else is supported). 

     

    Thanks for reading,

     

    Laura

     

     

    Dr. Laura Guerrero

    Assistant Professor of Management

    Marketing & Management Department

    College of Business Administration

    University of Texas at El Paso

    500 W. University Avenue

    El Paso, TX 79968-0539

    915-747-5014

    lguerrero5@utep.edu

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
    Sent: Monday, October 03, 2011 1:52 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions

     

    Hi Mike

     

    A great, though depressing, list.  I keep wondering why we don't view this collectively as an ethical failure and do more, also collectively to address and start to resolve such issues.

     

    I was interested that you'd come across these issues while doing a broad literature review.  I think you get quite a different perspective about claims made both in individual papers and by communities of scholars when standing back and conducting reviews (including systematic reviews).  Received wisdom often turns out to be not so wise.

     

    Rob

     

    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    -------------------------------------------
    From:     Organizational Behavior Division Listserv on behalf of Michael Frone[SMTP:FRONE@RIA.BUFFALO.EDU]
    Sent:     Monday, October 03, 2011 7:18:41 PM
    To:     OB@AOMLISTS.PACE.EDU
    Subject:     Re: [OB-LIST] Comfortable in our delusions
    Auto forwarded by a Rule



    Gayle,

    Well, this what I took away from it, and it was interesting because I've been having the same thoughts recently while working on a broad literature review.  The main issue is of the talk was **bad science.** Bad science leads to scientific claims made without providing full or even accurate information.  But bad science  doesn't just mean not reporting data, it also means ignoring (hiding) data that might be in plain sight when it doesn't agree with one's agenda. It also means creatively citing or presenting data that in the end leads to an inaccurate portrayal of the evidence.  Various issues can include:

    1) Author bias:  Conduct a lab study and fail to support a hypothesis or find the opposite. Try again and get support for the hypothesis.  What will be reported? Study 2 only? Or both study 1 and 2?  Test a predictor in relation to five outcomes and one is outcome is related and the other 4 are not.  What will be reported?

    2) Publication Bias: Journals only publish that which is statistically significant.  It's rare to see a paper that completely fails to support some general hypothesis get published even when there might be good reason to publish it.

    3) Cherry picking citations in manuscripts or reviews.  Citing a paper or two that agree with one's putative hypothesis or perspective and ignoring the preponderance of studies that fail to support it (nonsignificant findings do get published if embedded in a larger paper where some key relation may be  significant and certain disciplines are more likely to publish null or contrary effects).  This is especially problematic given the great reliance on convenience samples.  This is especially true when vested interests are involved--i.e., there exists a product, service, or ideology to hawk.  

    4) In the absence of papers to support some tenet in developing the rationale for an hypothesis, cite any paper in the area that by its title could be plausible support.  I've seen my own papers cited to support a relation when my study did not assess any of the constructs, sometimes after the citing paper was published.  Because I know a few literature very well, I've also found this to be true of other articles, not written by myself, cited to support a relation when they did not assess the variables in question.  Accident?  Carelessness? Perhaps sometimes, but not likely always.  I've seen single manuscripts that had many such accidents.  Moreover, none of the other reviewers picked up on these citation errors.  Then the next researcher who is a bit lazy cites the study to support a relation that it cannot support and the error proliferates.

    5) Findings are cited that can't be traced to any actual study.  They are cited because it supports one's perspective and they have been cited by others many times.  Or present data in such a way that is very misleading.  These fictitious data or distorted presentations often start in trade magazines and find their way into "scientific" papers because one can cite the trade article.

    6) Make a big deal over a statistically significant effect even though it may have absolutely no practical value.  For example, one might read in a review that X has been shown to lead to impaired psychomotor performance.  It is this general claims that develops a life of its own.  But go back and look at the literature, and one might find how incredibly small the effects really are, and this might even have been pointed out by the primary researchers. But statistical significance can serve an agenda, even when effects size does not.   So the latter fall by the roadside.

    And the list could go on.  #1 is hard to ever know.  The rest I've seen during years of reading, manuscript reviewing, and presently while working on a broad review of the workforce and workplace substance use literature.  Given the topic of the review, the level of data and evidence massaging in scientific and nonscientific outlets isn't surprising because there are strong ideologies and billions of dollars in consulting and product sales at stake.  And for the nonscientists, much information comes from the internet, which can be a dangerous source of information fro the naive.

    Most research is honest and any errors are honest.  But to suggest that what is published and not published in a general area, including OB. HRM, or I/O, and which studies are funded and which are not funded is not touched by politics, vested interests, ideology, or reward systems is just not congruent with reality. I guess this is the delusion we often would like to hang on to.

    Mike Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office:    716-887-2519
    Fax:        716-887-2477
    E-mail:     frone@ria.buffalo.edu
    Internet:     http://www.ria.buffalo.edu/profiles/frone.html
    ***************************************************************

     

    >  Very funny, Michael.  But you know something?  We're not medical
    > researchers.  And we don't shield real results from "dayllight" in
    > the way that the "comedian" suggested.  And finally, we don't do the
    > same type of research.  So what was the point?  If I'm ever funded
    > by a drug company, I'll be very careful.  But what else?  I'm not
    > really getting it.  --  Gayle
    >  
    >  
    > Gayle Baugh
    > Associate Professor
    > Co-Editor, Research in Careers Series
    >   published by Information Age Publishing
    > Associate Editor, Group & Organization
    >   Management
    > Department of Management & MIS
    > University of West Florida
    > 11000 University Parkway
    > Pensacola, Florida  32514-5752
    > (850) 474-2206  (Office)
    > (850) 474-2314  (FAX)
    > gbaugh@uwf.edu



  • 10.  Comfortable in our delusions

    Posted 10-04-2011 08:16
    A colleague of mine added the following points to Mike Frone's list:

    7) Hypotheses are not clearly stated. Indeed sometimes the hypotheses are not even explicit! In many papers this makes it impossible to assess validity whatever support is provided.

    8) Scope is not stated - no indication of the researchers' views, or evidence, for limitations. Nor possible alternative explanations.

    9) Findings are not interesting. I accept that non-findings can indeed be relevant. But many studies seem designed merely to show that well-established results apply to a somewhat novel population. It seems to me that the literature search section of many papers are intended to show that the authors are well-read and that by abduction the findings are 'original', rather than any genuine attempt to design a research programme to falsify orthodoxy.

    10) No attempt to indicate whether findings are useful. Of course to be useful findings do not have to be immediately, or even directly applicable, to a business situation. But it would be nice if more papers stated explicitly how they provide a new perspective on existing knowledge, or likely fruitful areas for further research. Or how they introduce a new methodological tool e.g Granger causality. If the authors can see no use for their findings, I don't see why readers should think that they will. And of course there is justification for some purely descriptive papers. But really the lack of concern about usefulness hints at more than modesty.

    Best regards

    Giles



    ________________________________

    From: Organizational Behavior Division Listserv on behalf of Gary Robinson
    Sent: Tue 10/4/2011 10:05 AM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions



    Orson Welles is reputed to have said: There is no passion so keen as that of one person to edit another's draft". That may be the reason we are not publishing more.



    Best,



    G.



    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Guerrero, Laura
    Sent: Monday, October 03, 2011 3:56 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions



    Hi all,



    I happen to be reading this guy's book at the moment so I'll throw in what I know from the book, not from the video. This guy is a fan of evidence-based medicine, and he likes to expose the quacks. He exposes dubious claims from homeopathy, the cosmetics industry, nutritionists, etc. They use science-like language but not rigorous scientific methods.



    Our methods are by no means perfect but we aren't quacks either. If the quality control is so lax in our journals, why are we not publishing more?



    I know that not all non-findings are published but we do publish papers with unsupported Hypotheses (admittedly as long as something else is supported).



    Thanks for reading,



    Laura





    Dr. Laura Guerrero

    Assistant Professor of Management

    Marketing & Management Department

    College of Business Administration

    University of Texas at El Paso

    500 W. University Avenue

    El Paso, TX 79968-0539

    915-747-5014

    lguerrero5@utep.edu







    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Rob Briner
    Sent: Monday, October 03, 2011 1:52 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions



    Hi Mike



    A great, though depressing, list. I keep wondering why we don't view this collectively as an ethical failure and do more, also collectively to address and start to resolve such issues.



    I was interested that you'd come across these issues while doing a broad literature review. I think you get quite a different perspective about claims made both in individual papers and by communities of scholars when standing back and conducting reviews (including systematic reviews). Received wisdom often turns out to be not so wise.



    Rob



    Rob B Briner | Professor of Organizational Psychology | School of Management | University of Bath

    -------------------------------------------
    From: Organizational Behavior Division Listserv on behalf of Michael Frone[SMTP:FRONE@RIA.BUFFALO.EDU]
    Sent: Monday, October 03, 2011 7:18:41 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions
    Auto forwarded by a Rule



    Gayle,

    Well, this what I took away from it, and it was interesting because I've been having the same thoughts recently while working on a broad literature review. The main issue is of the talk was **bad science.** Bad science leads to scientific claims made without providing full or even accurate information. But bad science doesn't just mean not reporting data, it also means ignoring (hiding) data that might be in plain sight when it doesn't agree with one's agenda. It also means creatively citing or presenting data that in the end leads to an inaccurate portrayal of the evidence. Various issues can include:

    1) Author bias: Conduct a lab study and fail to support a hypothesis or find the opposite. Try again and get support for the hypothesis. What will be reported? Study 2 only? Or both study 1 and 2? Test a predictor in relation to five outcomes and one is outcome is related and the other 4 are not. What will be reported?

    2) Publication Bias: Journals only publish that which is statistically significant. It's rare to see a paper that completely fails to support some general hypothesis get published even when there might be good reason to publish it.

    3) Cherry picking citations in manuscripts or reviews. Citing a paper or two that agree with one's putative hypothesis or perspective and ignoring the preponderance of studies that fail to support it (nonsignificant findings do get published if embedded in a larger paper where some key relation may be significant and certain disciplines are more likely to publish null or contrary effects). This is especially problematic given the great reliance on convenience samples. This is especially true when vested interests are involved--i.e., there exists a product, service, or ideology to hawk.

    4) In the absence of papers to support some tenet in developing the rationale for an hypothesis, cite any paper in the area that by its title could be plausible support. I've seen my own papers cited to support a relation when my study did not assess any of the constructs, sometimes after the citing paper was published. Because I know a few literature very well, I've also found this to be true of other articles, not written by myself, cited to support a relation when they did not assess the variables in question. Accident? Carelessness? Perhaps sometimes, but not likely always. I've seen single manuscripts that had many such accidents. Moreover, none of the other reviewers picked up on these citation errors. Then the next researcher who is a bit lazy cites the study to support a relation that it cannot support and the error proliferates.

    5) Findings are cited that can't be traced to any actual study. They are cited because it supports one's perspective and they have been cited by others many times. Or present data in such a way that is very misleading. These fictitious data or distorted presentations often start in trade magazines and find their way into "scientific" papers because one can cite the trade article.

    6) Make a big deal over a statistically significant effect even though it may have absolutely no practical value. For example, one might read in a review that X has been shown to lead to impaired psychomotor performance. It is this general claims that develops a life of its own. But go back and look at the literature, and one might find how incredibly small the effects really are, and this might even have been pointed out by the primary researchers. But statistical significance can serve an agenda, even when effects size does not. So the latter fall by the roadside.

    And the list could go on. #1 is hard to ever know. The rest I've seen during years of reading, manuscript reviewing, and presently while working on a broad review of the workforce and workplace substance use literature. Given the topic of the review, the level of data and evidence massaging in scientific and nonscientific outlets isn't surprising because there are strong ideologies and billions of dollars in consulting and product sales at stake. And for the nonscientists, much information comes from the internet, which can be a dangerous source of information fro the naive.

    Most research is honest and any errors are honest. But to suggest that what is published and not published in a general area, including OB. HRM, or I/O, and which studies are funded and which are not funded is not touched by politics, vested interests, ideology, or reward systems is just not congruent with reality. I guess this is the delusion we often would like to hang on to.

    Mike Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office: 716-887-2519
    Fax: 716-887-2477
    E-mail: frone@ria.buffalo.edu
    Internet: http://www.ria.buffalo.edu/profiles/frone.html
    *************************************************************** <http://www.ria.buffalo.edu/profiles/frone.html>



    > Very funny, Michael. But you know something? We're not medical
    > researchers. And we don't shield real results from "dayllight" in
    > the way that the "comedian" suggested. And finally, we don't do the
    > same type of research. So what was the point? If I'm ever funded
    > by a drug company, I'll be very careful. But what else? I'm not
    > really getting it. -- Gayle
    >
    >
    > Gayle Baugh
    > Associate Professor
    > Co-Editor, Research in Careers Series
    > published by Information Age Publishing
    > Associate Editor, Group & Organization
    > Management
    > Department of Management & MIS
    > University of West Florida
    > 11000 University Parkway
    > Pensacola, Florida 32514-5752
    > (850) 474-2206 (Office)
    > (850) 474-2314 (FAX)
    > gbaugh@uwf.edu


  • 11.  Comfortable in our delusions

    Posted 10-04-2011 13:09

    Because Mike McDaniel baited this discussion, perhaps he'd like to comment on what he was anticipating.  Perhaps all he wanted to do was point out that publication bias exists for the nonbelievers.  If so, it is true and, broadly defined, it has many more forms than mentioned in the video.


    Gary,

    > Orson Welles is reputed to have said: There is no passion so keen as
    > that of one person to edit another's draft".  That may be the reason
    > we are not publishing more.

    Regarding the passion, agreed. But often there is good reason for the passion.  To add to my list of how information is hidden (or another feature of bias in publications), I have a penchant for comparing reported degrees of freedom to those expected by the model specification, especially for SEM.  More than a few times they didn't agree.  Often it was because the researcher misspecified the model because they had little apparent experience with the method.  So it might have been a learning experience.  However, on occasion, the discrepancy was due to allowing error terms to correlate to increase model fit. Yet, it was never mentioned in the manuscript, and I would guess it wasn't an accident.  Again, and somewhat surprisingly, other reviewers rarely if ever uncovered these problems.  Sometimes there may not be enough well-intentioned passion because it can be very time consuming.  But passion may not be enough these days.  With the move from maximum likelihood estimation to more complex forms of estimation for more complex types of models and variable types, such as numerical integration, the degrees of freedom are estimates.  There is no easy way for a reviewer to verify if what is reported is what one should expect from the model specification.  With more complex statistics and increased output of articles on any given topic, the job of evaluation gets tougher.

    Regarding other reasons we aren't publishing more, see below.

    Laura,
     
    > I happen to be reading this guy's book at the moment so I'll throw
    > in what I know from the book, not from the video.  This guy is a fan
    > of evidence-based medicine, and he likes to expose the quacks.  He
    > exposes dubious claims from homeopathy, the cosmetics industry,
    > nutritionists, etc.  They use science-like language but not rigorous
    > scientific methods.  

    To this one can add dubious claims in government reports and reports of various advocacy groups.  They often appear to cite a reputable source for some claim until one takes the time to work backward to a dead end.  So this is an import job.  Many dubious claims make it into scientific articles from nonscientific sources. If one is vested enough in an idea, the source of the evidence often doesn't matter.  And often it seems rigorous enough if one is not too critical.  Below I have a link to a document that others might find interesting for stats/methods courses.  There is, among others, an interesting chapter on how science works.  In it, the author has a section on myths and facts about science. He admits to taking a "mildly irreverent look at some formidable subjects."  Two points were:

    Myth: The institution of peer review assures that all published papers are sound and dependable.

    Fact: Peer review generally will catch something that is completely out of step with majority thinking at the time, but it is practically useless for catching outright fraud, and it is not very good at dealing with truly novel ideas. Peer review mostly assures that all papers follow the current paradigm....It certainly does not ensure that the work has been fully vetted in terms of the data analysis and the proper application of research methods.

    Myth: Real science is easily distinguished from pseudoscience.

    Fact: This is what philosophers call the problem of demarcation: One of Popper's
    principal motives in proposing his standard of falsifiability was precisely to
    provide a means of demarcation between real science and impostors. For
    example, Einstein's theory of relativity (with which Popper was deeply impressed)
    made clear predictions that could certainly be falsified if they were
    not correct. In contrast, Freud's theories of psychoanalysis (with which Popper
    was far less impressed) could never be proven wrong. Thus, to Popper,
    relativity was science but psychoanalysis was not.
    As I've already shown, real scientists don't do as Popper says they should.
    But quite aside from that, there is another problem with Popper's criterion
    (or indeed any other criterion) for demarcation: Would-be scientists read
    books too. If it becomes widely accepted (and to some extent it has) that
    falsifiable predictions are the signature of real science, then pretenders to the
    throne of science will make falsifiable predictions, too. There is no simple,
    mechanical criterion for distinguishing real science from something that is
    not real science. That certainly doesn't mean, however, that the job can't be
    done.


    > Our methods are by no means perfect but we aren't quacks either.  If
    > the quality control is so lax in our journals, why are we not
    > publishing more?

    Perhaps the more central question is why our research has little impact outside the circle of researchers?  

    But getting back to the original question, there are two answers:

    1) The quality control isn't always bad, and it catches the fair amount of the work that is poor and unpublishable.

    2) Perhaps more to the point, we may not be publishing more because, ironically, we are publishing too much.  There has been a huge proliferation of print journals in the last 15 years or so, and with open access journals catching on, the sky (or almost unlimited server space) is the limit for journal pages.  Nonetheless, there still isn't enough pages per year.  So as the number of available pages goes up and as they get filled, leave another shortfall, we need to ask does this mean we are producing more useful research or just more research?

    As an aside mentioned above, this is both related to the subject line and a bit of an extension (distraction?).

    In Daubert v. Merrell Dow Pharmaceuticals (1993), the U.S. Supreme Court made a landmark ruling that tasked federal judges with deciding whether or not expert testimony is allowable.  Or said differently, judges need to determine whether the evidence is scientifically rigorous enough to be admissible.  Evidence must be based on the scientific method, but there are exacting criteria for research to meet.  In two subsequent rulings--General Electric v. Joiner (1997) and Kumho Tire Co., LTD v. Carmichael (1999)--additional guidance was given.  It is interesting to see how the courts might view the work that we do.  Is the quality high enough to be admissible in a court case? Or said differently is it rigorous enough that anyone should care.  Attached is a link to the Reference Manual on Scientific Evidence (3rd ed., 2011) that was prepared for and is used by federal judges to help them evaluate research evidence (though they can also use outside scientific advisors).  The following chapters in the manual should be of some interest, and many tap issues relevant to management researchers.  Also other chapters may be of interest to particular individuals:

    How Science Works
    Reference Guide on Statistics (which is broader than the title might imply if not only because if discuss issues regarding reporting of all analyses and data and how it can be used to mislead)
    Reference Guide on Regression
    Reference Guide on Survey Research
    Reference Guide on Epidemiology (which is relevant to research methods in management. The section on standards for general causation and meta-analysis are generally applicable.)

    http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$file/SciMan3D01.pdf


    Although it might not seems so, the following chapter might be useful. It draws out the implications of admissibility of research to federal courts.  The examples are primarily from the pharmaceutical industry.  But consider the court to be the management of a company that needs to be convinced of the usefulness (i.e. admissibility) of some management research:

    Hollingsworth, J.G., & Lasker, E. G. (2007). Testing claims of adverse drug effects in the courtroom (Ch. 14.2; pp. 1156-1173). In S. B. Karch (Ed.), Drug abuse handbook. New York: CRC Press.

    Well this has morphed past the original email, so its time to end it.

    Mike  Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office:    716-887-2519
    Fax:        716-887-2477
    E-mail:     frone@ria.buffalo.edu
    Internet:     http://www.ria.buffalo.edu/profiles/frone.html
    ***************************************************************  



  • 12.  Comfortable in our delusions

    Posted 10-04-2011 19:28
    Michael Frone has requested that I rejoin this debate.

    I would like to offer some readings of direct relevance to publication bias in the organizational sciences but we ignore the topic so well, that there is little to offer.

    I note that the administrative leadership in both SIOP and AOM have declined to participate in a research project to track conference papers to help determine the extent of publication bias in our literatures. I guess research is sometimes administratively inconvenient.

    This is a good book:

    Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta analysis: Prevention, assessment, and adjustments. West Sussex, UK: Wiley.

    The first author, Hannah Rothstein,  management professor at Baruch, is a good source of all things related to publication bias.

    A taxonomy of causes of publication bias of relevance to the organizational sciences can be found here:

    Banks, G.C. & McDaniel, M.A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 40-44.

    I assert that meta-analyses in organizational sciences are seldom conducted consistent with the meta-analysis standards in the 2010 APA style manual. Note that publication bias is in the meta-analysis standard that falls under the category of data censoring.

    The organizational science version of a pharmaceutical company is an organization that has a monetary interest in the science concerning a product or service. Thus, companies that sell employment tests or organizational interventions would have a monetary interest in hiding data that suggests that their product is less than adequate. I am not suggesting that they do hide such data, but sometimes the results of publication bias analyses are consistent with an inference that small magnitude effects are being suppressed (e.g., hidden).  Of relevance to this topic is:

    McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927-953. doi: 10.1111/j.1744-6570.2006.00059.x

    Most all of the good research on publication bias is in the medical literature (Look in Medline/Pubmed) and in the educational psychology literature.  There are lots of publication bias findings and methods out there, just not much of it is in our literature, or read by our scholars, or taught in our doctoral programs, or known by our journal reviewers. It is a shame that the organizational sciences and their professional associations do so little to prevent publication bias in our literatures.

    Mike

    Michael A. McDaniel, Ph.D.
    Professor - Human Resources and
    Organizational Behavior
    Department of Management, Research Professor, Department of Psychology
    Virginia Commonwealth University
    301 West Main Street, , PO Box 844000
    Richmond, VA 23284-4000

    http://www.people.vcu.edu/~mamcdani/
    voice: 804.827.0209
    e-mail:     MAMcDani@vcu.edu
    skype: MichaelAMcDaniel

    		Doctoral Program in Management

    The Management Department of the VCU School of Business offers a Ph.D. in Business. Participating faculty with research interests in OB and HR include: Ron Humphrey, Sven Kepes, Michael McDaniel, In-Sue Oh, Doug Pugh, & Anson Seers.

    Students with interest in the doctoral program, should contact Anson Seers.






    From:        Michael Frone <frone@RIA.BUFFALO.EDU>
    To:        <OB@AOMLISTS.PACE.EDU>
    Date:        10/04/2011 01:12 PM
    Subject:        Re: [OB-LIST] Comfortable in our delusions
    Sent by:        Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU>





    Because Mike McDaniel baited this discussion, perhaps he'd like to comment on what he was anticipating.  Perhaps all he wanted to do was point out that publication bias exists for the nonbelievers.  If so, it is true and, broadly defined, it has many more forms than mentioned in the video.


    Gary,

    > Orson Welles is reputed to have said: There is no passion so keen as
    > that of one person to edit another's draft".  That may be the reason
    > we are not publishing more.

    Regarding the passion, agreed. But often there is good reason for the passion.  To add to my list of how information is hidden (or another feature of bias in publications), I have a penchant for comparing reported degrees of freedom to those expected by the model specification, especially for SEM.  More than a few times they didn't agree.  Often it was because the researcher misspecified the model because they had little apparent experience with the method.  So it might have been a learning experience.  However, on occasion, the discrepancy was due to allowing error terms to correlate to increase model fit. Yet, it was never mentioned in the manuscript, and I would guess it wasn't an accident.  Again, and somewhat surprisingly, other reviewers rarely if ever uncovered these problems.  Sometimes there may not be enough well-intentioned passion because it can be very time consuming.  But passion may not be enough these days.  With the move from maximum likelihood estimation to more complex forms of estimation for more complex types of models and variable types, such as numerical integration, the degrees of freedom are estimates.  There is no easy way for a reviewer to verify if what is reported is what one should expect from the model specification.  With more complex statistics and increased output of articles on any given topic, the job of evaluation gets tougher.

    Regarding other reasons we aren't publishing more, see below.

    Laura,
     
    > I happen to be reading this guy's book at the moment so I'll throw
    > in what I know from the book, not from the video.  This guy is a fan
    > of evidence-based medicine, and he likes to expose the quacks.  He
    > exposes dubious claims from homeopathy, the cosmetics industry,
    > nutritionists, etc.  They use science-like language but not rigorous
    > scientific methods.  

    To this one can add dubious claims in government reports and reports of various advocacy groups.  They often appear to cite a reputable source for some claim until one takes the time to work backward to a dead end.  So this is an import job.  Many dubious claims make it into scientific articles from nonscientific sources. If one is vested enough in an idea, the source of the evidence often doesn't matter.  And often it seems rigorous enough if one is not too critical.  Below I have a link to a document that others might find interesting for stats/methods courses.  There is, among others, an interesting chapter on how science works.  In it, the author has a section on myths and facts about science. He admits to taking a "mildly irreverent look at some formidable subjects."  Two points were:

    Myth    : The institution of peer review assures that all published papers are sound and dependable.

    Fact:     Peer review generally will catch something that is completely out of step with majority thinking at the time, but it is practically useless for catching outright fraud, and it is not very good at dealing with truly novel ideas. Peer review mostly assures that all papers follow the current paradigm....It certainly does not ensure that the work has been fully vetted in terms of the data analysis and the proper application of research methods.

    Myth:     Real science is easily distinguished from pseudoscience.

    Fact:     This is what philosophers call the problem of demarcation: One of Popper's
    principal motives in proposing his standard of falsifiability was precisely to
    provide a means of demarcation between real science and impostors. For
    example, Einstein's theory of relativity (with which Popper was deeply impressed)
    made clear predictions that could certainly be falsified if they were
    not correct. In contrast, Freud's theories of psychoanalysis (with which Popper
    was far less impressed) could never be proven wrong. Thus, to Popper,
    relativity was science but psychoanalysis was not.
    As I've already shown, real scientists don't do as Popper says they should.
    But quite aside from that, there is another problem with Popper's criterion
    (or indeed any other criterion) for demarcation: Would-be scientists read
    books too. If it becomes widely accepted (and to some extent it has) that
    falsifiable predictions are the signature of real science, then pretenders to the
    throne of science will make falsifiable predictions, too. There is no simple,
    mechanical criterion for distinguishing real science from something that is
    not real science. That certainly doesn't mean, however, that the job can't be
    done.


    > Our methods are by no means perfect but we aren't quacks either.  If
    > the quality control is so lax in our journals, why are we not
    > publishing more?

    Perhaps the more central question is why our research has little impact outside the circle of researchers?  

    But getting back to the original question, there are two answers:

    1) The quality control isn't always bad, and it catches the fair amount of the work that is poor and unpublishable.

    2) Perhaps more to the point, we may not be publishing more because, ironically, we are publishing too much.  There has been a huge proliferation of print journals in the last 15 years or so, and with open access journals catching on, the sky (or almost unlimited server space) is the limit for journal pages.  Nonetheless, there still isn't enough pages per year.  So as the number of available pages goes up and as they get filled, leave another shortfall, we need to ask does this mean we are producing more useful research or just more research?

    As an aside mentioned above, t    his is both related to the subject line and a bit of an extension (distraction?).

    In Daubert v. Merrell Dow Pharmaceuticals (1993), the U.S. Supreme Court made a landmark ruling that tasked federal judges with deciding whether or not expert testimony is allowable.  Or said differently, judges need to determine whether the evidence is scientifically rigorous enough to be admissible.  Evidence must be based on the scientific method, but there are exacting criteria for research to meet.  In two subsequent rulings--General Electric v. Joiner (1997) and Kumho Tire Co., LTD v. Carmichael (1999)--additional guidance was given.  It is interesting to see how the courts might view the work that we do.  Is the quality high enough to be admissible in a court case? Or said differently is it rigorous enough that anyone should care.  Attached is a link to the Reference Manual on Scientific Evidence (3rd ed., 2011) that was prepared for and is used by federal judges to help them evaluate research evidence (though they can also use outside scientific advisors).  The following chapters in the manual should be of some interest, and many tap issues relevant to management researchers.  Also other chapters may be of interest to particular individuals:

    How Science Works
    Reference Guide on Statistics     (which is broader than the title might imply if not only because if discuss issues regarding reporting of all analyses and data and how it can be used to mislead)
    Reference Guide on Regression
    Reference Guide on Survey Research
    Reference Guide on Epidemiology     (which is relevant to research methods in management. The section on standards for general causation and meta-analysis are generally applicable.)

    http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$file/SciMan3D01.pdf


    Although it might not seems so, the following chapter might be useful. It draws out the implications of admissibility of research to federal courts.  The examples are primarily from the pharmaceutical industry.  But consider the court to be the management of a company that needs to be convinced of the usefulness (i.e. admissibility) of some management research:

    Hollingsworth, J.G., & Lasker, E. G. (2007). Testing claims of adverse drug effects in the courtroom (Ch. 14.2; pp. 1156-1173). In S. B. Karch (Ed.), Drug abuse handbook. New York: CRC Press.

    Well this has morphed past the original email, so its time to end it.

    Mike  Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office:    716-887-2519
    Fax:        716-887-2477
    E-mail:     frone@ria.buffalo.edu
    Internet:     http://www.ria.buffalo.edu/profiles/frone.html
    ***************************************************************  



  • 13.  Comfortable in our delusions

    Posted 10-05-2011 15:37

    Colleagues,

     

    What follows is an addition to what Michael Frone said here!

     

    ". . . this was hardly the first time that I had heard, or said myself, that behavioral scientists ought to 'say something useful" to managers." He continues by explaining that "American behavioral science has been –we have been-saying the wrong thing. For decades we have tried to say the wrong thing better and better, for within our disciplinary matrix we have had trouble saying anything else. As long as we continue to say the wrong thing-no matter how 'reliably' and 'validly,' no matter how elegantly and mesmerically-it still will be the wrong thing."

     

    Peter Vaill then follows with what he called the "facts-and-methods" when he refers to the "wrong thing" American behavioral science has been saying to practitioners. "We have busily collected facts and invented methods and have then told manager-leaders that if they want to be effective they have to absorb our facts and learn our methods." He indicates that this approach ignores three things:

     

    1.       "History has seen legions of managers and leaders who were unaware of our facts and indifferent to our methods, who nonetheless have been outstanding both in getting the job done and in attending to the needs of organization members"

    2.       "The overwhelming majority of managers and leaders, even the most dutiful and dependent, find our facts-and-methods only marginally useful and not very interesting. ..."

    3.       "The best of the lot of behavioral scientists-the Maslows, Rogerses, Lewins, Mayos, Roethlisbergers, Mcgregors, Tanenbaums, Trits, and so many others-are influential and memorable with managers and leaders for who they are, the way their minds work, the way they express themselves in their protean passions that we love and remember. One does not read Peter Drucker or Warren Bennis for the facts, but rather for the song of possibility that sounds through their writings. ... (Each of us has our own list of "giants"; it need not me mine. The point is the same.)"

     

    Finally I like what Vaill says that "the best among us are living proof that we have been saying the wrong thing, for the best among us have understood in our own ways, dim and acute, florid and dry, spare and prolix, what the enterprise is really about: the enterprise is really about what it means to be in the world with responsibility."

     

    It is not possible for us to capture with our "facts-and-methods" the complexity and uniqueness of the situations with which practitioners must deal on a daily basis. It is not possible for practitioners to select only the parts of the world the theory deals with for attention.

     

    I believe that the original sin was to refer to our field of study as a science. I don't call the "giants" in my list behavioral scientists, I call them behavioral artists! 

     

    A quick anecdote: Last night I used again the video-clip that shows the SAS Corp system. In discussing with my students the way this company treats its employees, I indicated that I wasn't sure if the CEO was aware of any theory of motivation, but what he did was like flattening Maslow's five levels all into one level. Great stuff!

     

    Thanks,


    Ivan

     

    Reference: Peter B. Vaill, "Spirited Leading and Learning: Process Wisdom for a New Age." San Francisco, CA: Jossey-Bass, 1998.

     

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael Frone
    Sent: Monday, October 03, 2011 1:19 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Comfortable in our delusions

     


    Gayle,

    Well, this what I took away from it, and it was interesting because I've been having the same thoughts recently while working on a broad literature review.  The main issue is of the talk was **bad science.** Bad science leads to scientific claims made without providing full or even accurate information.  But bad science  doesn't just mean not reporting data, it also means ignoring (hiding) data that might be in plain sight when it doesn't agree with one's agenda. It also means creatively citing or presenting data that in the end leads to an inaccurate portrayal of the evidence.  Various issues can include:

    1) Author bias:  Conduct a lab study and fail to support a hypothesis or find the opposite. Try again and get support for the hypothesis.  What will be reported? Study 2 only? Or both study 1 and 2?  Test a predictor in relation to five outcomes and one is outcome is related and the other 4 are not.  What will be reported?

    2) Publication Bias: Journals only publish that which is statistically significant.  It's rare to see a paper that completely fails to support some general hypothesis get published even when there might be good reason to publish it.

    3) Cherry picking citations in manuscripts or reviews.  Citing a paper or two that agree with one's putative hypothesis or perspective and ignoring the preponderance of studies that fail to support it (nonsignificant findings do get published if embedded in a larger paper where some key relation may be  significant and certain disciplines are more likely to publish null or contrary effects).  This is especially problematic given the great reliance on convenience samples.  This is especially true when vested interests are involved--i.e., there exists a product, service, or ideology to hawk.  

    4) In the absence of papers to support some tenet in developing the rationale for an hypothesis, cite any paper in the area that by its title could be plausible support.  I've seen my own papers cited to support a relation when my study did not assess any of the constructs, sometimes after the citing paper was published.  Because I know a few literature very well, I've also found this to be true of other articles, not written by myself, cited to support a relation when they did not assess the variables in question.  Accident?  Carelessness? Perhaps sometimes, but not likely always.  I've seen single manuscripts that had many such accidents.  Moreover, none of the other reviewers picked up on these citation errors.  Then the next researcher who is a bit lazy cites the study to support a relation that it cannot support and the error proliferates.

    5) Findings are cited that can't be traced to any actual study.  They are cited because it supports one's perspective and they have been cited by others many times.  Or present data in such a way that is very misleading.  These fictitious data or distorted presentations often start in trade magazines and find their way into "scientific" papers because one can cite the trade article.

    6) Make a big deal over a statistically significant effect even though it may have absolutely no practical value.  For example, one might read in a review that X has been shown to lead to impaired psychomotor performance.  It is this general claims that develops a life of its own.  But go back and look at the literature, and one might find how incredibly small the effects really are, and this might even have been pointed out by the primary researchers. But statistical significance can serve an agenda, even when effects size does not.   So the latter fall by the roadside.

    And the list could go on.  #1 is hard to ever know.  The rest I've seen during years of reading, manuscript reviewing, and presently while working on a broad review of the workforce and workplace substance use literature.  Given the topic of the review, the level of data and evidence massaging in scientific and nonscientific outlets isn't surprising because there are strong ideologies and billions of dollars in consulting and product sales at stake.  And for the nonscientists, much information comes from the internet, which can be a dangerous source of information fro the naive.

    Most research is honest and any errors are honest.  But to suggest that what is published and not published in a general area, including OB. HRM, or I/O, and which studies are funded and which are not funded is not touched by politics, vested interests, ideology, or reward systems is just not congruent with reality. I guess this is the delusion we often would like to hang on to.

    Mike Frone

    ****************************************************************
    Michael R. Frone, Ph.D.
    Senior Research Scientist
    Research Institute on Addictions
    State University of New York at Buffalo
    1021 Main Street
    Buffalo, New York 14203

    Office:    716-887-2519
    Fax:        716-887-2477
    E-mail:     frone@ria.buffalo.edu
    Internet:     http://www.ria.buffalo.edu/profiles/frone.html
    ***************************************************************

     

    >  Very funny, Michael.  But you know something?  We're not medical
    > researchers.  And we don't shield real results from "dayllight" in
    > the way that the "comedian" suggested.  And finally, we don't do the
    > same type of research.  So what was the point?  If I'm ever funded
    > by a drug company, I'll be very careful.  But what else?  I'm not
    > really getting it.  --  Gayle
    >  
    >  
    > Gayle Baugh
    > Associate Professor
    > Co-Editor, Research in Careers Series
    >   published by Information Age Publishing
    > Associate Editor, Group & Organization
    >   Management
    > Department of Management & MIS
    > University of West Florida
    > 11000 University Parkway
    > Pensacola, Florida  32514-5752
    > (850) 474-2206  (Office)
    > (850) 474-2314  (FAX)
    > gbaugh@uwf.edu