Discussion: View Thread

Dear Prof. Schneider,

I think it would valuable if we would differentiate between predictive power/utility (which as nothing to do with validity) and validity (i.e., do we really measure the entity of interest).

See:

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071.

In this paper, Borsboom et al. argue that a measure is valid when it measures the latent variable (i.e., is caused by the variable) that

it intends to measure.  In this regard, a measure can be very unreliable but nonetheless valid (because it reflects the latent).

Summing single preditive measures together is a constructivist approach  - thus, there is no entity which is measured and, hence, no validity (pp. 1069).

With best regards,

Holger

Hi,

You are of course correct if one conceptualizes validity as a unitary idea, including content, construct, and predictive varieties under one umbrella. But the sense in which the phrase "validity cannot exceed the square root of reliability" is used does not have reliability and validity under one umbrella. Reliability in the phrase is used very loosely so people assume it refers to internal consistency which of course has to do with the content and the construct but not the predictive power.

Ben

Dear Prof. Schneider,

I think it would valuable if we would differentiate between predictive power/utility (which as nothing to do with validity) and validity (i.e., do we really measure the entity of interest).

See:

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071.

In this paper, Borsboom et al. argue that a measure is valid when it measures the latent variable (i.e., is caused by the variable) that

it intends to measure.  In this regard, a measure can be very unreliable but nonetheless valid (because it reflects the latent).

Summing single preditive measures together is a constructivist approach  - thus, there is no entity which is measured and, hence, no validity (pp. 1069).

With best regards,

Holger

Ben, to expand on your comment, we need to be careful to understand both validity and reliability as multi-dimensional constructs.  Too often we think of the different types of reliability and validity as different "measures" of reliability and validity rather than distinct types.  One could have a measure that had very high alpha scores and yet upon re-test yielded dramatically different results.  How "reliable" is that measure?  Too many of us have simply decided that since alpha is easy to compute we should just use it as the single representation of reliability.  It is much less common these days to see test/re-test reported, and I think that is partly a reflection of the time and effort such validation takes.  As was mentioned earlier, simple constructs (which IMHO are increasingly boring) can be expected to have high alphas, but complex phenomena might have low alphas and in fact, alpha may be relatively meaningless for certain complex phenomena.

As a side note, it strikes me a somewhat ironic that we might be able to look at the multiple types of reliability for a measure and compute the "meta-reliability", the consistency between types of reliability ;-)


Jeff
Utah Valley University

On 3/29/10 11:28 AM, "Ben Schneider" <BSchneider@VALTERA.COM" target="_blank">BSchneider@VALTERA.COM> wrote:

Hi,
 
You are of course correct if one conceptualizes validity as a unitary idea, including content, construct, and predictive varieties under one umbrella. But the sense in which the phrase "validity cannot exceed the square root of reliability" is used does not have reliability and validity under one umbrella. Reliability in the phrase is used very loosely so people assume it refers to internal consistency which of course has to do with the content and the construct but not the predictive power.
 
Ben
 

Benjamin Schneider, Ph.D.
Senior Research Fellow, VALTERA
Professor Emeritus, University of Maryland
1363 Caminito Floreo, Suite G
La Jolla, CA 92037
tel/fx: 858-488-7594
bschneider@valtera.com" target="_blank">bschneider@valtera.com
 
VALTERA ®
Better Organizations Through Better Science ®
www.valtera.com
 
Chicago Corporate Office:
Valtera Corporation
1701 Golf Rd., 2-1100
Rolling Meadows, IL 60008-4257
www.valtera.com
 
This email and attachments, if included, may contain material that is
confidential. This material is intended for the sole use of the individual
or entity to whom it is addressed. If you received this message in error,
please contact the sender and delete all copies.

---

From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Holger Steinmetz
Sent: Monday, March 29, 2010 8:52 AM
To: OB@AOMLISTS.PACE.EDU" target="_blank">OB@AOMLISTS.PACE.EDU
Subject: [OB-LIST] AW: [OB-LIST] Responses to request for Low alpha scores of Big 5 TIPI scale----note on reliability and validity

Dear Prof. Schneider,
 
I think it would valuable if we would differentiate between predictive power/utility (which as nothing to do with validity) and validity (i.e., do we really measure the entity of interest).
 
See:
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071.
 
In this paper, Borsboom et al. argue that a measure is valid when it measures the latent variable (i.e., is caused by the variable) that
it intends to measure.  In this regard, a measure can be very unreliable but nonetheless valid (because it reflects the latent).
 
Summing single preditive measures together is a constructivist approach  - thus, there is no entity which is measured and, hence, no validity (pp. 1069).
 
With best regards,
Holger
 

___________________________________________________
Dr. Holger Steinmetz
University of Giessen
Faculty of Economics and Business Administration
Department of Human Resource Management, Small Business Enterprises, and Entrepreneurship
Licher Str. 66
35394 Giessen
Phone: +49 641 99 22103
Fax: +49 641 99 22109
 
Holger.Steinmetz@psychol.uni-giessen.de" target="_blank">Holger.Steinmetz@psychol.uni-giessen.de
http://wiwi.uni-giessen.de/home/personal
 
And:
Department of Work and Organizational Psychology
Otto-Behaghel-Strasse 10F
35394 Giessen/Germany
www.uni-giessen.biz
___________________________________________________
 


Von: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] Im Auftrag von Ben Schneider
Gesendet: Montag, 29. März 2010 16:50
An: OB@AOMLISTS.PACE.EDU" target="_blank">OB@AOMLISTS.PACE.EDU
Betreff: Re: [OB-LIST] Responses to request for Low alpha scores of Big 5 TIPI scale----note on reliability and validity

Hi All,
Several comments in this exchange inlcuded the idea that relaibility puts a limit on valididty; this is not true for internal relaiaiblity, the topic of the e-mails.
Think about it this way: Multiple regression takes uncorrelated variables to establish the prediction equation making the prediction equation unrelaible from an intenal consistency standpoint. Or, consider the use of criterion-keyed measures (like bio-data) where the predcitive power comes from combining items that are not internally comsistent.
I have seen people say this for my entire career and what is true is that re-test relaibility puts a limit on prediction since a variable can't predict another variable better than the square root of itself.
Ben
Benjamin Schneider, Ph.D.
Senior Research Fellow, VALTERA
Professor Emeritus, University of Maryland
1363 Caminito Floreo, Suite G
La Jolla, CA 92037
tel/fx: 858-488-7594
bschneider@valtera.com" target="_blank">bschneider@valtera.com
 
VALTERA ®
Better Organizations Through Better Science ®
www.valtera.com
 
Chicago Corporate Office:
Valtera Corporation
1701 Golf Rd., 2-1100
Rolling Meadows, IL 60008-4257
www.valtera.com
 
This email and attachments, if included, may contain material that is
confidential. This material is intended for the sole use of the individual
or entity to whom it is addressed. If you received this message in error,
please contact the sender and delete all copies.

-----Original Message-----
From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Stefan Volk
Sent: Monday, March 29, 2010 3:55 AM
To: OB@AOMLISTS.PACE.EDU" target="_blank">OB@AOMLISTS.PACE.EDU
Subject: [OB-LIST] Responses to request for Low alpha scores of Big 5 TIPI scale
 
Thanks to everyone who responded to my request concerning low alpha
scores of scales with small numbers of items. I've attached a Word
document that has the summary.
 
 
Thanks again!
 
 
 
 
>
>
>    On Tue, Mar 23, 2010 at 5:33 AM, Stefan Volk
>    <stefan.volk@uni-tuebingen.de" target="_blank">stefan.volk@uni-tuebingen.de
>    <mailto:stefan.volk@uni-tuebingen.de>> wrote:
>
>        Dear all,
>
>        we used the Gosling et al. (2003) 10-item personality
>        inventory (TIPI) and obtained low Cronbach's alpha scores. Sam
>        Gosling provides an explanation on his website indicating that
>        alphas are misleading when calculated on scales with small
>        numbers of items. I was wondering if someone could provide me
>        with or point me to some more arguments for reviewers apart
>        from the explanation given by Sam, in the ideal case something
>        that has been published. I see once in a while that authors do
>        not report alphas if they use two-item scales. What is the
>        theoretical argument of not reporting alphas, if scales
>        consist of only two items?
>
>        Many thanks in advance,
>        Stefan
>