Discussion: View Thread

  • 1.  individual-level studies should address the assumption of independence of observations

    Posted 04-28-2012 22:58
    Dear colleagues,

    I would like to start a discussion on the following issue:

    As a team researcher, I always have to deal with the issue of non-independence of observations, either by using multi-level analysis or by aggregating the data to the team level (after calculating aggregation stats, like rWGs and ICCs). Now, I have been struck by the fact that while the demand of dealing with non-independence of observation is being placed on team and team-level research (and justly so!), it is often not being placed on studies that examine variables exclusively at the individual level. That is, while team-level studies (have to) provide evidence that data ARE NOT independent (for instance, by calculating ICCs), individual-level studies often do not (have to) provide evidence that their data ARE independent. 

    The reason I raise this issue is because, I suspect that in many individual-level studies case are in fact non-independent. Take for instance leadership research. Due to the inherent nature of how most organizations are organized, any given leader will typically supervise multiple employees (i.e. the employees are embedded within the leader). This means that employees are exposed to the actions of this same leader and/or that they will share their individual experiences of their interaction with their leader will theri fellow employees. In other words, it's quite likely that employees embedded within the same leader will be more develop more homogeneous perceptions of that leader than employees that serve under different leaders. While this obviously implies non-independence of observations, many studies examining employees' perceptions of their leader often proceed as if these data are independent, without providing evidence of this.

    My point is not to criticize or get into the details of leadership research per se, it's just an illustration of an area of research where I think data are most likely to be non-independent. And, of course, I realize that there are individual-level studies out there that do take non-independence of observations into account. Rather, my point is that individual-level studies don't do this as often as they probably should, particularly in areas where there are clear indications (for instance, by their very topic, like leadership) that observations are non-independent.

    Therefore, my 2 statements for the discussion are:

    1) Future individual-level studies should explicitly address and justify their assumption of independent observations

    2) The demand on individual-level studies to explicitly address this issue should be formally institutionalized (for instance, through journal policies and reviewer comments on individual-level papers).          
     
    I look forward to your reactions!

    Best,
    Bart de Jong
    VU University Amsterdam


    Ps. for a reading on the consequences of treating non-independent observations as if they were independent, see:

    Bliese, P. D., & Hanges, P. J. (2004). Being Both Too Liberal and Too Conservative: The Perils of Treating Grouped Data as though They Were Independent. Organizational Research Methods, 7(4), 400-417.


  • 2.  individual-level studies should address the assumption of independence of observations

    Posted 04-30-2012 22:47
    Hi Bart and the OBlist,

       This issue is an interesting one - and one that should probably be addressed in our science that hopes to move more toward a meso-level paradigm/approach.  I have a couple of thoughts on this issue, that could further the discussion.

       I think a real issue here combines Bart's points 1 and 2:  To begin, one thing that makes team research a more obvious target for accounting for nesting/clustering is that there is a clear cluster in at which variance can be examined, the team.  By comparison, its not clear how to cluster observations other individual-oriented research.  Many "clusterings" are clearly possible (people in close vicinity at work, organization members, those in the geographic area, etc...), some of which more reasonable than others.  Without some established guidelines, who's to stop a researcher from examining a trivial cluster, determining independence at this trivial cluster (when in fact, a different, more meaningful cluster may produce clustering) and moving on?  Additionally, it may not be possible, in some cases, to uniquely identify variance attributable to the cluster vs. the individual response (i.e., when only 1 or 2 persons are present per cluster; though there are solutions to this issue out there: see http://smr.sagepub.com/content/35/3/311.abstract).  

      More fundamentally, I think the issue here is not so much clustering, as it is an issue that John Antonakis brought up this winter through his webcast: endogeneity or, if you'd prefer, omitted variables/effects like feedback loops.   As Bart mentions, clustering occurs for a reason.  Leaders affect follower perceptions, people in organizations develop similar beliefs owing to culture/climate, individuals in different metropolitan areas have specific local and municipal governments that provide specific services, attract different businesses and consequently have different labor markets, etc...  

       Clustering or nesting (i.e., a non-0 intraclass correlation) then results from cluster or group-level effects, omitted or otherwise - indeed, in multilevel modeling the purpose is to explain such cluster-level variance (incidentally, I have a presentation about a similar topic this year at AOM - yes, this is a shameless plug).  Thus, the problem seems to be one of not accounting for cluster-level effects leading to correlated error (i.e., cluster-level endogeneity), as opposed to independent observations, per se, which is just a symptom of cluster-level effects "waiting" to be explained.  Bliese and Hanges (2004) address significance testing in their simulation, that is, that showing that standard errors differ but point estimates do not differ between standard, fixed effects and mixed effects regression when no misspesification occurs and all Level I and Level I unexplained variance is random (or at least I presume that's the model underlying the simulation, the authors or others can correct me if I'm wrong on that).  However, in the end, I agree with John A. that evaluating the hypothesis (through a Hausman test, for instance) that our point estimates are, in fact, consistent (http://en.wikipedia.org/wiki/Consistent_estimator) is probably a more pressing concern especially given that clustering can happen in so many ways that are likely to produce non-completely-random variance and will affect more than the standard error.  

      A bunch of inconsistent point estimates can't be "fixed" with a meta-analysis (which is, itself, based on the consistency principle) - but, a bunch of non-significant effects, that are consistently estimated, can be meta-analyzed to find the population parameter of interest.

       - joe 

     Joseph Nicholas Luchman, M.A.
    ----
    Senior Research Associate | Fors Marsh Group
    Desk: 571 858 3770
    Email: jluchman@forsmarshgroup.com
    forsmarshgroup.com
    ----
    Doctoral Candidate
    Industrial Organizational Psychology
    George Mason University
    http://sites.google.com/site/jluchman/

    "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."
    - Donald T. Campbell



    2012/4/28 Jong, B.A. de <bart.de.jong@vu.nl>
    Dear colleagues,

    I would like to start a discussion on the following issue:

    As a team researcher, I always have to deal with the issue of non-independence of observations, either by using multi-level analysis or by aggregating the data to the team level (after calculating aggregation stats, like rWGs and ICCs). Now, I have been struck by the fact that while the demand of dealing with non-independence of observation is being placed on team and team-level research (and justly so!), it is often not being placed on studies that examine variables exclusively at the individual level. That is, while team-level studies (have to) provide evidence that data ARE NOT independent (for instance, by calculating ICCs), individual-level studies often do not (have to) provide evidence that their data ARE independent. 

    The reason I raise this issue is because, I suspect that in many individual-level studies case are in fact non-independent. Take for instance leadership research. Due to the inherent nature of how most organizations are organized, any given leader will typically supervise multiple employees (i.e. the employees are embedded within the leader). This means that employees are exposed to the actions of this same leader and/or that they will share their individual experiences of their interaction with their leader will theri fellow employees. In other words, it's quite likely that employees embedded within the same leader will be more develop more homogeneous perceptions of that leader than employees that serve under different leaders. While this obviously implies non-independence of observations, many studies examining employees' perceptions of their leader often proceed as if these data are independent, without providing evidence of this.

    My point is not to criticize or get into the details of leadership research per se, it's just an illustration of an area of research where I think data are most likely to be non-independent. And, of course, I realize that there are individual-level studies out there that do take non-independence of observations into account. Rather, my point is that individual-level studies don't do this as often as they probably should, particularly in areas where there are clear indications (for instance, by their very topic, like leadership) that observations are non-independent.

    Therefore, my 2 statements for the discussion are:

    1) Future individual-level studies should explicitly address and justify their assumption of independent observations

    2) The demand on individual-level studies to explicitly address this issue should be formally institutionalized (for instance, through journal policies and reviewer comments on individual-level papers).          
     
    I look forward to your reactions!

    Best,
    Bart de Jong
    VU University Amsterdam


    Ps. for a reading on the consequences of treating non-independent observations as if they were independent, see:

    Bliese, P. D., & Hanges, P. J. (2004). Being Both Too Liberal and Too Conservative: The Perils of Treating Grouped Data as though They Were Independent. Organizational Research Methods, 7(4), 400-417.