Discussion: View Thread

  • 1.  Interrater Agreement and Scale Internal Consistency for Multilevel Analysis

    Posted 11-12-2015 12:28
    Dear OB colleagues, 

    I am working on a multilevel dataset, which contains peer ratings of each other in each team using multi-item scales, as well as group member ratings on team characteristics using multi-item scales. I have several research questions regarding relationships among individual level variables (e.g., personality, performance rated by others). I also have several research questions involving team-level moderators (e.g., team-level task characteristics). 

    For data aggregation, I will need evidence for peer agreement on each team member's behaviors, team member agreement on team characteristics, and good internal consistency for each multi-item scale. Currently, I run into a question related to this procedure: What is the correct order of computing Cronbach's alpha, computing rwg/ICC, and aggregating data? I have come up with two plausible options, but I am not sure about the appropriateness of each:

    1) Using the peer rating data, compute Cronbach's alpha for each scale before computing agreement indices. Is dependency issue a concern since each rater provided multiple sets of ratings for other team members? 

    2) Compute agreement indices first. After aggregating peer ratings for each individual (given good agreement), compute Cronbach's alpha for each scale using individual level data. Is good internal consistency a requirement for testing rwgj? If I am computing ICCs using scale scores, I will need to demonstrate appropriateness of computing scale scores first. 

    Each one seems to have its pros and cons. Any advice would be highly appreciated.

    Best Regards,
    Shan 

    --
    Shan Ran, M.A.
    Doctoral Candidate of Industrial and Organizational Psychology
    Graduate Consultant at the Research Design and Analysis (RDA) Unit
    Wayne State University


  • 2.  Interrater Agreement and Scale Internal Consistency for Multilevel Analysis

    Posted 11-12-2015 16:25
    Hi Shan,

    You may find this article to be helpful:

    LeBreton, J. M., & Senter, J. L. (2007). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods.

    Best,
    Victoria


    On Thu, Nov 12, 2015 at 10:28 AM, Shan Ran <fa3074@wayne.edu> wrote:
    Dear OB colleagues, 

    I am working on a multilevel dataset, which contains peer ratings of each other in each team using multi-item scales, as well as group member ratings on team characteristics using multi-item scales. I have several research questions regarding relationships among individual level variables (e.g., personality, performance rated by others). I also have several research questions involving team-level moderators (e.g., team-level task characteristics). 

    For data aggregation, I will need evidence for peer agreement on each team member's behaviors, team member agreement on team characteristics, and good internal consistency for each multi-item scale. Currently, I run into a question related to this procedure: What is the correct order of computing Cronbach's alpha, computing rwg/ICC, and aggregating data? I have come up with two plausible options, but I am not sure about the appropriateness of each:

    1) Using the peer rating data, compute Cronbach's alpha for each scale before computing agreement indices. Is dependency issue a concern since each rater provided multiple sets of ratings for other team members? 

    2) Compute agreement indices first. After aggregating peer ratings for each individual (given good agreement), compute Cronbach's alpha for each scale using individual level data. Is good internal consistency a requirement for testing rwgj? If I am computing ICCs using scale scores, I will need to demonstrate appropriateness of computing scale scores first. 

    Each one seems to have its pros and cons. Any advice would be highly appreciated.

    Best Regards,
    Shan 

    --
    Shan Ran, M.A.
    Doctoral Candidate of Industrial and Organizational Psychology
    Graduate Consultant at the Research Design and Analysis (RDA) Unit
    Wayne State University



    --
    Victoria Prescott Mattingly, MS
    Industrial/Organizational Psychology Graduate Student
    Department of Psychology
    Colorado State University
    Fort Collins, CO
    Office: 327 Behavioral Sciences Building


  • 3.  Interrater Agreement and Scale Internal Consistency for Multilevel Analysis

    Posted 11-13-2015 08:50
    Hi Shan,

    It looks like you are interested in looking at the reliability of your scale measures, the reliability of the means of aggregate measures, and the amount of agreement in group measures. The need to demonstrate evidence of agreement in group measures is first determined by your theory; some group measures may not be dependent on agreement. For example, a group measure of climate strength could be the amount of variability around the group mean. Here is a great overview of aggregation issues:

    Chan, D. (1998).  Functional levels among constructs in the same content domain at different levels of analysis: A typology of composition models.  Journal of Applied Psychology, 83, 234-246.

    For your group measures that are dependent on agreement, and for the distinction between agreement and reliability, here are a few articles to supplement Victoria's  suggestion:

    Bliese, P.D. (2000).  Within-group agreement, non-indepedence, and reliability: Implications for data aggregation and analysis. In K.J. Klein & S.W.J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations.  San Francisco:  Jossey Bass. 

    Bliese, P. (2013). Multilevel Functions: R Package 'multilevel', Version 2.5.

    Biemann, T., Cole, M.S., & Voelpel, S. (2012). Within-Group Agreement: On the Use (and Misuse) of rWG and rWG(J) in Leadership Research and Some Best Practice Guidelines. Leadership Quarterly, 23, 66-80.

    Liao, S.C., Hunt, E.A., & Chen, W. (2010). Comparison between inter-rater reliability and inter-rater agreement in performance assessment. ANNALS Academy of Medicine Singapore, 39(8), 613-618.

    Wagner, S.M., Rau, C., & Lindemann, E. (2010). Multiple informant methodology: a critical review and recommendations. Sociological Methods & Research, 38(4), 582-618. 
    Cohen, A., Doveh, E., & Nahum-Shani, I. (2007). Testing agreement for multi-item scales with the indices rWG(J) and AD M(J). Organizational Research Methods, 12(1), 148-164.

    Good luck with your research!

    Mark 


    On Nov 12, 2015, at 4:24 PM, Victoria Mattingly <Victoria.Mattingly@COLOSTATE.EDU> wrote:

    Hi Shan,

    You may find this article to be helpful:

    LeBreton, J. M., & Senter, J. L. (2007). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods.

    Best,
    Victoria


    On Thu, Nov 12, 2015 at 10:28 AM, Shan Ran <fa3074@wayne.edu> wrote:
    Dear OB colleagues, 

    I am working on a multilevel dataset, which contains peer ratings of each other in each team using multi-item scales, as well as group member ratings on team characteristics using multi-item scales. I have several research questions regarding relationships among individual level variables (e.g., personality, performance rated by others). I also have several research questions involving team-level moderators (e.g., team-level task characteristics). 

    For data aggregation, I will need evidence for peer agreement on each team member's behaviors, team member agreement on team characteristics, and good internal consistency for each multi-item scale. Currently, I run into a question related to this procedure: What is the correct order of computing Cronbach's alpha, computing rwg/ICC, and aggregating data? I have come up with two plausible options, but I am not sure about the appropriateness of each:

    1) Using the peer rating data, compute Cronbach's alpha for each scale before computing agreement indices. Is dependency issue a concern since each rater provided multiple sets of ratings for other team members? 

    2) Compute agreement indices first. After aggregating peer ratings for each individual (given good agreement), compute Cronbach's alpha for each scale using individual level data. Is good internal consistency a requirement for testing rwgj? If I am computing ICCs using scale scores, I will need to demonstrate appropriateness of computing scale scores first. 

    Each one seems to have its pros and cons. Any advice would be highly appreciated.

    Best Regards,
    Shan 

    --
    Shan Ran, M.A.
    Doctoral Candidate of Industrial and Organizational Psychology
    Graduate Consultant at the Research Design and Analysis (RDA) Unit
    Wayne State University



    --
    Victoria Prescott Mattingly, MS
    Industrial/Organizational Psychology Graduate Student
    Department of Psychology
    Colorado State University
    Fort Collins, CO
    Office: 327 Behavioral Sciences Building



  • 4.  Interrater Agreement and Scale Internal Consistency for Multilevel Analysis

    Posted 11-16-2015 10:57
    Dear Shan,
    the problem you described reminds me of the issue how to calculate reliabilities in experience sampling studies. You might find the following reference helpful:
    Nezlek, J. B. (2007). Multilevel modeling in research on personality. Handbook of research methods in personality psychology, 502-523.
    Best
    Sandra


    Am 12.11.2015 um 18:28 schrieb Shan Ran:
    BfEU1cgrU7Edw@mail.gmail.com" type="cite">
    Dear OB colleagues, 

    I am working on a multilevel dataset, which contains peer ratings of each other in each team using multi-item scales, as well as group member ratings on team characteristics using multi-item scales. I have several research questions regarding relationships among individual level variables (e.g., personality, performance rated by others). I also have several research questions involving team-level moderators (e.g., team-level task characteristics). 

    For data aggregation, I will need evidence for peer agreement on each team member's behaviors, team member agreement on team characteristics, and good internal consistency for each multi-item scale. Currently, I run into a question related to this procedure: What is the correct order of computing Cronbach's alpha, computing rwg/ICC, and aggregating data? I have come up with two plausible options, but I am not sure about the appropriateness of each:

    1) Using the peer rating data, compute Cronbach's alpha for each scale before computing agreement indices. Is dependency issue a concern since each rater provided multiple sets of ratings for other team members? 

    2) Compute agreement indices first. After aggregating peer ratings for each individual (given good agreement), compute Cronbach's alpha for each scale using individual level data. Is good internal consistency a requirement for testing rwgj? If I am computing ICCs using scale scores, I will need to demonstrate appropriateness of computing scale scores first. 

    Each one seems to have its pros and cons. Any advice would be highly appreciated.

    Best Regards,
    Shan 

    --
    Shan Ran, M.A.
    Doctoral Candidate of Industrial and Organizational Psychology
    Graduate Consultant at the Research Design and Analysis (RDA) Unit
    Wayne State University

    --  ****************************************** Prof. Dr. Sandra Ohly Fachgebiet Wirtschaftspsychologie Universität Kassel  Business Psychology Group University of Kassel ******************************************


  • 5.  Interrater Agreement and Scale Internal Consistency for Multilevel Analysis

    Posted 11-16-2015 13:30

    Dear Shan,

     

    in the context of experience sampling studies, I also found the following article very helpful for estimating reliabilities:

    Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72-91.

     

    Best regards,

    Roman

    ___________________

     

    Mag. Roman Prem

     

    University of Vienna

    Department of Applied Psychology:

    Work, Education, Economy

    Universitätsstraße 7 / Room D0604

    A-1010 Vienna, Austria

     

    T +43-1-4277-473 43

    F +43-1-4277-8 473 43

    roman.prem@univie.ac.at

    http://homepage.univie.ac.at/roman.prem/

    http://ao-psy.univie.ac.at/

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Sandra Ohly
    Sent: Monday, November 16, 2015 4:57 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] Interrater Agreement and Scale Internal Consistency for Multilevel Analysis

     

    Dear Shan,
    the problem you described reminds me of the issue how to calculate reliabilities in experience sampling studies. You might find the following reference helpful:

    Nezlek, J. B. (2007). Multilevel modeling in research on personality. Handbook of research methods in personality psychology, 502-523.
    Best
    Sandra

     

    Am 12.11.2015 um 18:28 schrieb Shan Ran:

    Dear OB colleagues, 

     

    I am working on a multilevel dataset, which contains peer ratings of each other in each team using multi-item scales, as well as group member ratings on team characteristics using multi-item scales. I have several research questions regarding relationships among individual level variables (e.g., personality, performance rated by others). I also have several research questions involving team-level moderators (e.g., team-level task characteristics). 

     

    For data aggregation, I will need evidence for peer agreement on each team member's behaviors, team member agreement on team characteristics, and good internal consistency for each multi-item scale. Currently, I run into a question related to this procedure: What is the correct order of computing Cronbach's alpha, computing rwg/ICC, and aggregating data? I have come up with two plausible options, but I am not sure about the appropriateness of each:

     

    1) Using the peer rating data, compute Cronbach's alpha for each scale before computing agreement indices. Is dependency issue a concern since each rater provided multiple sets of ratings for other team members? 

     

    2) Compute agreement indices first. After aggregating peer ratings for each individual (given good agreement), compute Cronbach's alpha for each scale using individual level data. Is good internal consistency a requirement for testing rwgj? If I am computing ICCs using scale scores, I will need to demonstrate appropriateness of computing scale scores first. 

     

    Each one seems to have its pros and cons. Any advice would be highly appreciated.

     

    Best Regards,

    Shan 

     

    --

    Shan Ran, M.A.
    Doctoral Candidate of Industrial and Organizational Psychology
    Graduate Consultant at the Research Design and Analysis (RDA) Unit
    Wayne State University



    -- 
    ******************************************
    Prof. Dr. Sandra Ohly
    Fachgebiet Wirtschaftspsychologie
    Universität Kassel
     
    Business Psychology Group
    University of Kassel
    ******************************************