Discussion: View Thread

  • 1.  How much variability in college academic achievement is due to teachers?

    Posted 12-09-2015 15:40
    I recognize that my question would be best addressed to an educational research listserv but I don't belong to any of them. Also, most all members of this listserv are college faculty, so maybe you know.

    The question:  With reference to the academic achievement of college students, what percent of the variability of academic achievement can be attributed to teachers?

    The American Statistical Association issued a policy statement in 2014 that concluded that "teachers account for about 1% to 14% of the variability in test scores" (available at http://www.amstat.org/policy/pdfs/asa_vam_statement.pdf). But, as far as I can tell, the data reviewed were not from college students.

    Of relevance, there is an interesting book "G is for Genes: The Impact of Genetics on Education and Achievement" by K. Asbury and Robert Plomin (2014). Like intelligence ("g", not the make-believe intelligences like emotional, practical, football, etc.), educational achievement is substantially heritable and all of the environmental sources of educational achievement  are necessarily malleable. 

    In sum, can anyone refer me to a literature that directly addresses what percent of the variability of college academic achievement can be attributed to teachers?

    Thanks,

    Mike


    Professor, Department of Management, 
    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 
    Richmond, VA 23284-4000
    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu
    Research Professor, VCU Department of Psychology



  • 2.  How much variability in college academic achievement is due to teachers?

    Posted 12-10-2015 18:18

    Mike,

     

    I am not aware of studies examining the degree of variability in achievement attributed to teachers. But, there is evidence that variability in the relation between test scores (e.g., SAT) and achievement (e.g., GPA) is due to colleges. Specifically, the following article describes the presence of variability in test bias (i.e., differential prediction) across institutions and, overall, calls raises questions about the fairness of using the SAT as a predictor of GPA:

     

    ·       Aguinis, H., Culpepper, S.A., & Pierce, C.A. (in press). Differential prediction generalization in college admissions testing. Journal of Educational Psychology. [available at http://mypage.iu.edu/~haguinis/pubs.html]

     

    Abstract

    We introduce the concept of differential prediction generalization in the context of college admissions testing. Specifically, we assess the extent to which predicted first-year college grade-point average (GPA) based on high-school grade point average (HSGPA) and SAT scores depends on a student's ethnicity and gender and whether this difference varies across samples. We compared 257,336 female and 220,433 male students across 339 samples, 29,734 Black and 304,372 White students across 247 samples, and 35,681 Hispanic and 308,818 White students across 264 samples collected from 176 colleges and universities between the years 2006 and 2008. Overall, results show a lack of differential prediction generalization because variability remains after accounting for methodological and statistical artifacts including sample size, range restriction, proportion of students across ethnicity- and gender-based subgroups, subgroup mean differences on the predictors (i.e., HSGPA, SAT-Critical Reading, SAT-Math, and SAT-Writing), and standard deviations for the predictors. We offer an agenda for future research aimed at understanding several contextual reasons for a lack of differential prediction generalization based on ethnicity and gender. Results from such research will likely lead to a better understanding of the reasons for differential prediction and interventions aimed at reducing or eliminating it when it exists.

     

    I hope this helps!

     

    All the best,

     

    --Herman.

     

    Herman Aguinis

    John F. Mee Chair of Management

    Professor of Organizational Behavior and Human Resources

    Founding and Managing Director, Institute for Global Organizational Effectiveness

     

    Indiana University

    Kelley School of Business

    http://mypage.iu.edu/~haguinis/

     

    GO FROM MOMENT TO MOMENTUM

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: Wednesday, December 09, 2015 3:40 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    I recognize that my question would be best addressed to an educational research listserv but I don't belong to any of them. Also, most all members of this listserv are college faculty, so maybe you know.

     

    The question:  With reference to the academic achievement of college students, what percent of the variability of academic achievement can be attributed to teachers?

     

    The American Statistical Association issued a policy statement in 2014 that concluded that "teachers account for about 1% to 14% of the variability in test scores" (available at http://www.amstat.org/policy/pdfs/asa_vam_statement.pdf). But, as far as I can tell, the data reviewed were not from college students.

     

    Of relevance, there is an interesting book "G is for Genes: The Impact of Genetics on Education and Achievement" by K. Asbury and Robert Plomin (2014). Like intelligence ("g", not the make-believe intelligences like emotional, practical, football, etc.), educational achievement is substantially heritable and all of the environmental sources of educational achievement  are necessarily malleable. 

     

    In sum, can anyone refer me to a literature that directly addresses what percent of the variability of college academic achievement can be attributed to teachers?

     

    Thanks,

     

    Mike

     

     

    Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 

    Richmond, VA 23284-4000

    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu

    Research Professor, VCU Department of Psychology

     



  • 3.  How much variability in college academic achievement is due to teachers?

    Posted 12-10-2015 19:51

    Mike,

    Although this is not college academic achievement, we have done substantial work since 2006 investigating foreign language learning in the military. In many cases, the Government assessments and courses use a similar proficiency framework to those used in universities and K-12 (e.g., the American Council on the Teaching of Foreign Languages [ACTFL] scale came out of the Government's ILR scale). Our work includes investigating the impact of instructor differences on various learning outcomes. Kemp Ellington and I presented some of this work at SIOP:

     

    Surface, E. A., & Ellington, J. K. (2008, April). Rethinking context in training effectiveness research: Instructor as learning context . A paper presented in E. C. Dierdorff's symposium [Illuminating the 'Murky Ground': Linking Context Theory to Empirical Research] at the 23rd annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.

     

    Ellington, J. K., & Surface, E. A. (2009, April). The trainer's impact in the learning environment: A multilevel examination. In A. Towler and S. Bell (Chairs), The people factor: Considering trainee and trainer effects on learning. Symposium presented at the 24th annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.

     

    In our initial study of archival data, we investigated instructor influence using a 3-level model in HLM (students within classes within instructors-instructors taught multiple classes/sections over time but same course and learn population) and standardized proficiency assessment scores in reading, listening and speaking. We found that instructors accounted for up to 41% of the variability in the learner scores. In subsequence studies, we found significant variability in reaction measures, affective outcomes (e.g., motivation to transfer, post-training self-efficacy), formative assessments, GPA, pass-fail, met standard, etc. Some measures, such as reaction measures focused on the instructor, approached 60% of the variability accounted for by instructor assignment.

     

    Having looked at instructor impact over a number of military instructional contexts, it is clear that some contexts allow instructor differences to influence learning and outcomes more than others. As an advocate of Johns (2006), I think the context can constrain or facilitate the impact of the instructor on learning and learning outcomes.  As we are trying to improve training effectiveness for our clients, when instructors account for significant variability in learning outcomes, our focus is on understanding the factors driving this instructor variability in outcomes and how we can create interventions to reduce instructor variability while increasing learning/learning outcomes. Since instructors have their influence on student learning through their decisions and classroom performance, our goal is to identify and improve the high-impact instructor performance factors linked to learning or its proxy, achievement scores.

     

    We have about a decade's worth of data on various studies and interventions that followed from the initial work. We have focused on instructor behaviors in the classroom. Ken Brown and others have been after to me to publish our work. But, I haven't gotten around to it. Earlier this year, I presented a summary presentation on our instructor work related to language learning to a mixed audience of practitioners and researchers from Government agencies. I would be more than happy to share it or the SIOP presentations above, if you think they are useful. Again, they are military learners, not college students. So, related information but not exactly what you wanted. However, regardless of learner population, our work reinforces that instructors/teachers/trainers can have a meaningful impact on learning as measured by standardized proficiency assessment and other metrics.

     

    Best,

    Dr. Eric A. Surface
    President and Principal I/O Psychologist    , ALPS Solutions

    Secretary, Society for Military Psychology (Division 19 APA), 2014-2016

    Fellow, American Psychological Association and Society for Military Psychology

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Aguinis, Herman
    Sent: Thursday, December 10, 2015 6:18 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    Mike,

     

    I am not aware of studies examining the degree of variability in achievement attributed to teachers. But, there is evidence that variability in the relation between test scores (e.g., SAT) and achievement (e.g., GPA) is due to colleges. Specifically, the following article describes the presence of variability in test bias (i.e., differential prediction) across institutions and, overall, calls raises questions about the fairness of using the SAT as a predictor of GPA:

     

    ·         Aguinis, H., Culpepper, S.A., & Pierce, C.A. (in press). Differential prediction generalization in college admissions testing. Journal of Educational Psychology. [available at http://mypage.iu.edu/~haguinis/pubs.html]

     

    Abstract

    We introduce the concept of differential prediction generalization in the context of college admissions testing. Specifically, we assess the extent to which predicted first-year college grade-point average (GPA) based on high-school grade point average (HSGPA) and SAT scores depends on a student's ethnicity and gender and whether this difference varies across samples. We compared 257,336 female and 220,433 male students across 339 samples, 29,734 Black and 304,372 White students across 247 samples, and 35,681 Hispanic and 308,818 White students across 264 samples collected from 176 colleges and universities between the years 2006 and 2008. Overall, results show a lack of differential prediction generalization because variability remains after accounting for methodological and statistical artifacts including sample size, range restriction, proportion of students across ethnicity- and gender-based subgroups, subgroup mean differences on the predictors (i.e., HSGPA, SAT-Critical Reading, SAT-Math, and SAT-Writing), and standard deviations for the predictors. We offer an agenda for future research aimed at understanding several contextual reasons for a lack of differential prediction generalization based on ethnicity and gender. Results from such research will likely lead to a better understanding of the reasons for differential prediction and interventions aimed at reducing or eliminating it when it exists.

     

    I hope this helps!

     

    All the best,

     

    --Herman.

     

    Herman Aguinis

    John F. Mee Chair of Management

    Professor of Organizational Behavior and Human Resources

    Founding and Managing Director, Institute for Global Organizational Effectiveness

     

    Indiana University

    Kelley School of Business

    http://mypage.iu.edu/~haguinis/

     

    GO FROM MOMENT TO MOMENTUM

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: Wednesday, December 09, 2015 3:40 PM
    To:     OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    I recognize that my question would be best addressed to an educational research listserv but I don't belong to any of them. Also, most all members of this listserv are college faculty, so maybe you know.

     

    The question:  With reference to the academic achievement of college students, what percent of the variability of academic achievement can be attributed to teachers?

     

    The American Statistical Association issued a policy statement in 2014 that concluded that "teachers account for about 1% to 14% of the variability in test scores" (available at http://www.amstat.org/policy/pdfs/asa_vam_statement.pdf). But, as far as I can tell, the data reviewed were not from college students.

     

    Of relevance, there is an interesting book "G is for Genes: The Impact of Genetics on Education and Achievement" by K. Asbury and Robert Plomin (2014). Like intelligence ("g", not the make-believe intelligences like emotional, practical, football, etc.), educational achievement is substantially heritable and all of the environmental sources of educational achievement  are necessarily malleable. 

     

    In sum, can anyone refer me to a literature that directly addresses what percent of the variability of college academic achievement can be attributed to teachers?

     

    Thanks,

     

    Mike

     

     

    Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 

    Richmond, VA 23284-4000

    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu

    Research Professor, VCU Department of Psychology

     



    19 W. Hargett Street | Suite 701 | Raleigh, NC 27601

    This email, including any attachments, is only for the intended addressee. It is subject to copyright, is confidential and may be the subject of legal or other privilege, none of which is waived or lost by reason of this transmission. If the receiver is not the intended addressee, please accept our apologies, notify us by return, delete all copies and perform no other act on the email. Unfortunately, we cannot warrant that the email has not been altered or corrupted during transmission. Thank you.


  • 4.  How much variability in college academic achievement is due to teachers?

    Posted 12-11-2015 09:08

    Hello Mike,


    I think the two articles below might be useful and both have cool data.


    All the best,


    George


    Dale, B. S. & Kreuger, A. B. (1999). Estimating the payoff to attending a more selective college: An application of selection on observables and unobservables.


    Abstract:


    There are many estimates of the effect of college quality on students' subsequent earnings. One difficulty interpreting past estimates, however, is that elite colleges admit students, in part, based on characteristics that are related to their earnings capacity. Since some of these characteristics are unobserved by researchers who later estimate wage equations, it is difficult to parse out the effect of attending a selective college from the students' pre-college characteristics. This paper uses information on the set of colleges at which students were accepted and rejected to remove the effect of unobserved characteristics that influence college admission. Specifically, we match students in the newly colleted College and Beyond (C&B) Data Set who were admitted to and rejected from a similar set of institutions, and estimate fixed effects models. As another approach to adjust for selection bias, we control for the average SAT score of the schools to which students applied using both the C&B and National Longitudinal Survey of the High School Class of 1972. We find that students who attended more selective colleges do not earn more than other students who were accepted and rejected by comparable schools but attended less selective colleges. However, the average tuition charged by the school is significantly related to the students' subsequent earnings. Indeed, we find a substantial internal rate of return from attending a more costly college. Lastly, the payoff to attending an elite college appears to be greater for students from more disadvantaged family backgrounds.


    Fuller, A. (2015). Parents' fears confirmed: Liberal arts students earn less.


    http://www.wsj.com/articles/parents-fears-confirmed-liberal-arts-students-make-less-1446582592

     


    On Thu, Dec 10, 2015 at 7:51 PM, Eric Surface <esurface@alpssols.com> wrote:

    Mike,

    Although this is not college academic achievement, we have done substantial work since 2006 investigating foreign language learning in the military. In many cases, the Government assessments and courses use a similar proficiency framework to those used in universities and K-12 (e.g., the American Council on the Teaching of Foreign Languages [ACTFL] scale came out of the Government's ILR scale). Our work includes investigating the impact of instructor differences on various learning outcomes. Kemp Ellington and I presented some of this work at SIOP:

     

    Surface, E. A., & Ellington, J. K. (2008, April). Rethinking context in training effectiveness research: Instructor as learning context . A paper presented in E. C. Dierdorff's symposium [Illuminating the 'Murky Ground': Linking Context Theory to Empirical Research] at the 23rd annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.

     

    Ellington, J. K., & Surface, E. A. (2009, April). The trainer's impact in the learning environment: A multilevel examination. In A. Towler and S. Bell (Chairs), The people factor: Considering trainee and trainer effects on learning. Symposium presented at the 24th annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.

     

    In our initial study of archival data, we investigated instructor influence using a 3-level model in HLM (students within classes within instructors-instructors taught multiple classes/sections over time but same course and learn population) and standardized proficiency assessment scores in reading, listening and speaking. We found that instructors accounted for up to 41% of the variability in the learner scores. In subsequence studies, we found significant variability in reaction measures, affective outcomes (e.g., motivation to transfer, post-training self-efficacy), formative assessments, GPA, pass-fail, met standard, etc. Some measures, such as reaction measures focused on the instructor, approached 60% of the variability accounted for by instructor assignment.

     

    Having looked at instructor impact over a number of military instructional contexts, it is clear that some contexts allow instructor differences to influence learning and outcomes more than others. As an advocate of Johns (2006), I think the context can constrain or facilitate the impact of the instructor on learning and learning outcomes.  As we are trying to improve training effectiveness for our clients, when instructors account for significant variability in learning outcomes, our focus is on understanding the factors driving this instructor variability in outcomes and how we can create interventions to reduce instructor variability while increasing learning/learning outcomes. Since instructors have their influence on student learning through their decisions and classroom performance, our goal is to identify and improve the high-impact instructor performance factors linked to learning or its proxy, achievement scores.

     

    We have about a decade's worth of data on various studies and interventions that followed from the initial work. We have focused on instructor behaviors in the classroom. Ken Brown and others have been after to me to publish our work. But, I haven't gotten around to it. Earlier this year, I presented a summary presentation on our instructor work related to language learning to a mixed audience of practitioners and researchers from Government agencies. I would be more than happy to share it or the SIOP presentations above, if you think they are useful. Again, they are military learners, not college students. So, related information but not exactly what you wanted. However, regardless of learner population, our work reinforces that instructors/teachers/trainers can have a meaningful impact on learning as measured by standardized proficiency assessment and other metrics.

     

    Best,

    Dr. Eric A. Surface
    President and Principal I/O Psychologist    , ALPS Solutions

    Secretary, Society for Military Psychology (Division 19 APA), 2014-2016

    Fellow, American Psychological Association and Society for Military Psychology

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Aguinis, Herman
    Sent: Thursday, December 10, 2015 6:18 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    Mike,

     

    I am not aware of studies examining the degree of variability in achievement attributed to teachers. But, there is evidence that variability in the relation between test scores (e.g., SAT) and achievement (e.g., GPA) is due to colleges. Specifically, the following article describes the presence of variability in test bias (i.e., differential prediction) across institutions and, overall, calls raises questions about the fairness of using the SAT as a predictor of GPA:

     

    ·         Aguinis, H., Culpepper, S.A., & Pierce, C.A. (in press). Differential prediction generalization in college admissions testing. Journal of Educational Psychology. [available at http://mypage.iu.edu/~haguinis/pubs.html]

     

    Abstract

    We introduce the concept of differential prediction generalization in the context of college admissions testing. Specifically, we assess the extent to which predicted first-year college grade-point average (GPA) based on high-school grade point average (HSGPA) and SAT scores depends on a student's ethnicity and gender and whether this difference varies across samples. We compared 257,336 female and 220,433 male students across 339 samples, 29,734 Black and 304,372 White students across 247 samples, and 35,681 Hispanic and 308,818 White students across 264 samples collected from 176 colleges and universities between the years 2006 and 2008. Overall, results show a lack of differential prediction generalization because variability remains after accounting for methodological and statistical artifacts including sample size, range restriction, proportion of students across ethnicity- and gender-based subgroups, subgroup mean differences on the predictors (i.e., HSGPA, SAT-Critical Reading, SAT-Math, and SAT-Writing), and standard deviations for the predictors. We offer an agenda for future research aimed at understanding several contextual reasons for a lack of differential prediction generalization based on ethnicity and gender. Results from such research will likely lead to a better understanding of the reasons for differential prediction and interventions aimed at reducing or eliminating it when it exists.

     

    I hope this helps!

     

    All the best,

     

    --Herman.

     

    Herman Aguinis

    John F. Mee Chair of Management

    Professor of Organizational Behavior and Human Resources

    Founding and Managing Director, Institute for Global Organizational Effectiveness

     

    Indiana University

    Kelley School of Business

    http://mypage.iu.edu/~haguinis/

     

    GO FROM MOMENT TO MOMENTUM

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: Wednesday, December 09, 2015 3:40 PM
    To:     OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    I recognize that my question would be best addressed to an educational research listserv but I don't belong to any of them. Also, most all members of this listserv are college faculty, so maybe you know.

     

    The question:  With reference to the academic achievement of college students, what percent of the variability of academic achievement can be attributed to teachers?

     

    The American Statistical Association issued a policy statement in 2014 that concluded that "teachers account for about 1% to 14% of the variability in test scores" (available at http://www.amstat.org/policy/pdfs/asa_vam_statement.pdf). But, as far as I can tell, the data reviewed were not from college students.

     

    Of relevance, there is an interesting book "G is for Genes: The Impact of Genetics on Education and Achievement" by K. Asbury and Robert Plomin (2014). Like intelligence ("g", not the make-believe intelligences like emotional, practical, football, etc.), educational achievement is substantially heritable and all of the environmental sources of educational achievement  are necessarily malleable. 

     

    In sum, can anyone refer me to a literature that directly addresses what percent of the variability of college academic achievement can be attributed to teachers?

     

    Thanks,

     

    Mike

     

     

    Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 

    Richmond, VA 23284-4000

    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu

    Research Professor, VCU Department of Psychology

     



    19 W. Hargett Street | Suite 701 | Raleigh, NC 27601

    This email, including any attachments, is only for the intended addressee. It is subject to copyright, is confidential and may be the subject of legal or other privilege, none of which is waived or lost by reason of this transmission. If the receiver is not the intended addressee, please accept our apologies, notify us by return, delete all copies and perform no other act on the email. Unfortunately, we cannot warrant that the email has not been altered or corrupted during transmission. Thank you.



    --
    George C. Banks, Ph.D.
    Assistant Professor of Management
    The Belk College of Business
    University of North Carolina at Charlotte


  • 5.  How much variability in college academic achievement is due to teachers?

    Posted 12-11-2015 13:33
    Herman,

    Thank you for your citation. I am more persuaded by the Sackett & Kuncel research coming out of their 1,000,000 + data base of SAT scores that makes it clear there is no underprediction of minority academic performance with the SAT. They made a presentation of some of their work at the September 2015 meeting of the International Society of Intelligence Research (ISIR). The video of their presentation was placed on You Tube is available here: 

    Thank you for sharing your research.

    Best wishes,

    Mike

    Professor and National Treasure, Department of Management, 
    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 
    Richmond, VA 23284-4000
    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu
    Research Professor, VCU Department of Psychology


    On Thu, Dec 10, 2015 at 6:18 PM, Aguinis, Herman <haguinis@indiana.edu> wrote:

    Mike,

     

    I am not aware of studies examining the degree of variability in achievement attributed to teachers. But, there is evidence that variability in the relation between test scores (e.g., SAT) and achievement (e.g., GPA) is due to colleges. Specifically, the following article describes the presence of variability in test bias (i.e., differential prediction) across institutions and, overall, calls raises questions about the fairness of using the SAT as a predictor of GPA:

     

    ·       Aguinis, H., Culpepper, S.A., & Pierce, C.A. (in press). Differential prediction generalization in college admissions testing. Journal of Educational Psychology. [available at http://mypage.iu.edu/~haguinis/pubs.html]

     

    Abstract

    We introduce the concept of differential prediction generalization in the context of college admissions testing. Specifically, we assess the extent to which predicted first-year college grade-point average (GPA) based on high-school grade point average (HSGPA) and SAT scores depends on a student's ethnicity and gender and whether this difference varies across samples. We compared 257,336 female and 220,433 male students across 339 samples, 29,734 Black and 304,372 White students across 247 samples, and 35,681 Hispanic and 308,818 White students across 264 samples collected from 176 colleges and universities between the years 2006 and 2008. Overall, results show a lack of differential prediction generalization because variability remains after accounting for methodological and statistical artifacts including sample size, range restriction, proportion of students across ethnicity- and gender-based subgroups, subgroup mean differences on the predictors (i.e., HSGPA, SAT-Critical Reading, SAT-Math, and SAT-Writing), and standard deviations for the predictors. We offer an agenda for future research aimed at understanding several contextual reasons for a lack of differential prediction generalization based on ethnicity and gender. Results from such research will likely lead to a better understanding of the reasons for differential prediction and interventions aimed at reducing or eliminating it when it exists.

     

    I hope this helps!

     

    All the best,

     

    --Herman.

     

    Herman Aguinis

    John F. Mee Chair of Management

    Professor of Organizational Behavior and Human Resources

    Founding and Managing Director, Institute for Global Organizational Effectiveness

     

    Indiana University

    Kelley School of Business

    http://mypage.iu.edu/~haguinis/

     

    GO FROM MOMENT TO MOMENTUM

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: Wednesday, December 09, 2015 3:40 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    I recognize that my question would be best addressed to an educational research listserv but I don't belong to any of them. Also, most all members of this listserv are college faculty, so maybe you know.

     

    The question:  With reference to the academic achievement of college students, what percent of the variability of academic achievement can be attributed to teachers?

     

    The American Statistical Association issued a policy statement in 2014 that concluded that "teachers account for about 1% to 14% of the variability in test scores" (available at http://www.amstat.org/policy/pdfs/asa_vam_statement.pdf). But, as far as I can tell, the data reviewed were not from college students.

     

    Of relevance, there is an interesting book "G is for Genes: The Impact of Genetics on Education and Achievement" by K. Asbury and Robert Plomin (2014). Like intelligence ("g", not the make-believe intelligences like emotional, practical, football, etc.), educational achievement is substantially heritable and all of the environmental sources of educational achievement  are necessarily malleable. 

     

    In sum, can anyone refer me to a literature that directly addresses what percent of the variability of college academic achievement can be attributed to teachers?

     

    Thanks,

     

    Mike

     

     

    Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 

    Richmond, VA 23284-4000

    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu

    Research Professor, VCU Department of Psychology

     




  • 6.  How much variability in college academic achievement is due to teachers?

    Posted 12-11-2015 21:27

    Mike (cc: OB list),

     

    I appreciate your skepticism given that the topic of differential prediction has been dormant for decades. Also, I appreciate your sharing the youtube video which you introduced by saying that the speakers' research has been funded by the College Board (vendor of the SAT). Their presentation (starting at about minute 37) describes research addressing the average degree of differential prediction (i.e., predictive bias) across colleges. The conclusion is that, on average and across colleges, there is no underprediction against minorities or women and our own study replicated that same result. However, as we noted in our article to appear in Journal of Educational Psychology [available at http://mypage.iu.edu/~haguinis/pubs.html], our results reveal that the conclusion of past research suggesting that the use of the SAT results in minimal differential prediction is only reached when we examine summary statistics collapsing across the 348 samples collected from the 176 colleges and universities. In contrast, differential prediction generalization analysis suggests that there is substantial variability in differential prediction across samples. In other words, differential prediction in some colleges benefits Blacks (women) and in others benefits Whites (men), but averaging across colleges cancels out these effects giving the impression that there is no differential prediction and the test predicts achievement similarly across groups.

     

    Why is it that in some contexts there are prediction differences in favor of Black students whereas for others the opposite is true? We currently do not know which institution-level factors cause differential prediction, and which particular form of differential prediction, across contexts. Our results point to the need for future research to examine which factors cause differential prediction to vary in magnitude and direction across contexts. To guide future research, our article offers a detailed description of how and why each of several mechanisms may serve as possible explanations for the presence of differential prediction and differential prediction variability across institutions. In terms of practice, our results suggest that lack of differential prediction when using the SAT cannot be assumed in making college admissions decisions. Depending on the institution and its local practices (e.g., admissions, grading, affirmative action policies), and various contextual and societal factors, it is possible that there may be differential prediction-and the form of such differential prediction is unlikely to be the same across samples. Obviously, these are important societal issues given that, since 2005, between 1.4 million and 1.7 million students have taken the SAT annually. I think this offers a great opportunity for us (i.e., OB/HR/I-O psychology researchers) to conduct further research on these important issues with clear implications for individuals, organizations, and society in general.

     

    I will be glad to continue this exchange outside of the OB list if you have any specific questions.

     

    All the best,

     

    --Herman.

     

    Herman Aguinis

    John F. Mee Chair of Management

    Professor of Organizational Behavior and Human Resources

    Founding and Managing Director, Institute for Global Organizational Effectiveness

     

    Indiana University

    Kelley School of Business

    http://mypage.iu.edu/~haguinis/

     

    GO FROM MOMENT TO MOMENTUM

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: Friday, December 11, 2015 1:33 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    Herman,

     

    Thank you for your citation. I am more persuaded by the Sackett & Kuncel research coming out of their 1,000,000 + data base of SAT scores that makes it clear there is no underprediction of minority academic performance with the SAT. They made a presentation of some of their work at the September 2015 meeting of the International Society of Intelligence Research (ISIR). The video of their presentation was placed on You Tube is available here: 

     

    Thank you for sharing your research.

     

    Best wishes,

     

    Mike


    Professor and National Treasure, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 

    Richmond, VA 23284-4000

    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu

    Research Professor, VCU Department of Psychology

     

     

    On Thu, Dec 10, 2015 at 6:18 PM, Aguinis, Herman <haguinis@indiana.edu> wrote:

    Mike,

     

    I am not aware of studies examining the degree of variability in achievement attributed to teachers. But, there is evidence that variability in the relation between test scores (e.g., SAT) and achievement (e.g., GPA) is due to colleges. Specifically, the following article describes the presence of variability in test bias (i.e., differential prediction) across institutions and, overall, calls raises questions about the fairness of using the SAT as a predictor of GPA:

     

    ·       Aguinis, H., Culpepper, S.A., & Pierce, C.A. (in press). Differential prediction generalization in college admissions testing. Journal of Educational Psychology. [available at http://mypage.iu.edu/~haguinis/pubs.html]

     

    Abstract

    We introduce the concept of differential prediction generalization in the context of college admissions testing. Specifically, we assess the extent to which predicted first-year college grade-point average (GPA) based on high-school grade point average (HSGPA) and SAT scores depends on a student's ethnicity and gender and whether this difference varies across samples. We compared 257,336 female and 220,433 male students across 339 samples, 29,734 Black and 304,372 White students across 247 samples, and 35,681 Hispanic and 308,818 White students across 264 samples collected from 176 colleges and universities between the years 2006 and 2008. Overall, results show a lack of differential prediction generalization because variability remains after accounting for methodological and statistical artifacts including sample size, range restriction, proportion of students across ethnicity- and gender-based subgroups, subgroup mean differences on the predictors (i.e., HSGPA, SAT-Critical Reading, SAT-Math, and SAT-Writing), and standard deviations for the predictors. We offer an agenda for future research aimed at understanding several contextual reasons for a lack of differential prediction generalization based on ethnicity and gender. Results from such research will likely lead to a better understanding of the reasons for differential prediction and interventions aimed at reducing or eliminating it when it exists.

     

    I hope this helps!

     

    All the best,

     

    --Herman.

     

    Herman Aguinis

    John F. Mee Chair of Management

    Professor of Organizational Behavior and Human Resources

    Founding and Managing Director, Institute for Global Organizational Effectiveness

     

    Indiana University

    Kelley School of Business

    http://mypage.iu.edu/~haguinis/

     

    GO FROM MOMENT TO MOMENTUM

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: Wednesday, December 09, 2015 3:40 PM
    To:     OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    I recognize that my question would be best addressed to an educational research listserv but I don't belong to any of them. Also, most all members of this listserv are college faculty, so maybe you know.

     

    The question:  With reference to the academic achievement of college students, what percent of the variability of academic achievement can be attributed to teachers?

     

    The American Statistical Association issued a policy statement in 2014 that concluded that "teachers account for about 1% to 14% of the variability in test scores" (available at http://www.amstat.org/policy/pdfs/asa_vam_statement.pdf). But, as far as I can tell, the data reviewed were not from college students.

     

    Of relevance, there is an interesting book "G is for Genes: The Impact of Genetics on Education and Achievement" by K. Asbury and Robert Plomin (2014). Like intelligence ("g", not the make-believe intelligences like emotional, practical, football, etc.), educational achievement is substantially heritable and all of the environmental sources of educational achievement  are necessarily malleable. 

     

    In sum, can anyone refer me to a literature that directly addresses what percent of the variability of college academic achievement can be attributed to teachers?

     

    Thanks,

     

    Mike

     

     

    Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 

    Richmond, VA 23284-4000

    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu

    Research Professor, VCU Department of Psychology

     

     



  • 7.  How much variability in college academic achievement is due to teachers?

    Posted 12-11-2015 03:34

    Here is an article that discusses the role of individual teacher's on student learning. One of the conclusions is that the theacher effect is overrated due designs that do not take into account that the teacher is nested within a team, which in turn is nested within a school, a district and so on.

     

    Hallinger, P., Heck, R. H., & Murphy, J. (2014). Teacher evaluation and school improvement: An analysis of the evidence. Educational Assessment Evaluation and Accountability, 26(1), 5-28. doi:10.1007/s11092-013-9179-5

     

     

    Bård Kuvaas
    Professor - Department of Leadership and Organizational Behaviour
    Visiting address: Nydalsveien 37, 0484 Oslo
    Mailing address: NO-0442 Oslo
    Direct: +4746410731 - Switchboard: +4746410000
    Internet: www.bi.no

    Twitter: @BKuvaas

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of Michael A McDaniel
    Sent: 9. desember 2015 21:40
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] How much variability in college academic achievement is due to teachers?

     

    I recognize that my question would be best addressed to an educational research listserv but I don't belong to any of them. Also, most all members of this listserv are college faculty, so maybe you know.

     

    The question:  With reference to the academic achievement of college students, what percent of the variability of academic achievement can be attributed to teachers?

     

    The American Statistical Association issued a policy statement in 2014 that concluded that "teachers account for about 1% to 14% of the variability in test scores" (available at http://www.amstat.org/policy/pdfs/asa_vam_statement.pdf). But, as far as I can tell, the data reviewed were not from college students.

     

    Of relevance, there is an interesting book "G is for Genes: The Impact of Genetics on Education and Achievement" by K. Asbury and Robert Plomin (2014). Like intelligence ("g", not the make-believe intelligences like emotional, practical, football, etc.), educational achievement is substantially heritable and all of the environmental sources of educational achievement  are necessarily malleable. 

     

    In sum, can anyone refer me to a literature that directly addresses what percent of the variability of college academic achievement can be attributed to teachers?

     

    Thanks,

     

    Mike

     

     

    Professor, Department of Management, 

    Virginia Commonwealth University 301 West Main Street, PO Box 844000, 

    Richmond, VA 23284-4000

    voice: 804.827.0209     e-mail: MAMcDani@vcu.edu

    Research Professor, VCU Department of Psychology