Discussion: View Thread

  • 1.  MTurk

    Posted 01-20-2014 11:59
    Hi Netters

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

    Best
    John


  • 2.  MTurk

    Posted 01-20-2014 12:35

    John,

     

    The following article describes the use (including logistical issues) regarding Mechanical Turk as well as other similar eLancing websites (e.g., guru.com, etc.):

     

    Aguinis, H., & Lawal, S. O. 2012. Conducting field experiments using eLancing's natural environment. Journal of Business Venturing, 27: 493-505-available at http://mypage.iu.edu/~haguinis/pubs.html

     

    I hope this helps!


    Regards,

     

    --Herman.

     

    Herman Aguinis, Ph.D.

    John F. Mee Chair of Management

    Professor of Organizational Behavior and Human Resources

    Founding Director, Institute for Global Organizational Effectiveness

    Department of Management and Entrepreneurship

    http://mypage.iu.edu/~haguinis/

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of John Michel
    Sent: Monday, January 20, 2014 11:59 AM
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] MTurk

     

    Hi Netters

     

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

     

    Best

    John



  • 3.  MTurk

    Posted 01-20-2014 13:35
    Greetings John,

    Here is a good blog post about Mturk and the quality of the data and other issues pertaining to that method:


    Also, I know of a number of publications in A journals that obtained their data from Mturk so it is definitely a viable resource.

    Best of luck,


    On Mon, Jan 20, 2014 at 8:59 AM, John Michel <jwmichel@loyola.edu> wrote:
    Hi Netters

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

    Best
    John



    --
    Bradford E. Baker
    PhD Student, Management & Organization
    Robert H. Smith School of Business
    University of Maryland
    206.499.8427


  • 4.  MTurk

    Posted 01-20-2014 14:19

    John

    You probably want to take a look at an AMJ paper using MTurk

    http://amj.aom.org/cgi/content/abstract/56/6/1545

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of John Michel
    Sent: Monday, January 20, 2014 11:59 AM
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] MTurk

     

    Hi Netters

     

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

     

    Best

    John



  • 5.  MTurk

    Posted 01-20-2014 17:01
    Dear John, the following reference might be of your interest:

    Amazon's Mechanical Turk : A New Source of Inexpensive, Yet High-Quality, Data?
    Michael Buhrmester, Tracy Kwang and Samuel D. Gosling Perspectives on Psychological Science 2011 6: 3 DOI: 10.1177/1745691610393980

    Good luck!

    Hector

    Hector Madrid, PhD
    Assistant Professor
    School of Psychology
    Pontificia Universidad Católica de Chile
    University Profile: hector_madrid


    On 20 January 2014 16:19, Kai Wang <wangkaicv@gmail.com> wrote:

    John

    You probably want to take a look at an AMJ paper using MTurk

    http://amj.aom.org/cgi/content/abstract/56/6/1545

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of John Michel
    Sent: Monday, January 20, 2014 11:59 AM
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] MTurk

     

    Hi Netters

     

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

     

    Best

    John




  • 6.  MTurk

    Posted 01-20-2014 17:31
    I have used MTurk for subjects and it has worked well. Your friend should definitely do some research about how to best use it, as there can be some pitfalls.

    One issue is that a huge percentage of Turkers are in India. I mistakenly didn't specify countries on a batch thinking I would get an international sample and I ended up almost exclusively with Indian respondents.

    Another issues is that there is some gaming of MTurk so you have to be creative about building checks into your task so that you can eliminate the non-conscientious subjects.

    Jeff Peterson, Ph.D.
    jeff.peterson@uvu.edu
    801-863-6026 (work)
    425-773-6262 (mobile)
    Sent from my iPhone

    > On Jan 20, 2014, at 9:59 AM, John Michel <jwmichel@LOYOLA.EDU> wrote:
    >
    > Hi Netters
    >
    > I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?
    >
    > Best
    > John


  • 7.  MTurk

    Posted 02-10-2014 12:18

    John,

     

    mTurk is now common in studies of decision-making and in experimental economics. Initially, there were concerns about validity, especially since the identities of participants are unverifiable. The references below discuss replicability by showing that experiments using mTurk population generally produce the same results as those in the laboratory. They also discuss some benefits of mTurk that are not available in laboratory research. I hope you'll find these useful.

     

    Horton, J., Rand, D., & Zeckhauser, R. (2011). The online laboratory: conducting experiments in a real labor market. Experimental Economics, 1-27. doi: 10.1007/s10683-011-9273-9

    Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411-419.

    Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1), 155-167. doi: 10.3758/s13428-010-0039-7

     

     

    Warmly,

     

    SSL

     

    Sheen S. Levine, PhD

    Institute for Social and Economic Research and Policy

    Columbia University

     

    twitter.com/sslevine

     

    Open Collaboration: Principles and Performance

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of John Michel
    Sent: Monday, January 20, 2014 11:59
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] MTurk

     

    Hi Netters

     

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

     

    Best

    John



  • 8.  MTurk

    Posted 02-10-2014 12:53

    John,

    Here is another citation as well:

    Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PloS one, 8, e57410. doi: 10.1371/journal.pone.0057410

     

    Erin Cooley
    Doctoral Candidate & Teaching Fellow
    Social Psychology Program
    University of North Carolina - Chapel Hill
    325 Davie Hall
    From: Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU> on behalf of Sheen S Levine <sslevine@COLUMBIA.EDU>
    Sent: Monday, February 10, 2014 12:17 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] MTurk
     

    John,

     

    mTurk is now common in studies of decision-making and in experimental economics. Initially, there were concerns about validity, especially since the identities of participants are unverifiable. The references below discuss replicability by showing that experiments using mTurk population generally produce the same results as those in the laboratory. They also discuss some benefits of mTurk that are not available in laboratory research. I hope you'll find these useful.

     

    Horton, J., Rand, D., & Zeckhauser, R. (2011). The online laboratory: conducting experiments in a real labor market. Experimental Economics, 1-27. doi: 10.1007/s10683-011-9273-9

    Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411-419.

    Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1), 155-167. doi: 10.3758/s13428-010-0039-7

     

     

    Warmly,

     

    SSL

     

    Sheen S. Levine, PhD

    Institute for Social and Economic Research and Policy

    Columbia University

     

    twitter.com/sslevine

     

    Open Collaboration: Principles and Performance

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of John Michel
    Sent: Monday, January 20, 2014 11:59
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] MTurk

     

    Hi Netters

     

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

     

    Best

    John



  • 9.  MTurk

    Posted 02-10-2014 13:50
    Hi John, 

    I recently published a paper at AMLE replicating Mturk effects with MBA data and also discussed the validity of the sample in the general discussion. 

    Here is the citation: 

    Mor, S., Morris, M., & Joh, J. (2013). Identifying and Training Adaptive Cross-Cultural Management Skills: The Crucial Role of Cultural Metacognition. Academy of Management Learning & Education.

    Also, Gabriele Paolacci from RSM is an Mturk expert you may want to consult or read his blog:


    Good luck!
    Shira 

    On Feb 10, 2014, at 6:53 PM, "Cooley, Erin" <ecooley@LIVE.UNC.EDU> wrote:

    John,

    Here is another citation as well:

    Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PloS one, 8, e57410. doi: 10.1371/journal.pone.0057410

     

    Erin Cooley
    Doctoral Candidate & Teaching Fellow
    Social Psychology Program
    University of North Carolina - Chapel Hill
    325 Davie Hall
    From: Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU> on behalf of Sheen S Levine <sslevine@COLUMBIA.EDU>
    Sent: Monday, February 10, 2014 12:17 PM
    To: OB@AOMLISTS.PACE.EDU
    Subject: Re: [OB-LIST] MTurk
     

    John,

     

    mTurk is now common in studies of decision-making and in experimental economics. Initially, there were concerns about validity, especially since the identities of participants are unverifiable. The references below discuss replicability by showing that experiments using mTurk population generally produce the same results as those in the laboratory. They also discuss some benefits of mTurk that are not available in laboratory research. I hope you'll find these useful.

     

    Horton, J., Rand, D., & Zeckhauser, R. (2011). The online laboratory: conducting experiments in a real labor market. Experimental Economics, 1-27. doi: 10.1007/s10683-011-9273-9

    Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411-419.

    Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1), 155-167. doi: 10.3758/s13428-010-0039-7

     

     

    Warmly,

     

    SSL

     

    Sheen S. Levine, PhD

    Institute for Social and Economic Research and Policy

    Columbia University

     

    twitter.com/sslevine

     

    Open Collaboration: Principles and Performance

     

     

     

    From: Organizational Behavior Division Listserv [mailto:OB@AOMLISTS.PACE.EDU] On Behalf Of John Michel
    Sent: Monday, January 20, 2014 11:59
    To: OB@AOMLISTS.PACE.EDU
    Subject: [OB-LIST] MTurk

     

    Hi Netters

     

    I have a colleague who is getting ready to conduct a study using Mturk and Qualtrics as his data collection method. Does anyone know of any good published papers that have used this methodology? Also, does anyone have any experience getting such data through the process?

     

    Best

    John