Christin,I empathize with the problem of finding a decent sample for one's dissertation or other research. Just keep in mind that opt-in panel samples are convenience samples with all of the qualifications that come with them. Just because participants happen to be distributed across some country does not make them a representative national sample, same can be said about any other geographic level or occupation/industry. So as samples go in organizational and other social science research, they are probably no worse than any other method of obtaining a convenience sample. However, convenience samples can differ in terms of data quality. If those in opt-in samples only do the survey, and do it quickly with little motivation, to obtain their "tokens," they could provide poorer quality data than obtaining a sample from an organization where there is some direct contact between participants and researchers. I'm not saying this additional problem is inevitable, but it is something to consider. I contacted several panel companies to see what kind of samples I could get. For a basic convenience sample. it seemed that Qualtrics was inexpensive and very responsive and accommodating in terms of customer service. Below Therese mentioned a panel company that was not cheap. I'm not sure what not cheap means, but if one has reasonable funding, there is a panel company that should be considered. Knowledge Networks is unique among panel samples. It is not opt-in. One cannot volunteer to be in the sample. They use well-designed, random digit dialing to randomly select people from the US population. If the selected person is interested and does not have internet access, they set them up with free of charge. There has been some research comparing results from RDD data collection, Knowledge networks, and several opt-in panels. The results from the Knowledge Networks sample was similar to RDD (which includes cell phones these days), but the opt-in panel samples had more error. Basically with RDD and a Knowledge networks panel, you will have a probability sample and can compute sampling weights, which Knowledge Networks does for a study. All other opt-in panels are typical convenience samples. The downside of Knowledge Networks, is that they are not cheap by most researchers' standards. But as with many things in life, your get what you pay for--good samples are not cheap in general, and everything is relative. A colleague is using them to do a cross-sectional survey of 1,000 adolescents, and a 6-wave panel survey over two years with 2,000 young adults. Knowledge Networks does all data management. The cost is about $600,000. As a comparison, the last RDD study that I did in-house a few years ago was single wave of about 3,000 completed interviews and cost me around 2.2 million in direct and indirect costs. Knowledge Networks cuts out the need for a very large staff and the huge personnel costs that go with it. Studies with smaller samples and less ambitious study designs can be much cheaper. For example, for a study with the following characteristics:
· Pretest: n=25 interviews
· Sample: Employed adults, age 18+, English-language survey-takers
· Screening: Up to 1 minute to confirm current status as employed
· Number of completed interviews: n=400
· Median survey length: 30 or 45 minutes
· Incentive: $5 for 30 minute, $10 for 45 minutes
· Multimedia/Graphics: None
· GfK will field survey in 2014
The cost is about $31,000 for a 30 minute survey and about $44,000 for a 45 minutes survey. I know these costs may seem high for most, but if one has the resources and wants a probability sample, this a very cost-efficient means of getting such a sample compared to an RDD survey. Household surveys have been out of reach for a long time, and frankly costs for RDD studies depending on who is fielding the survey are getting out of reach even if one has substantial funding.
Of course, the samples required for many research questions (group processes) would be very difficult or impossible to obtain through probability sampling. But I thought that some might be interesting in knowing about Knowledge Networks if they haven't heard of this company.
Mike Frone
********************************************************************************
Michael R. Frone, Ph.D.
Senior Research Scientist
Research Institute on Addictions
State University of New York at Buffalo
1021 Main Street
Buffalo, New York 14203
Office: 716-887-2519
Fax: 716-887-2477
E-mail: frone@ria.buffalo.edu
Website: http://www.buffalo.edu/ria/staff/scientists/mfrone.html
*******************************************************************************
Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU> wrote on 11/19/2013 09:37:22 AM:
> Therese Sprinkle <therese.a.sprinkle@GMAIL.COM>
> Sent by: Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU>
>
> 11/19/2013 09:37 AM
>
> Please respond to
> Organizational Behavior Division Listserv <OB@AOMLISTS.PACE.EDU>
>
> To
>
> OB@AOMLISTS.PACE.EDU
>
> cc
>
> Subject
>
> Re: [OB-LIST] Advice Needed: Participant Recruitment with Ready Samples
>
> Hi Chris
> I worked with Toluna, Inc on my dissertation. They have a US panel
> of over 800,000 and were able to get 500 working adults, spread
> across age groups and balanced. They aren't cheap though, that
> portion of my study was funded.
> Therese Sprinkle,PhD
> University of Dallas
> On Nov 18, 2013 12:21 PM, "Christin Moeller" <moellerc@uwindsor.ca> wrote:
> **Apologies for cross-postings!**
>
> Hello all,
>
> My name is Christin and I'm a PhD Candidate at the University of
> Windsor. For my dissertation, I'm considering using a "ready sample"
> from an online service provider (i.e., paying an online recruitment
> service for participant responses).
>
> I've worked with StudyResponse for past investigations, but I'm
> wondering whether you can recommend any alternatives? If you have
> used any participant recruitment services in the past, what has your
> experience been (e.g. with regards to the quality of the data; the
> time needed for data collection)?
>
> I'd greatly appreciate any advice or insights you have!
>
> Thank you,
>
> Christin
>
> Christin Moeller, M.A. (Ph.D. Candidate)
> Operations Director & Lab Manager - Occupational Health & Well-Being
> Research Group
> Department of Psychology
> University of Windsor
> Windsor, ON Canada N9B3P4
> moellerc@uwindsor.ca