Hi Jacqueline,
We used Qualtrics both with and without a panel partner to collect data for our pilot studies with my advisor. I would suggest the followings based on our experience.
- I think it works great especially if you want to pilot test your measures/surveys before distributing them to a larger sample of participants that you actually know. It saves you time and allows you to check if everything works fine for your next big data collection.
- I highly recommend using attention checking questions in your surveys. We usually use 3 to 4 of them depending on the length of our surveys. It may be a very simple math question (e.g., what does 2 plus three equal to?) or ask them to select an option for that question (e..g, Select Strongly Disagree for this question). Qualtrics allows you to drop out the respondents who failed to answer these questions correctly from your sample automatically.
- You can specify the characteristics of your sample in many ways such as requiring specific quotas to have variability on age, gender, education level, working status, managerial status, etc. My experience is that the more specific you are the more expensive is the quote. They generally suggest targeting a much higher sample size compared to your desired one. I think you may need to negotiate this based on what you really need and your budget.
- After launching the survey, they offer an option to pause collecting data and allow you to see the initial responses and check if everything works just fine (I think when it hits 10% of your desired sample size).
- The question formats are really diverse and helpful. You can select different validation options for each item such as forced-choice, request response, etc. It also allows a couple of specific entry formats such as numeric values or non-numeric values only.
- I suggest playing with the survey flow and advance survey options a bit to get a sense of what features are available. These are where you can set up conditions for your items or send remainder e-mails to your respondents, and so on. I was able to find out useful resources in the web about how to use those features.
-In our study with a panel partner, we used single-source multi-wave approach. If you are going to work with a panel partner, make sure to remind them that they will need to collect data from the same sample for your second study. This would prevent any confusion between Qualtrics and the panel partner, and help them set up some embedded questions to match the responses from both surveys.
- For another pilot study, we used Qualtrics to create our surveys and then recruit our sample from MTurk. I can say that the quality of the data that we collected via Qualtrics panel looked better than MTurk -but it may be due to several reasons such as we provided lower incentives to Mturk sample.
I hope these help. Good luck with your studies!
Lale
Lale M. YaldizDoctoral StudentGraduate Assistant Industrial/Organizational PsychologyPortland State UniversityP.O. Box 751Portland, OR 97207https://pdx.edu/psy
On May 15, 2015, at 11:13 AM, Justin DeSimone <
jad@UNL.EDU> wrote:
Hi Jacqueline,
I have used Qualtrics a number of times. Here is what I have learned so far:
1. For Likert-type items, I would recommend using the "matrix table" format instead of the 'multiple choice" format. You can get a lot more questions onto a single page that way and use it to organize your scales.
2. To my knowledge, there is no good method of presenting forced-choice items in Qualtrics. If anyone knows of one, please let us know.
3. The "slider" question type can be used if you are a fan of visual analog scales. It's not a perfect analog to the paper-and-pencil version, but it's closer than using a Likert-type system.
4. The "Survey Options" link has a few useful tools such as the ability to add a progress bar, the ability to prevent multiple responses from the same IP address, and the ability to enable/disable the use of the back button.
5. The "survey flow" link allows you to route participants based on their responses (e.g., ending the survey if they do not provide consent, randomizing the order of scales or items within a scale, or assigning half of the participants to fill out one set of items and the other half to fill out a different set).
6. If you are linking to Qualtrics from an online sample (e.g., MTurk), I highly recommend following the message boards (e.g.,
turkernation.com) to see what participants are saying about your survey and if anyone has found a way to "break" or "figure out" your scales or items (if this is a possibility).
7. The "timing" option in "Question Type" allows you to calculate the amount of time participants spend on the survey (or individual pages of the survey). This is an excellent tool for screening participants who do not take your survey seriously.
8. Another good screening technique is to include some "bogus" or "instructed" items as well as a few items at the end asking participants to rate how seriously they took the survey. You can find some good examples (as well as a bunch of other useful screening techniques in the references below (with apologies for the self-reference):
DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2014). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171-181.
Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299-311.
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort
responding to surveys. Journal of Business and Psychology, 27, 99–114.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
Good luck!
-Justin
On Thu, May 14, 2015 at 1:53 PM, Jacqueline Power
<powerja@uwindsor.ca> wrote:
Dear OB researchers,
I am looking for studies that have collected data using on-line sampling services such as Qualtrics or StudyResponse.
I would be grateful for any references that you could provide. Does anyone have any advice on the use of these services in
providing data for studies? Do you have any tips for dealing with methodological issues such as responder inattention or deception?
Many thanks,
Jacqueline L. Power
Associate Professor of Management
401 Sunset Avenue
Windsor, Ontario CANADA N9B 3P4
Email: powerja@uwindsor.ca
Ph: 519-253-3000 ext. 3111
Fax: 519-973-7073
*This message is intended only for the person or entity named above, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. Any disclosure, copying, or distribution of this message or the taking of any action based upon it, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please delete this message immediately and notify me by email. Thank you.