In the last module we talked about non-probability samples like convenience samples and purposive samples and how they can be great for research. In this module we're going to talk about our pros and cons of panels as a method of recruiting survey participants, as well as specific features of Mechanical Turk as a particular source for recruiting survey participants. So, just to define them, panels are basically a group of potential survey participants or respondents who have been previously arranged by somebody. In classic survey literature panels have had a long history. There are ways that people can track people's changes over time. So for instance there's a really famous study called the panel study of income dynamics, that has tracked people for 30 years to see how different types of income outcomes lead to different types of things like health and well-being. But panels are used increasingly in professional survey research. Because it can be really hard to recruit survey respondents through other means. As the years have progressed, it's been more and more hard to recruit people through traditional methods like phoning them in their homes. Or getting them to mail back paper questionnaires or even if you get them to respond to email questionnaires. So panels are a way of helping to facilitate, access to survey participants. Increasingly, this panels are constructed by companies, sometimes professional survey organizations who make them available to clients. So for instance, you ca go to a company and say I want to ask questions to one of your panels, and they can then send your survey instrument directly to this panel they’ve already recruited. A couple of examples, there's a very large survey company called GFK, they're based out of Germany, that has their KnowledgePanel. So if you're a survey researcher or interested in asking survey questions of really well constructed set of panelists, you can go to KnowledgePanel, and you can go to GFK. And you can say hey, could you ask your panel these sets of questions, then they will list you a price basically for access to their panel and you form that relationship. A company like GFK goes through a lot of efforts to recruit people. They make sure that their panels are representative of different populations. They do a lot of demographic research so they know a lot of the people who are participating in their panel. So for instance the panel from KnowledgePanels has 55,000 members in it. Those members are paid by JFK to participate in their surveys and they get a share of basically of the money for all the services they participate in. And JFK knows a lot about them that can lead to demographic knowledge about your participants that you might not otherwise have. Couple of other examples of companies that do this. Pinecone Research is a company that makes panels available and actually here's one where you could go as a particular person and sign up to take surveys. So if you're interested in joining their panel, you can join, they will ask you a series of demographic questions. Then they'll put you in a panel and send you a list of surveys that you can participate in. As a client of Pinecone, you can go in and again, just like with JFK, you can say, hey, I have this questionnaire. I want to ask this question of this many people, they'll quote you a price and you can move ahead. yougov.com or yougov.us, they are both domain names is another company that does this. I like this company because they also make some of their raw stats available. So you can go in to YouGov, for instance, and see how their panel response to some general question. You can also get some great insights into how people think what their panel looks like, some customer research insights. For instance, how people think about current mobile phone technologies or who has landline phone left. All those types of things are information that YouGov makes available to you and something that likely really interesting for you to take a look at. So panels have a lot of great pros and a few cons that you should be aware of. The biggest thing I think with panels to consider is that they reduce recruitment costs. Remember, the different types of non-probability sampling that we talked about. The lowest cost method is probably just to put up a convenient sample and ask people to participate. You still have to do some hustle on that one. You still have to request people to maybe push and keep asking for help if you're not getting the types of responses that you want. You can also from a panel characterize the sample, because the companies that arrange these panels do a lot of work to collect this demographic information. You can actually say how different this panel is from the population you're interested in, if you're doing your survey in the United States for instance. You can compare your panel that you get from YouGov or from GFK to data.gov, the data drive from the US census and see how different your population is. Do we have more women than you would normally get in a regular population? Do we have more white people? All these are different ways that you can basically start to identify what error you might be introducing from using that panel. The other really nice thing about these panels is that you can target demographics a priority before the fact. Because the companies have gone through all the work of saying, well, we're going to have this many people from this many income brackets from this many educational backgrounds, from this type of industry. You can set specific demographics. I only want to interview women who work in the service industry and the panels can deliver those types of populations to you. A couple of cons with panels that you should really think through as you're designing your survey, one is that panels of course remain non probability. Even though it's easier to characterize the error that you're introducing by using this non-probability sample, it's still there something to be consider. The other thing and probably the biggest con that I was worry about when I use panels is participants basically are professional survey takers, right? As they do more and more of these surveys they learn how to gain the system a little bit. A lot of the especially the more quality companies have ways of rotating people in and out to try to avoid this professionalization of participants. But it can lead to something you ought be aware of which is that people who become professionals, who become experts basically at taking surveys answer survey questions differently than people who haven't taken surveys before. So you just want to be very careful about your expectations for how exceptional that panel is when you're characterizing your results. I want to talk a little bit about a specific type of panel that's becoming more and more popular in survey research for a variety of reasons. And that panel is Amazon's Mechanical Turk. Now if you've never heard of Mechanical Turk, it's a crowd sourcing application that is made by Amazon, and it can go any it can assign what are called basically human intelligence task or HITS. As a client you can go in and say, I have the task and I want the crowd or people to respond to this task. If you're a worker, and they call themselves Turkers, you can go in and you can see a list of tasks and you can do them for small amounts of money. Why Mechanical Turk I think is more popular than some of the other panels that we recently talked about, is because it's so cheap to participate and easy to get involved. It's basically a cloud application for getting access to crowds of people who will answer you survey. There's a lot of, however, provisos with that when you're thinking about Turk. So Mechanical Turk has a list of all the human intelligence tasks that have the word survey embedded in them. And I know the font on this is small, if you look at these tasks, they basically set the amount of time. So the first one for instance is 18 minutes. And then they set how much they're willing to pay you, in this case $0.10. Right, so you can see here that's the range of money that we're often talking about. $0.10 for a very short survey up to a $2 you can see for a 60 minute survey in the second row. Now these low numbers allow you to recruit more people and get more value out of your dollar for participating in Mechanical Turk. And I think that's partially why so many people have been using it recently. When you drill into any one of these human intelligent tasks or HITS, you can see basically each one of these has a description of the task, in this case it's a consumer decision survey. And the person, the Turker can accept to do that task or skip it. As the person who assigns the task, you can look at all the work that somebody did and you can agree to pay for it or not. So people who are gaming the system, who throw in junk data or just answer yes to everything. There are ways to check for that and you can go in and refuse to pay them and throw out their data if it's not working well. So Mechanical Turk has really ethnically revolutionized survey research over the past few years, partially because now is so reasonably inexpensive to get access to large groups of people. A few things that you should be aware of when you're using Mechanical Turk, there's a growing ethical consensus that what you should do is pay a HIT rate, a rate for the task that matches reasonable wages in your area, right? So $0.10 for 15 minutes would probably not be considered an ethical amount of money. You're going to want to pay something where you're at least matching a minimum wage or a typical respondent incentive, divided by the total amount of time that you think your survey is going to take. For Mechanical Turk, the panel information is shakier than for some other services. So, the panel information on Mechanical Turk, people do fill in the information that they have, the demographics, but it's not verified in the same way that somebody like GFK would. So the panels are slightly less quality than you'd get with a GfK or YouGov, but they're easier to access and they're cheaper. However, in Mechanical Turk, some population are just not accessible. The majority of Turkers are from the United States and India. Who is a Turker? It's very dependent on a whole set of things, it's not as short as just low-income people. There are actually people who relatively high-income, but who are at under performing jobs where they have free time. There is a lot of people are stay at home parents, who work on Turk, do Turk and their free time. For a lot of people it's a hobby but does mean that it's not systematically looking at different populations especially international populations. So it's not going to get you access to some types of people. Turk is often used to multi stage surveys so this is where you can add a screener survey to do a light weight thing for very little money. But then you screen the population for certain experiences, and you can do a much richer experience. It's a great way of basically saving your money. So you do the screener survey which is very short and you don't have to pay much. And then the longer survey can get you access to a larger Turker population than you're paying to participate in a much deeper survey. And then I think increasingly one of the things I see with Mechanical Turk is that it's incredibly good at doing these experiments that are embedded in surveys. So a lot of AB testing or preference solicitation can happen through mechanical Turk. Where you just take two conditions, maybe two designs or two UIs and you have people react differently to those, and see what their preferences to those are. Mechanical Turk and other panels like it, like the GFK or YouGov or Pinecone, allow you to access people and reduce your overall costs of soliciting respondents for your surveys. Often, when you're doing a survey, one of the hardest things to do can actually be getting enough respondents to participate. It takes an incredible amount of work and an incredible amount of push that we're going to talk more about when we talk about non-response error. However, these panels provide you, for reasonable amounts of money, direct access to a good group of people. Another thing to remember with Mechanical Turk is that the Turkers themselves like we talked about with other panels are organizing and are not typical necessarily. They do a lot of these types of tasks, they become basically expert survey takers. Turkopticon is a browser plug-in that a lot of Mechanical Turkers use to basically track these tasks and the amount of money that is being paid for them. And it allows them to basically organize themselves around work that you assign them in Mechanical Turk. It's just something to know about, that these Turkers are also thinking about these surveys and other types of these cross-sourcing tasks in very systematic ways. So in summary, panels are a good way to get access to decent samples quickly. It can be hard to get samples and knowing how to do that is itself an art form. Another thing to consider is that Mechanical Turk is increasingly being used for surveys as a type of panel. There's been actually a lot of research over the last few years, comparing responses that we get from Mechanical Turk versus responses you get from other forms of doing surveys. Mechanical Turk is robust, it's not returning in just junk data. It actually is a reasonable way to solicit responses from people. However, researchers should always be careful and thinking trough the data that they get from these panels. The biggest issues, of course, are that these panels aren't necessarily representative of all types of human experiences, and the people who participate in them aren't going to represent everybody. Especially if you're interested in international responses the panels can be very limited, depending on which panel you choose. Or if you're interested in very diverse experiences the panels while they have lots of different people based on demographics and can often provide you with a large sample size. They're often not as diverse as we would like them to be. And finally, you should keep considering that the panels often have people who become expert in taking surveys and that shapes a little bit how they respond to surveys. But panels are great and a wonderful way for you to get access to responses fairly quickly.
