I’ll give you an example: We once conducted a survey about a concert series we host each spring to try to find out what kind of music our guests wanted to hear. When we go the results back, we were surprised to find that, more than anything else, our visitors wanted to listen to 80s music. By a big margin too.

Fine by me — there is a lot of 80s music that I like. But it seemed a little funny. So we ran a few cross-tabs, looked at some other variables, and found that a disproportionate number of our respondents were between 35-44 years old. Given that research has shown that people tend to be attached to the music they listened to as teenagers, is it so surprising that 80s music was so popular among a bunch of 35-44 year olds? After weighting the data to match attendance at the event, other forms of music, including pop, rock, and a couple of other genres jumped to the front.

Imagine if we had just sent our survey to a bunch of 15-18 year olds. Or if we had sent it to women over 65 years old. Or if we had just sent it to respondents in Mexico.

First rule of survey research: make sure your survey people who represent your customers.

Second role of survey research: make sure you survey them in the right proportion to your customer base.

I’m guessing the the reason for these two rules is fairly straightforward. If you don’t survey the right people in the right proportion, how will you know if your results reflect the behavior and attitudes of all of your customers (or the specific group of customers you are trying to understand)? If the people you survey aren’t representative of the people you’re making decisions about, there is very much the possibility that in conducting your survey you could be doing more harm than good. Its as if, in the example above, we got the results of our survey and then went about featuring 80s bands at all of our concerts. Nobody would come. We’d lose tons of money. And we’d assume that the research must have been flawed.

There are three important steps to having the right survey respondents take your survey. Step #1 is figuring out who the right survey respondents are. Step #2 is finding those people. Step #3 is actually convincing the right people to take your survey. The remainder of this post is about Step #1.

If you want to be technical about it, step #1 could also be called “setting your quotas.” It is the process of making sure in advance that you select people in the right proportions (that is, 50% male, 50% female; or 60% male and 40% female) to best represent the people you want to understand.

If You Want To Survey The World

Maybe you just want to learn more about people living in a particular city, or in a particular state, or even in a particular country.

This information is actually easier to come by than information about your customers, and it may even be possible that the information is available for free. The census bureau offers a great deal of information which can be broken down in a number of different ways, and can be narrowed down to very specific regions. You can find information about age, income, how many cars people in a particular region have, and lot more. Other countries offer similar information.

You’ll find that you have a lot of data to choose from. What you need to do at this point is determine which of it is relevant. For example, if you’ve created a new kind of dental implant, you may not want to spend much time collecting demographic information about people under the age of 40. Conversely, it may turn out that it doesn’t matter how old your respondent is (because you have a product that is equally useful to everyone of every age) but it is really important to survey only women. Make these decisions as you are pulling the data so that you can make sure your “quotas” are in the right proportions.

If You’re Trying To Survey YOUR Customers

If you don’t know the demographic and behavioral makeup of your response base, you may find yourself a little challenged with step #1. After all, if you’ve never surveyed your customers before, how are you going to know who you should survey?

When I used to work in the marketing department of a summer-stock theater, we used to go into the parking lot and write down the states on everyone’s license plates. Pretty crude. But if a substantial number of your customers pay cash and you don’t have a way to know where they are from, it can be a reasonably effective method for understanding where your customers come from.

If you’re an online business, or a business that has invoices that contain content information for your customers, then you can probably get some useful information right from your database. You can examine the breakdown by gender and figure out where they live. You can tell, for example, whether or not all of your customers come from a specific geographic area (indicating that you should only survey people from that area) and even how frequently they visit you (if you’re trying to find out about your repeat customers, it may not be appropriate to survey people who haven’t been visiting you for long).

Theme parks and other out-of-home entertainment destinations periodically conduct something called a “Point of Origin” survey. This is a survey, conducted of randomly selected guests as they enter the facility, is used to establish a baseline understanding of where people are coming from, how old they are, gender, ethnicity, and other information.  Because it is a random survey of customers, and theoretically reflects all of your customers, it is a valid way of understanding who your customers are — which means that you can use it to make such statements as “25% of my customers are between 25-35 years old” and then use it to make sure that approximately 25% of your respondents fall within that age range.

A Point of Origin survey doesn’t have to be long. And it isn’t hard to conduct. All you need to do is have an employee stand just inside the front door of your store and ask people to take a short, anonymous survey before they start shopping. Alternatively, you could conduct the survey at the cash register, which would give you the added capability of know who purchased and who didn’t.

Remember — keep your Point of Origin survey short so that it won’t create any hardship for your paying customers. Let them know why you’re doing the survey (you’re trying to learn just a little bit more about the people who are your customers) and perhaps offer an incentive for participating (chance to win a pair of stereo speakers, or something that you actually sell).

Once you’ve conducted your Point of Origin survey, you’ll have a good basis for determining who should participate (and in what proportion) in your other surveys.

What If There Is No Easy Way To Determine Your Quotas

Sometimes you want to survey someone other than you customers. Maybe you’re creating a business plan and don’t yet have any customers. Or maybe there is no easy to way to survey your customers. Or maybe there isn’t time to survey your customers.

When you’re in a situation like this, sometimes the best option is to make an educated guess. That is, watch the customers walking in the front door. If it looks like there are more women than men coming in, then maybe you survey 70% women and 30% men. If it looks like you have a lot of customers with kids, maybe you say that you’re going to survey 60% families with kids and 40% families without kids.

What is important, when you’re doing things this way, is that you make sure that you’re surveying enough people from each group that you want to be able to describe to make your results meaningful. For example, your customers may be mostly women with just a few men and you might decide to survey 95% women and only 5% men. While this will probably give you good overall results, you need to ask yourself up front whether or not you’re going to want to look at the results from the men separately from the women (perhaps you want to know why the men are coming into your store).

To effectively do this, you need to collect at least enough male responses so that you can analyze them properly (do you really want to launch a new men’s product line based on what 10 men had to say?).

In my experience, as a rule of thumb, any time I want to be able to summarize the results from a particular subgroup of respondents, I try to make sure that I have a minimum of 50 responses in that group (although, to be fair, I should say that my average sample size for most projects is about 1,000 responses so collecting 50 from a particular subgroup isn’t much of a challenge for me). If you’re only collecting a few hundred responses you may feel comfortable going as low as 25, but you probably don’t want to go lower than that (and even then you’re going to have to watch your margin of error).

The point of all this is that even if you guess, as long as you have enough of a sample from each of the segments (that is, as long as you have enough male and female respondents) you should be able to compensate if you later find that you’ve collected too many responses from one group or too few responses from another group.

Guessing isn’t optimal, but when it comes down to it, its better than not having any quotas at all, and 90% of the time it is going to get you into the right ball park.

Establish the Right Quotas or Get the Wrong Answers

Remarkably, none of this has anything to do with writing the questions, programming the survey, or analyzing the results. But it is extremely important. If you don’t survey the right group of people you are, 6 times out of 10, going to find that you don’t get the right answer — not because you didn’t ask the right questions, but because you didn’t ask the right people for the responses.

Let’s first acknowledge that when it comes to conducting any survey, the most important element — the most value adding component is the respondent (not the survey software). Most people only have access to a limited number of respondents AND chances are they are only going to get one opportunity to collect the information they need.

Which is why we as researchers tend to want to ask as many questions as we can of each respondent who we manage to convince to participate in our survey. Ask now, or you may never get another chance. Or you’ll have to pay for a whole new panel. Or you’ll have to squeeze, squeeze, squeeze the information out of the data you were able to collect the first time. And squeezed data is never pretty.

The trouble is, when you make the survey too long respondents drop out.  I remember a couple of years ago participating in a phone survey. The survey went on, and on, and on and every time we got to the end of topic, I thought it would be over. But it never ended. I eventually hung up.

So it is a balance. And capturing that balance can be challenging. I’ve personally come up with four key considerations that more than anything else dictate how long a survey can be.

1) What does the respondent get for taking the survey? The bigger the incentive, the longer the survey can be. But keep in mind, unless it is a guaranteed incentive (finish the survey and we’ll send you a T-Shirt) you’re only going to get so far. For example, a sweepstakes in which the respondent can win $10,000 will do better than a sweepstakes for $1,000 but not necessarily 10 times better and not nearly as well as a survey in which the respondent automatically receives $20.

2) How interesting in the survey subject to the respondent? I will get the same dropout rate for a long survey about new rollercoasters than I will for a relatively short survey about toothpaste. People like to take surveys about subjects that they find interesting and will stay in a survey a lot longer if it is one that lets them think or discuss matters that are of interest to them.

3) How much fun is it to complete the survey? Believe it or not, a survey doesn’t have to be a boring, unpleasant experience. Questions can be written in an interesting or witty way, and response options can be configured in such a way that that the survey is fun for the respondent to take. For example, it is much more fun to click on pictures of things than on words — and it is more fun to drag a list of items into a specific order than it is to rank items using drop down boxes. The more fun your survey is to take, the longer it can be.

4) What impact will completing the survey have on the respondent? If I’m conducing a survey in my neighborhood to decide whether or not the lot on the corner is turned into a new park or a junk yard, I can feel confident that I’ll get a good response rate from the people living on my street no matter how long the survey is. People will spend more time on a survey that they know impacts them in some way — much more than they’ll spend on a survey that they know only benefits you.

By thinking about how these four factors come together within the context of your survey, you can make a determination regarding how you want to present your survey and whether or not you need to offer a better incentive or — if all else fails — reconsider the length of your survey.

Three main

If you’re like most people, you probably only have a limited number of potential respondents for your survey. You may have a list of your club members, or a list of fellow students, or even a list of customers you’ve painstakenly collected over a period of months if not years. You’re all set to send them a survey, and figure you’re all set to go — after all, your survey software provider has made it a snap to upload your e-mail addresses and send out personalized invitations on your behalf. Right?

Stop right there! How do you know that e-mails sent by your survey software company aren’t going to end up in the bulk mail folder or spam folder of your recipients?

If there is one thing I’ve become painfully aware of as I’ve looked at the various survey software packages out there, it is that very few survey systems put much effort into how they manage the e-mail they are sending. And since these systems send out such a high volume of e-mail, many of them are blacklisted by internet service providers or blocked using standard spam filters.

Here are some things to ask about if you plan to use your survey software provider to send e-mail on your behalf or manage your e-mail list.

See if your survey software provider uses certified e-mail

There are several companies that offer something called e-mail certification. These companies work closely with ISPs and companies to guarantee that anyone using their service will automatically be “trusted” by the e-mail providers. Getting certified is somewhat of a complicated process, but in my experience using SenderScore Certified, it is worth it since just about every e-mail you send out makes it to its destination and doesn’t end up in the spam box.

There are four big companies that offer e-mail certification. These are Habeas, SenderScore Certified, Goodmail, and SuretyMail.

Ordinarily, you as a survey system client wouldn’t need to work directly with any of the companies I just named — optimally, it would be the survey system itself who bonds their e-mail. If your company is bonded separately, then it would behoove you to send your e-mail through your own e-mail system and not through your survey software provider.

Incidentally, the way these certified e-mail providers work is that the client pays an annual fee or a per-email fee for each message that is sent (per recipient). If any of the e-mails are reported as spam (in certain ways) the sender is charged a penalty. Back in the day when I was using Sender Score Certified, every e-mail that was reported as spam cost us $20.

Now, to be fair — survey software providers don’t have a lot of control over what e-mails their clients are uploaded and sending to so it might be very risky for them to participate in such a program. Plus these programs cost several thousand dollars a year to belong to. However, if you’re serious about your e-mail surveys, or if you make your living this way, you may want to investigate sending out your invitations through a separate e-mail service that does offer certified e-mail.

What relationships does the survey vendor have with the ISPs?

Even if a survey software provider doesn’t certify their mail, they can at least develop relationships with the major ISPs such as Google, Hotmail, Yahoo, etc. to help ensure that as many e-mails get through as possible. Getting “white listed” among the major e-mail systems isn’t a terribly complicated thing to do, and it doesn’t cost very much. It will also help to ensure that the vast majority of e-mails make it to their destination.

Is the survey software provider on any blacklists? And is there a way to tell how likely my invitations are to go through?

There are tools on the internet that make it possible for you to get a sense of whether or not your invitations sent through an survey software provider are likely to go through. These tools can also help you know whether or not your survey software provider is on any blacklists. 

An e-mail blacklist is a service that compiles lists of domains and IP addresses that regularly send out spam. These lists are used by companies and ISPs to filter mail before it even makes it to your recipient’s e-mail software. There are dozens of these lists, and once a company is on one it is very difficult to get off.

Here’s what I would do to check on the deliverability of e-mails from a survey software provider. I use a free tool called SenderScore.

1. Go to http://www.senderscore.org

2. Punch in the domain name of the survey system. (Try “zoomerang.com”). You’ll be shown some basic information about how the provider sends out e-mail, such as whether or not they even send out their own e-mail (if they have an MX record), whether or not they have an SPF record (good for deliverability), and whether or not they have a Sender ID record (also good for deliverability).

 You can see on the list above that Zoomerang sends out their e-mail using two different domains — “mailsender.zoomerang.com” and “mailsender2.zoomerang.com.’ You’ll also note that these are both clickable links — so try clicking them. The first link, you’ll notice, indicates absolute no volume of mail being sent, while the second link (“mailsender2.zoomerang.com”) has a very high volume of e-mail. Note the clickable IP address on the left side. Click it.

The page that appears (specifically, the SenderScore information for 63.251.58.111) contains a lot of information that you can use to decide whether or not this survey software provider can get your e-mails through.

Starting from the top: the Sender Score is a summary number that you can use as a measure of this IP address’s “reputation” on the Internet. This number can range from 0 to 100. Let’s just say that if you sent out good e-mail from your own server all of the time you shouldn’t have a problem keeping this number above 90. If you send out to a lot of big e-mail lists where people come and go a lot, you could be at around 60-70. If you send out a lot of e-mail from clients who upload their own lists (like a survey software company, a 40 isn’t surprising.

Some other survey companies and their ratings:

• InsightExpress.com: 98
• ConfirmIt: 80
• SurveyGizmo: 80 [not 25]
• SurveyMonkey: 75
• Qualtrics: 50

Working down the page, you’ll see that next to “blacklists” there is one listing for blackholes.five-ten-sg.com, a fairly aggressive blacklist that lists e-mail senders for many reasons (actually, if you do a search on google you’ll see that this particular blacklists lists so many people that most e-mail servers are not encouraged to takes the list seriously).

Next to “Sender Score Certified” it says no, indicating that Zoomerang does nog subscribe to this particular e-mail certification program.

Under deliverability it says that about 91% of the e-mail sent through this system gets through, and that e-mail sent through this system has a “medium” risk of deliverability issues. Medium isn’t bad and isn’t something you should be worried about. You should definitely be worried if the system you are using says “High.”

Reputiation measures cover a variety of facotrs that ultimatley contribute to the overall score. Higher numbers are always better.

What if I want to improve my deliverability?

Even if you don’t want to switch survey software providers in order to send out your e-mails, you can still get good deliverability by sending your e-mails through a separate service that focuses on sending out e-mails such as CheetahMail, Constant Contact, ExactTarget, Lyris and several others.

Personally, I think it is just as easy to eliminate the problem by using the validation options of whichever survey package I’m using to ensure that respondents get an error message if they try to enter anything other than a number in a numeric field (and shame on any survey packages that don’t offer such validation).

Validation aside, there is something valuable from a semantic/questionnaire design perspective in understanding what can be done to minimize the number of respondents who enter inappropriate values in open-ended question boxes.

For Fuch’s experiment, he tried putting a default value in the box so that respondents could see the desired format. While it didn’t have a substantial impact on the number of people who put in inappropriately formatted values,  it did substantially increased the number of people who put pure numbers into the box (that is, while they didn’t guess as much as they did before, they did at least put the numbers in the right format).

My only concern about putting default values such as those that Fuchs described into the text boxes is that it is very difficult to know what to do if the default value is left unchanged in the box. If the respondent leaves the “0″ does it mean they meant to convey a zero or they just never changed the value?

One solution to this challenge that some survey software packages are doing these days is using pre-filled values that disappear after the user clicks in the text box. It isn’t that hard to implement (just a quick bit of Javascript) but I have to imagine it significantly improves the likelihood that respondents will use appropriately formatted values.

• Asking for Numbers and Quantities: Visual Design Effects in Web Surveys

What they found was this: 22% of those who received just the promise of the $10 completed the survey while 59% of those who got the $5 coupon in the mail took the survey. Of course, that means that 41% of the people who received the gift card but didn’t take the survey got a free $5 prize.

I won’t keep you in suspense. It actually turns out that it is more efficient and less expensive to send out the $5 gift cards to the appropriate number of potential respondents in advance than it is to send prizes only to people who complete the survey.

Consider  the efficiency first: In order to get 500 responses with a 22% response rate, you need to send out 2,273 surveys — while with a 59% response rate you only need to send out 847 surveys. Assuming that your list is of limited size (and this is especially true of B2B surveys), you may want or need to maximize the return rate on your potential response base if you expect to meet your 500 response goal.

Now lets consider the cost. We know if we go the $10 per complete route we’re going to have to pay out 500 x $10 = $5000 in incentives. But how much are we going to have to pay out if we send every potential respondent a survey? As indicated above, it will take 847 invitees in order to get 500 responses rates so if we’re going to send all of them $5, it will cost of 847 x $5 = $4,235 — a savings of $765.

So not only were we more efficient with our list, but we saved a significant amount of money!

Of course, your results may vary, and you may want to do a few test groups before you go all out sending pre-survey incentives to all of your potential respondents. But if you have a limited response base, pre-survey responses may be the way to go.

Incidentally, Kaplan & White note that pre-paid survey incentives of $1 or $2 “is nearly as effective in generating response rates above 50 percent as a $5 incentive.” If this were true, it would mean that the total cost of collecting our 500 response sample above would drop from $4,235 to $850.

The Kaplan & White article is worth reading. In addition to what I’ve described below, they talk about the possible impact of pre-incentives on the data quality (i.e., whether or not there is any response bias) and provide a list of useful references.

• Incentives in a Business Survey: A Study in Improving Response Rates

Of course, it isn’t always possible to prove that this is what is going on. However, there are certain tell-tale signs:

• The same “review” appears on numerous web sites.
• The name of the reviewer isn’t published.
• The review appears on a blog that only features that review.
• The review appears on a blog that appears to have no history.
• Several survey software packages are reviewed, but only one has links.
• The review points to a landing page on the site it is reviewing, not the home page.
• The software gets a practically unblemished review.
• All of the other software reviewed on the site is compared to the one piece of software that received the practically unblemished review.
• Lots of unnecessary punctuation. Oh, they are so excited!!!!
• Lots of inexplicable links — people do this to boost themselves in the rankings.

The issue here isn’t that people shouldn’t do what they can to promote their software — but at the same time, it is absolutely wrong to create fake reviews and web sites that are “unbiased” for the purpose of manipulating people to visit your web site, and I would think twice before ever using such a company to conduct your important research.

1. You need to select some a survey system. This part is fairly easy to come by as there are all sorts of options out there to choose from. If you don’t expect to do a lot of surveying then you are probably OK to use a fairly low cost option. If you’re doing it for a real life professional business I would be careful not to do it on a free system that doesn’t let you remove their branding (there is nothing more tacky then having your survey end on the surveymonkey signup page – your survey should end on your home page or something similar). If you’re going to show images, make sure that you choose a system that makes it easy for you to upload and insert images – don’t choose a system that makes you upload the images to your own web server since that will make things too complicated (especially if you don’t have a web site).

2. You need respondents. This is often the most tough part of conducting a survey online – you need to find people, preferably customers, to take your survey. If you have a web site you can definitely solicit your web site visitors to take your survey, keeping in mind that you don’t want to do anything that will stop them from making purchases. One option is to invite past customers to take your survey – hopefully you have plenty of e-mail address from past customers, and a carefully worded, friendly e-mail to them may get enough of them to take your survey – especially if you make it clear to them that you’ll be using their feedback in the development of your new products. If you don’t have any e-mail addresses and you want to get your feedback from a group of non-customers, then what you may want to do is rent an outside survey panel. You’ll pay upwards of $5 or more per response, but if the feedback is important enough, it may be worth it to you.

3. You need to offer an incentive. There are probably many people who will take your survey for free. Maybe they like you, maybe they like your product, or maybe they just like to take surveys. In most cases, there won’t be a enough people like this to give you a fleshed out sample. By introducing a simple incentive – the chance to win a drawing or even some kind of an exclusive discount – you can greatly increase the number of people who will take your survey without significantly increasing your costs. For example, one of my favorite incentives is to offer a chance to win a $200 gift certificate to giftcertificates.com. You’ll double or even triple your response rate which means that the quality of your responses will be significantly higher without spending significantly more money overall.

Lemaster contrasts full-service providers, such as M/A/R/C and SSI against complex and potentially expensive "self-serve" packages from Confirmit, Globalpark, GMI, and SurveyZ to low cost providers such as QuestionPro, Survey.com, SurveyMoney, and Zoomerang.

The abstract of the paper summarizes it as providing…

"…an introductory guide to managers who have been asked or have decided it is time to start looking into online market research for their company.  The size of the company you work for does not matter, as we will discuss several methods that can be used for any size company or investment level.  This paper is for managers who want to quickly learn the basic issues of online market research panels.  It will also be useful to managers who need to become familiar with some of the major providers of online panel management."

In addition to reviewing the providers and the various strategies for organizing your panel, Lemaster discusses a variety incentives that can be used to motivate and retain participants, such as lotteries, bonus points, and raffles.

• Online Panel Management and Strategies: An Introduction for Manager

The paper is entitled "An Optimal Contact Model for Maximizing Online Panel Response Rates," was written by Scott A. Nelsin, Thomas P. Novak, Kenneth R. Baker, and Donna L. Hoffman. Here are a few of the points I found interesting in the background section of the paper:

• Response rates from online surveys tend to be lower than from traditional survey methods. Some of the theories that have been put forth to explain this include respondent privacy concerns, technical problems, respondent confusion, and poor design.
• Standard methods used to increase response rates in the real world may not work online.
• Nearly 80% of consumer goods and 74% of B2B companies use online panels.
• Newly formed online panels often experience high response, although these levels drop quickly without proper management, which includes attractive incentives, pruning of non-responders, recruitment of new panelists, personalization of messages, and creating a sense of "community" among panelists.
• Online panels are very inexpensive: telephone surveys cost anywhere from $15 to $20 per respondent; mall intercepts cost around $10 per complete, while online surveys tend to cost $1 to $2 per response (for the panel owner).

The "meat" of the paper discusses a predictive model for categorizing respondents into different classes based on their expected response rate and then using those classifications develop an optimization model for determining how many invitations to send to each group in order to maximize a broad-based response.

The abstract of the paper:

We develop and test an optimization model for maximizing response rates for online marketing research survey panels.  The model consists of:  (1) a decision tree predictive model that classifies panelists into “states” and forecasts the response rate for panelists in each state, and (2) a linear program that derives a plan specifying how many panelists should be solicited from each state in order to maximize response rates.  The linear program is forward looking in that it optimizes over a finite horizon during which S studies are to be fielded.  It takes into account the desired number of responses for each study, the likely migration pattern of panelists between states as they are invited and respond or don’t respond, as well as demographic requirements.  The model is implemented using a rolling horizon, whereby the optimal solution for S successive studies is derived and implemented for the first study; then, as results are observed, an optimal solution is derived for the next S studies, and the solution is implemented for the first of these studies, etc.  The procedure is field tested and shown to increase response rates significantly, compared to random selection and the heuristic currently being used by panel management.  Implications are discussed for further model development, implementation issues for online panel managers, and for the broader area of optimal contact models in customer relationship management. 

Why you wouldn’t just send all of your invitations to the most responsive group? Well, if you send out surveys infrequently and if you feel confident that you high responders are representative of your entire market then there is not reason not to do that. However, if you do send out a lot of surveys and you want to capture the broadest sample, then you need to find a way to maximize your entire panel without relying too heavily on a few dedicated panelists. That is where this model can come in handly (if you can figure out how to implement it, of course).

• Optimal Contact Model for Maximizing Online Panel Response Rates" at UCR eLab and Sloan Center

The presentation is available at the BayaSoft web site.