If you’re like most people, you probably only have a limited number of potential respondents for your survey. You may have a list of your club members, or a list of fellow students, or even a list of customers you’ve painstakenly collected over a period of months if not years. You’re all set to send them a survey, and figure you’re all set to go — after all, your survey software provider has made it a snap to upload your e-mail addresses and send out personalized invitations on your behalf. Right?

Stop right there! How do you know that e-mails sent by your survey software company aren’t going to end up in the bulk mail folder or spam folder of your recipients?

If there is one thing I’ve become painfully aware of as I’ve looked at the various survey software packages out there, it is that very few survey systems put much effort into how they manage the e-mail they are sending. And since these systems send out such a high volume of e-mail, many of them are blacklisted by internet service providers or blocked using standard spam filters.

Here are some things to ask about if you plan to use your survey software provider to send e-mail on your behalf or manage your e-mail list.

See if your survey software provider uses certified e-mail

There are several companies that offer something called e-mail certification. These companies work closely with ISPs and companies to guarantee that anyone using their service will automatically be “trusted” by the e-mail providers. Getting certified is somewhat of a complicated process, but in my experience using SenderScore Certified, it is worth it since just about every e-mail you send out makes it to its destination and doesn’t end up in the spam box.

There are four big companies that offer e-mail certification. These are Habeas, SenderScore Certified, Goodmail, and SuretyMail.

Ordinarily, you as a survey system client wouldn’t need to work directly with any of the companies I just named — optimally, it would be the survey system itself who bonds their e-mail. If your company is bonded separately, then it would behoove you to send your e-mail through your own e-mail system and not through your survey software provider.

Incidentally, the way these certified e-mail providers work is that the client pays an annual fee or a per-email fee for each message that is sent (per recipient). If any of the e-mails are reported as spam (in certain ways) the sender is charged a penalty. Back in the day when I was using Sender Score Certified, every e-mail that was reported as spam cost us $20.

Now, to be fair — survey software providers don’t have a lot of control over what e-mails their clients are uploaded and sending to so it might be very risky for them to participate in such a program. Plus these programs cost several thousand dollars a year to belong to. However, if you’re serious about your e-mail surveys, or if you make your living this way, you may want to investigate sending out your invitations through a separate e-mail service that does offer certified e-mail.

What relationships does the survey vendor have with the ISPs?

Even if a survey software provider doesn’t certify their mail, they can at least develop relationships with the major ISPs such as Google, Hotmail, Yahoo, etc. to help ensure that as many e-mails get through as possible. Getting “white listed” among the major e-mail systems isn’t a terribly complicated thing to do, and it doesn’t cost very much. It will also help to ensure that the vast majority of e-mails make it to their destination.

Is the survey software provider on any blacklists? And is there a way to tell how likely my invitations are to go through?

There are tools on the internet that make it possible for you to get a sense of whether or not your invitations sent through an survey software provider are likely to go through. These tools can also help you know whether or not your survey software provider is on any blacklists. 

An e-mail blacklist is a service that compiles lists of domains and IP addresses that regularly send out spam. These lists are used by companies and ISPs to filter mail before it even makes it to your recipient’s e-mail software. There are dozens of these lists, and once a company is on one it is very difficult to get off.

Here’s what I would do to check on the deliverability of e-mails from a survey software provider. I use a free tool called SenderScore.

1. Go to http://www.senderscore.org

2. Punch in the domain name of the survey system. (Try “zoomerang.com”). You’ll be shown some basic information about how the provider sends out e-mail, such as whether or not they even send out their own e-mail (if they have an MX record), whether or not they have an SPF record (good for deliverability), and whether or not they have a Sender ID record (also good for deliverability).

 You can see on the list above that Zoomerang sends out their e-mail using two different domains — “mailsender.zoomerang.com” and “mailsender2.zoomerang.com.’ You’ll also note that these are both clickable links — so try clicking them. The first link, you’ll notice, indicates absolute no volume of mail being sent, while the second link (“mailsender2.zoomerang.com”) has a very high volume of e-mail. Note the clickable IP address on the left side. Click it.

The page that appears (specifically, the SenderScore information for 63.251.58.111) contains a lot of information that you can use to decide whether or not this survey software provider can get your e-mails through.

Starting from the top: the Sender Score is a summary number that you can use as a measure of this IP address’s “reputation” on the Internet. This number can range from 0 to 100. Let’s just say that if you sent out good e-mail from your own server all of the time you shouldn’t have a problem keeping this number above 90. If you send out to a lot of big e-mail lists where people come and go a lot, you could be at around 60-70. If you send out a lot of e-mail from clients who upload their own lists (like a survey software company, a 40 isn’t surprising.

Some other survey companies and their ratings:

• InsightExpress.com: 98
• ConfirmIt: 80
• SurveyGizmo: 80 [not 25]
• SurveyMonkey: 75
• Qualtrics: 50

Working down the page, you’ll see that next to “blacklists” there is one listing for blackholes.five-ten-sg.com, a fairly aggressive blacklist that lists e-mail senders for many reasons (actually, if you do a search on google you’ll see that this particular blacklists lists so many people that most e-mail servers are not encouraged to takes the list seriously).

Next to “Sender Score Certified” it says no, indicating that Zoomerang does nog subscribe to this particular e-mail certification program.

Under deliverability it says that about 91% of the e-mail sent through this system gets through, and that e-mail sent through this system has a “medium” risk of deliverability issues. Medium isn’t bad and isn’t something you should be worried about. You should definitely be worried if the system you are using says “High.”

Reputiation measures cover a variety of facotrs that ultimatley contribute to the overall score. Higher numbers are always better.

What if I want to improve my deliverability?

Even if you don’t want to switch survey software providers in order to send out your e-mails, you can still get good deliverability by sending your e-mails through a separate service that focuses on sending out e-mails such as CheetahMail, Constant Contact, ExactTarget, Lyris and several others.

Lemaster contrasts full-service providers, such as M/A/R/C and SSI against complex and potentially expensive "self-serve" packages from Confirmit, Globalpark, GMI, and SurveyZ to low cost providers such as QuestionPro, Survey.com, SurveyMoney, and Zoomerang.

The abstract of the paper summarizes it as providing…

"…an introductory guide to managers who have been asked or have decided it is time to start looking into online market research for their company.  The size of the company you work for does not matter, as we will discuss several methods that can be used for any size company or investment level.  This paper is for managers who want to quickly learn the basic issues of online market research panels.  It will also be useful to managers who need to become familiar with some of the major providers of online panel management."

In addition to reviewing the providers and the various strategies for organizing your panel, Lemaster discusses a variety incentives that can be used to motivate and retain participants, such as lotteries, bonus points, and raffles.

• Online Panel Management and Strategies: An Introduction for Manager

Enter Wufoo.com, the Web 2.0 answer to online forms. Creating forms has really never been so easy or so much fun. Or so much fun to look at. Or so smooth. As someone else pointed out somewhere else, this service makes you wish you had a need for a form so that you could use Wufoo to build it. It really is that neat.

Can you make surveys using Wufoo? Not really. I mean, I suppose if you had a really simple survey that you wanted to execute, you probably could. It does let you do multiple choice questions, open-ended questions, and single-response questions. It even has special fields that help respondents enter prices, addresses, e-mail addresses and dates (all of the things one might regularly enter in a form). But there are also a lot of survey-related things that it doesn’t let you do, like create multiple pages, or validate responses in a meaningful way (other than to require the fields). I would hazard to say that as far as survey capabilities go, SurveyMonkey and Zoomerang have more features, as well as libraries of templates you can use to get you started. These are, after all, programs that were specifically designed to help you create surveys.

One thing that was a little annoying — I spent about five minutes designing a form and clicked the "Themes" button to make it look extra pretty — only to discover that my form had completely disappeared! I had assumed that my changes were being saved as I went and never even noticed the "Save Form" button at the bottom-right corner of the page. Make sure you push it when you are done or your simple survey will end up be simpler than you intended. And then you’ll have to start over from scratch.

With that said, I wish SurveyMonkey, Zoomerang, and even the package I use, GMI Net-MR, would take a page from the Wufoo design/survey development book and create an interface that is as fun and as easy to use. I would get a lot more work done.

Wufoo offers a variety of pricing plans, ranging from free accounts (the "gratis" plan) which lets you create 3 forms with up to 10 fields each and accept up to 100 entries a month. The next plan up (the "ad hoc" plan) costs only $9 a month and allows you to have up to 10 forms with an unlimited number of fields and accept up to 500 entries. There is also the "bona fide" account (unlimited forms, 3000 responses a month), the "carpe diem" plan (15,000 responses a month) and the "ad infinitum" plan (no limits, total cost is $199 a month).

And if you are trying to create an online form to collect data, and don’t want to mess around with the basic HTML building blocks, and if you want the chance to play with the newest and greatest of what’s out there, then give Wufoo a try.

• Wufoo.com

In terms of the branding, Richard was concerned because the free version of Zoomerang ends with a message inviting respondents to use the system to create their own services, and one could argue that this would be somewhat unprofessional-looking way for a "real" company to end an exchange with a customer (although to be fair, a company that at least goes to the trouble to collect feedback from customers — even free online software — is much better than all of those companies that don’t collect any feedback at all). And to be completely fair, the paid version of Zoomerang does allow you to remove the branding.

But Richard raises a good point — if you are going to go to the trouble of collecting invaluable data from your customers, you at least owe it to yourself to use a process where you control the look and feel of the experience from beginning to end (which, I should again point out, is possible using the paid version of Zoomerang. But even there you need to make sure you change the settings appropriately and test your survey to make sure it looks exactly as you want it to look).

Richard’s second point is that services such as Zoomerang store your data on their servers. This is, of course, true for all web-based survey systems and there is no getting around it unless you only use survey software installed on your own computers that are maintained in your own facilities. You can to some degree mitigate the risk by making sure you understand the level of security and regular backup procedures of the company that you choose to use for your survey research.

Zoomerang doesn’t say precisely how your data is stored other than to indicate that information on members and panelists are stored in "secure databases" protected by passwords, and that data from surveys is stored at "a secure hosting facility with both physical and software-based security systems." For most people, this probably sounds secure enough but if you want to know more and if security really is a concern you may want to contact the provider before you start your survey.

Data retention policies are also important to consider. For example, if you use the free version of Zoomerang they say that your data will only be saved for 10 days following the start of the survey. The impression I get is that they do retain it longer than that and it can be accessed again if you sign up for a paid account, but it is always important to understand the data retention policies before you get started as well as whether or not it will be easy for you to download your data in a managable format when you are finished. Richard expressed concern that Zoomerang only allows a flat file format — well, I think for most people that is probably enough. There are a few services that will allow you to download your survey into a relational file format (such as an Access data file), but these services tend to cost more and frankly I’m not sure what most researchers would ever do with the data in that format anyways. SPSS takes only flat files.

Finally Richard raises the question of cost and why there aren’t more open-source alternatives. He speaks the truth when he notes that survey software isn’t really that hard to write and to that I really don’t have that much of an answer. I imagine it will happen eventually.

• Read Richard’s article about Zoomerang at Kassblog

Visit the Zoomerang Education Research Center

Read the full press release at Yahoo Business