Update: It turns out that SurveyGizmo already offers email validation lookup. This means that when you do a survey on their system and validate it using their software, it will check to make sure that the server exists! See their blog posting for details.

Zip Code Validation

There are about 42,000 zip codes in the United States. In Canada there are over 800,000 unique postal codes, although for most purposes (at least, for most survey purposes I’ve ever heard of) the first three characters (the “FSA”) is more than enough and there are only about 1,500 of those. It wouldn’t be hard for a survey software to offer a feature that validates postal codes as they were entered. Which is to say that with a database of less than 45,000 records it would be fairly easy for to automatically validate zip codes as they were entered.

Any why just validate? I can only imagine that a lot of clients would appreciate the option of knowing the city, state, census region, and even latitude and longitude of the people who take their survey based on the zip codes that they enter. All this information is inexpensively available at places like zipcodedownload.com and can be easily integrated into online applications.

E-Mail Address Validation

Real-time e-mail address validation is even easier because it doesn’t require a database. While a lot of survey systems (but again, not all) offer syntax validation, what is much more useful is real-time e-mail address validation where the system uses a component to contact the the respondent’s e-mail server to verify (a) that there is a server there (dns lookup); (b) that it is a mail server (mx record lookup); (c) that it can accept e-mail (smtp server validation); and (d) that the user has an e-mail address on that system.

All of these things are very easy and very fast to do (the deeper you get into it, the longer it can take — for example, verifying that an account exists on a server can take 30 seconds if the system even gives a reply — which is why I usually only go to the level of testing whether there is an active smtp server). Plus it almost guarantees that the e-mail address that ends up in the data base is a valid, working e-mail address.

For example, whenever someone starts one of my surveys I have them first enter their e-mail address — this is so that I can later award them a prize in whatever sweepstakes I have going on at the time. Since it would be troublesome for everyone if the e-mail address they give us doesn’t work, before I start the survey I first validate their e-mail address using a component called aspNetMX. If the e-mail address is valid (or at least links to a valid e-mail server) then I start the survey. If not, I have them re-enter their address until they get it right. You’d be amazed how many people will mistype an e-mail address over and over again (I keep a log of the errors so I can make sure it is working properly, and the errors usually are really respondent errors).

Once again, it would be a great feature to the client (and something you could promote as a feature of your software) if you offered real validation whenever possible which helps the client have better, cleaner data.

One example of the new technology is the ability of Confirmit users to have their survey automatically rendered “outside the HTML/browser paradigm.” For example, Confirmit Flex makes it easy to program your survey for regular web users and then have that survey also appear on iPhones, or in Flash, or as a Widget on a blog.

The software makes it possible for Confirmit to launch new elements of it’s system quickly, without disrupting the regular “main” Confirmit codebase which is so important to so many research agencies. As they describe it:

Effectively, Confirmit Flex has allowed us to create a system that at its heart is a super-tanker (Confirmit) but which can be improved by speed boats (Flex components) enabling us, our partners, and clients to respond rapidly to market changes and take advantageof emerging technologies.”

Read more about Confirmit Flex (including more technical details)

Confirmit Flex Press Release

Before sending me the note telling me just how wrong I am, record the following:

As I figure it, there are four different kinds of security as it applies to online surveys. The first kind of security is what we’ll refer to here as back-end security. This basically means that strangers can’t simply log into the survey software server and start making changes to your survey, or download your data, or do all sorts of other things which really ought to be protected by a good password. Most survey systems do a pretty good job with this, assigning separate accounts, usernames, and passwords to each survey respondent. Some even take it to the next level, by allowing account administrators to set different permission levels for each user on the account, thereby ensuring that people who are meant to have access to your survey can’t “accidentally” go and make changes.

The second kind of survey security is where only authorized people are allowed to take your survey. Some implement this feature by giving you one respondent password that you can share will all of your respondents, or some systems even allow you to give every respondent their own password (or access key) which not only keeps out the riff raff, but also makes sure that each respondent can take the survey only once. This type of security not only ensure that only the “right” people take your survey, but also that the wrong people can’t get access to the content of the survey.

I’ve seen this kind of survey software security implemented well, but I’ve also seen it implemented poorly. Some systems, instead of assigning each survey an ID made up of a random collection of letters and numbers, use a sequential, easy to guess series. Which means that it really isn’t all that hard to view (and possibly edit!) surveys written by other clients of the survey system. So watch out for that.

The third kind of security about the connection between the respondent’s computer and the survey software server. Is it secure (look for the https: in the URL). Frankly, unless you’re doing super secret stuff and your concerned that hackers and government agents are trying to listen in on your respondents, I’m not sure it is something you need to worry too much about. But if you are afraid of industrial espionage or that someone is going to tap into your wireless signal, then you may want to take the precaution of choosing a system that allows for encrypted connections.

It’s easy enough for a well thought out survey software program to provide good security surrounding these first three measures. Where they get into trouble — and where you get into trouble if you believe them — is when they start to protect your survey content.

The fourth kind of survey software security is content protection, which basically means keeping your untrustworthy respondents from copying the top secret images and product descriptions that you include in your survey onto public web sites or your competitor’s email account.

I know, I know…there are survey systems that feature technologies that make it harder to copy images or capture video. This can be done using javascript, or by doing some crazy encryption to the feed. One could even go so far as taking over the computer so that nothing else works except the survey.

But I have a camera. And I have a video camera. And if you don’t completely lock down my computer, I have some really good screen capture software that lets me capture both audio, images and pictures. If I want to capture your survey content, believe you me, I will capture your survey content.

And then I can do pretty much anything I want with it, can’t I? No doubt 600-1000 people took that confidential survey of yours, and how are you going to know which one of them posted it to the anonymous message posting board using an anonymous IP address?

I suppose it is theoreticaly possible to embed a visual identifier into each image (a watermark of sorts) so you can trace the image back to whoever posted it online, but I’ve never heard of technologies built into survey software (if it does exist please let me know — I’ll report it here!).

There are also ways you can reduce the risk. You can, for example, use a pre-screened panel made up of people you trust. Employees, for example. Another approach is to threaten your respondents with legal action (although most will figure out pretty quickly that you have no way of identifying them). You could also threaten to stop doing online surveys if they leak your secret.

But the best way to ensure that your super secret new business idea doesn’t leak out onto the Internet is not to do online research (there, I said it). If your entire business model is based around keeping something confidential, do not put it in an online environment in front of strangers. Period. And don’t trust any research firm that says they have a foolproof method of keeping your images, video and other content safe. If they do, just pull out your camera or video camera and press record.

Qualtrics: (internet survey software) has a fantastic variety of question types with a really easy to use survey development system. Also features advanced survey branching capabilities including looping, random blocks of questions and more.

SurveyGizmo: (internet survey software) Also extremely easy to use survey package with a lot of features. Probably the most “Web 2.0″ survey system out their. Friendly, irreverant, with plenty of features for both the advanced user who wants to control every element of the survey experience and the analyst who just wants to get the project done. Has good e-mail invite capabilities, basic analysis tools. Plus, the free account has more features than any other “free” accounts I’ve seen.

SurveyZ: (internet survey software) This is the “light” version of Qualtrics, great for academic surveys. The survey development system isn’t anywhere near as pretty as the Qualtrics engine, but it offers a lot of advanced capabilities and question types. If you’re not doing a ton of surveys and you don’t want to spend a crazy amount of money, take a look at this package.

Digivey Survey Suite: (kiosk survey software) If you’re looking to do a survey in your store on a computer or kiosk, this software is absolutely for you. It is super easy to program, allows you to create really great looking surveys, and is fairly indestructable in the field (that is, respondents have a hard time breaking it). Great company too willing to work with you to make sure you have what you need. I’ve been a customer for quite a while now.

Techneos Entryware Software: (handheld survey software) If you’re trying to do surveys on handhelds, then I absolutely have to recommend Techneos Survey Software. I’ll admit I don’t love their pricing model, but I’m stuck because no handheld based survey platform offers as many features and is as easy to use — and as customizable — as the Techneos Entryware platform. I’ve been a customer of this group for over 5 years.

Detailed reviews of all these packages can be found on this site, including everything I liked and didn’t like about each of them. Many more reviews are coming shortly. Thanks for reading!

Questionnaire Features

Qualtrics offers a lot of different questions types. More, perhaps, than I remember seeing in any other program. A lot of these are variations on old favorites (there are 21 different multiple-choice question types, 13 matrix type questions, 11 constant-sum type questions, 12 rank order type questions and 5 different text entry questions — and that doesn’t even include the “Pick, Group and Rank” questions, the “Side by Side” questions or the “Drill Down” questions. Without listing all of them in all of their variations, I think it is safe to say that Qualtrics has all of the standard question types covered.

[Please note: this review covers Qualtrics' new survey development engine, which is still in "preview" mode. There may or may not be more features to come that aren't described here]

Question types are extensive. In terms of special question types, Qualtrics has a variety to choose from. There are a variety of easy to build matrix questions/table questions that allow you to easily create bi-polar questions, Likert scales, text entry matrices, and even scales with different descriptions above every box in the matrix. There are constant sum questions that instead of asking you to enter numbers, it instead has you draw bars of different lengths. It is very well done, and goes a long way to ensure that your respondents have a good survey taking experience. Another great example of a well designed question type is the ranking question, which lets you drag the items in list into whatever order you want (much more fun than entering numbers or selecting values from drop down lists).

Qualtrics offers over 70 differeny question types

Qualtrics isn’t the first to offer a sliding scale question type, but they are the first I’ve seen to let you control little gauges with your slider. Or a stop light. Or a smiley face (you can control several different variations of smiles that work their way into a frown).

Qualtrics has a wonderful “Pick, Group and Rank” question type that lets you present the user with a number of items in the column on the left and then lets them drag each item into a number of different bins you create (you can create as many options or as many bins as you like). One variation of this question type even lets the respondent define each bin themself.

These types of interactive question types are wonderful not just because they make your survey more fun to take, but because they let you be much more creative and efficient in how you collect your data. I can’t tell you how many times I’ve struggled to figure out how to mimic a methodology that works fine in real life but is difficult to execute using just drop down boxes or radio buttons. In my opinion, the more question types you have available, the better.

The Qualtrics survey system has a few more question types that are worth noting — the “drill-down” question type does exactly what it says — your survey might ask for a country, and after they do the dropdown right below shows them cities within just that country to choose from. “Gap Analysis” questions let respondents answer a question and then automatically asks them to explain their answer if they rate it positively or negative (depending on which you choose). Qualtrics also has a “timer” question type that will keep track of how long they spend on a page, and a “Meta” question type that will automatically insert all of the respondent’s browser related information in the response data (browser type, browser version, operating system, screen resolution, etc).

Qualtrics offers a “Conjoint” type question which asks you to first specify all of your attributes and then the levels for each. It is fairly easy to configure, although inserting it in the survey is a little awkward (it doesn’t appear into your list of questions — you have to insert it using the “Survey Flow” page). It appears to use a form of adaptive conjoint methodology (which includes a little max-diff) which can be great for evaluating product features but not so accurate for pricing research (in adaptive conjoint pricing tends to be understated in importance compared to choice-based conjoint). You should also be “warned” that the conjoint question exports only the raw data and does not calculate part-worth utilities for you — which means that you will need to have some other methods of doing the analysis (if you don’t have such a method then the conjoint question will be fairly useless to you). My impression is also that there isn’t enough documentation for the conjoint question to make it really useful to many researchers.

Example of Qualtrics’ Conjoint Question

Qualtrics also offers a “Conjoint” question which actually seems pretty robust. Instructions for using it weren’t available and I would have to play with it a bit more to understand its methodology (if you’re looking for conjoint capabilities, you might want to play with it and ask questions).

Although Qualtrics makes it is exceptionally easy to create a basic survey without delving into any of the advanced options, there are a lot of advanced options for every question type. Question text and responses can all handle embedded HTML code, which offers a lot of flexibility. In multiple choice questions, for example, you can edit and easily rearrange answer choices. You can randomize some of the choices or all of the choices, and you can insert choices that respondents selected in other questions. You can even assign “recode” values to all of your questions so you can show the respondent a text value but actually export a number you assign to your data file. This is exceptionally useful if you’re trying to match your values to another table. You can even give each of the answer choices a name, and every question can have a unique ID that you assign.

A variety of different validation types are available in Qualtrics. Multiple choice questions, for example can require that a certain number of responses are chosen or that no more than a certain number are chosen. Text fields can be validated to ensure that the contents are a valid e-mail address, or a valid postal code, or even a valid state code or date. One (minor) limitation here is that you can’t validate against regular expressions — but frankly, I’m not sure how many people would use that feature if it were available.

Qualtrics features [tag]skipping[/tag] and [tag]branching[/tag] which appears to be both friendly and functional. For example, if I click the button to add skip logic, it automatically inserts not just a “Skip Logic” item, but also a page break (skip questions in almost all software packages only work when next to a page break, and most make you insert them manually). Then, using drop down boxes, it has me describe what I want to do: If Blue is Not Selected then skip to What Color Are My Shoes? [Done] … where the bold represents drop down boxes that tie to questions in my survey. Very easy, doesn’t require a lot of thought.

Qualtrics offers more advanced survey flow options, such as [tag]advanced branching[/tag], [tag]looping[/tag] and [tag]section randomization[/tag] is accomplished through the use of the “Survey Flow” page, which while functional struck me as somewhat less intuitively designed as the rest of the survey editor. It allows you to set up any number of question blocks which can be individually randomized. It also allows you to insert advanced survey elements such as the conjoint question mentioned earlier.

The ability to create multiple criteria for a branching decision (for example, if you only want to branch if their favorite color is red AND they like to eat pizza) is available, but it is not nearly as easy to implement as the basic branching.

Other advanced survey elements in Qualtrics include the capability to set up survey [tag]quotas[/tag] based on individual responses. That is, you can have certain questions displayed only if certain other questions have been answered a certain way a specified number of times; or, alternately, you can display certain groups of questions only if a specified question has not been answered an indicated number of times.  This is a neat feature — my only concern here is that the criteria definition capabilities are limited — you can’t, for example, combine multiple conditions.

Qualtrics doesn’t offer any scripting or advanced programming capabilities. While most potential users probably won’t consider this an issue, it will definitely be a concern to high end advanced users who want absolute control over survey flow (for example, sometimes I like to bounce back and forth through a survey). Nor does Qualtrics offer an API that lets you work directly with the flow and the survey data mid survey. But these are fairly advanced features, and most people won’t know that they’re missing (if you’re the type of person who would notice that they are missing then you probably want to look at something more advanced).

Perhaps the best way of summing up the questionnaire related capabilities and features of Qualtrics is to say that it is a great survey package for someone who wants to write great looking surveys that are fun/easy to take where you have a lot of options in your questionnaire development. Or it is good for someone who doesn’t care about advanced question types but who wants good looking surveys that are easy to write.

Qualtrics Survey Development Features

Qualtrics offers a very friendly environment for basic survey development. In addition to a very clean interface, it uses a lot of [tag]AJAX[/tag] in its interface, which makes the experience of developing your survey very pleasant and visual. The annoyances I experienced were mostly related to the integration of the advanced features — if you’re just doing basic survey development you should be very pleased with the experienced offered by Qualtrics.

I should point out that this survey development environment is new. The old Qualtrics survey development environment — especially the “Survey Wizard” was very slow and very frustrating. This new version is completely different and much better.

Creating and editing questions is absolutely painless in Qualtrics. There are very few postbacks to slow you down. When you press the “Create a New Question” button the new question pops onto the screen without having to refresh the page. Adding choices to multiple-choice questions or editing the basic question options for each question type have an instant impact on the question as it appears in your preview mode. This is especially helpful for more complex question types such as matrixes which most programs force you to guess at what the finished product is going to look like.

The Qualtrics development environment

Setting the question type in Qualtrics is also fun because as you hover over each question type it shows you samples of what the question will look like which are pretty close to the same thing. This will make your survey development process go much faster.

Copying and moving questions within the survey is extremely easy and the software offers several different ways to do it. Deleting questions is easy too, and Qualtrics gets extra points for saving questions you delete in the “Trash/Unused Questions” bin which makes it very easy to restore a question you thought you didn’t need any more back to the survey.

Qualtrics does not allow you to import questions from Microsoft Word or a text editor which, depending on your personal survey development process, could be somewhat frustrating.

Qualtrics offers a very well executed [tag]library[/tag] that includes dozens of survey templates, question templates, graphics, and messages. You can use questions and templates created by Qualtrics or you can create your own. Adding questions to a survey that come from the library is very easy as well and is integrated into the new question process.

As I mentioned earlier, Qualtrics offers you advanced options for most question types. These appear in an embedded pop-up window so it doesn’t disturb your survey development environment (although it makes it pretty much impossible to see your survey while you’re looking at the advanced options — I don’t know how frustrating this would be in real life).

I got a little bit frustrated with how some of the advanced survey flow features were integrated into the Qualtrics questionnaire editor. While all of the basics are very well integrated into the survey design (including basic skipping and branching) items such as question randomization, survey blocks, and looping are handled on the “Survey Flow” page and were much more confusing to work with than the main survey features. For example, when I added a conjoint question set to my survey I got no indication of its existence on the main question development page. And while Qualtrics gives you the ability to add survey triggers that will send you an e-mail message if respondents choose a particular option in the survey, there is no indication of where and when these items occur within the survey itself.

In term of the look and feel of the surveys, Qualtrics doesn’t provide a lot of direct customization options. When creating a survey you can choose from a limited library of design schemes or you can use a custom scheme that the company develops for you. Depending on the package that you buy, a certain number of these custom design templates will come with the product, so unless you like to change your look a lot you will probably be fine with what they set up for you. Qualtrics does not let you add your own CSS files or page templates. Once again, these are only going to be issues for you if you are an advanced user who, perhaps, have a number of different clients who require their own personal look and feel.

Qualtrics lets you change the text of your “Next” and “Previous” buttons, add or remove a progress bar, and add survey-wide header and footer text to your survey. This is the extent of your control over the overall look and feel of the survey.

On especially wonderful option that I really appreciate is the ability to be able to re-edit a survey once it is published. For example, if my question is “What is your favorite color” and after publishing the survey I realize that I forgot to include “Yellow” I can easily go and add that choice. Or if I misspelled “green” I can go back and edit the existing choices. Many survey software solutions don’t let you edit the choice lists or the questions after a survey i fielded because (a) it could mess up how you interpret the results (if, for example, you changed “green” to “red” after collecting 300 responses) and (b) it messes with the way that they store data. It is a dangerous capability — but come on — we’re not children. If I want to mess up my survey, let me make that decision! I very much applaud Qualtrics for making it so easy for me to go back and make changes once a survey is in the field.

There is one thing about the Qualtrics interface that is somewhat frustrating: the “Back” button on your browser doesn’t work. Not only that, but if you push it, you get a “Webpage has expired” error message and the only solution is the edit the address bar (pressing the back button or the forward button once you get this error message just gives you more error messages).

Overall, the Qualtrics survey development environment is among the best, most friendly, most easy to use that I’ve seen and is perfect for the beginner and intermediate user. Advanced users — that is, advanced survey programmers who like to tweak anything and everything — may feel somewhat stifled. Those who like to regularly edit the overall look and feel of their surveys may not be happy with the inability to manipulate the basic page code or the cascading style sheets that control the look/feel/design of the page. Also, you can’t directly important surveys from Microsoft Word or a text editor — each question has to be built manually. But generally speaking, the survey development process was a pleasant one.

Qualtrics Data Collection Features

Qualtrics supports a variety of different ways to promote your survey including e-mail blasts, survey links that you post on your web site, popup surveys, embedded surveys and something called a “web site feedback link.” All of these are very easy to implement and provide you with several options for doing so.

I am personally always a little leery of using e-mail invitation systems built into survey systems. First there is the issue of allowing them to manage your list and ensuring that they properly follow the e-mail related laws. Then there is the question of whether or not they are doing everything in their power to ensure that your survey invitations don’t end up in the spam folder or get stopped by the ISP before they even make it to the spam folder. Which is why I generally tend to use either my own e-mail server (where I can control my own deliverability) or use a professional e-mail list company to make sure all of my messages get through.

So while the Qualtrics e-mail setup function is fairly straightforward, I had mixed results when it came to deliverability. One test message I sent went through just fine, the other got stopped by Outlook’s spam filters. Your experience will vary and there is nothing to indicate that Qualtrics’ e-mail server is any less reliable than other e-mail systems connected to other survey software programs. I can verify from looking at the header information that other than SPF header verification, Qualtric’s isn’t using any special deliverability services such as Sender Score or Habeas. Meanwhile, Qualtrics isn’t on any blacklists and according to SenderScore has a 90% e-mail acceptance rate with only a medium risk of non-deliverability which is better than SurveyMonkey(88% deliverability, high risk of non-deliverability) and much better than SurveyGizmo(33% deliverability, high risk of non-deliverability).

Qualtrics E-mail Deliverability Statistics

You can’t just upload an e-mail list and send to it — you need to first important your e-mail list to a Qualtrics panel (or you can type in the e-mail addresses one at a time, but I wouldn’t recommend it!). My demo account didn’t give me access to the panel management features of the system, so I couldn’t upload an e-mail list to test it out. I can say that it does at least have panel management capabilities and sample management capabilities and if/when you demo the software you will want to get more details about how these work.

Incidentally, every individual you send your e-mail to gets their own unique link which makes it fairly easy to track who has taken the survey and who has not. It is also possible to set the system to send out reminder e-mail messages.

Qualtrics also supports popup email invitations, and it is worth noting that the Qualtrics style of javascript popup is very good and is unlikely to be stopped by any popup software that your users may be using. I didn’t give it a full test, but in my experience this particular style of javascript popup is very hard to stop. And you can control how frequently it appears (does it show up for every user or for every 10th user?), whether or not it is animated and how big it is.

Qualtrics also lets you embed surveys in a web page on your site through the use of an IFrame (which is a little like embedding a mini web browser in the middle of a page). This type of execution can be prone to user issues because actions to the page in which the survey is embedded in can have a negative impact on the Iframe and the contents of the Iframe. However, whether or not this style of embedding is to some degree up to the taste of the web site developer.

Qualtrics gives you some control over the survey experience. You can, for example, control whether or not the respondent gets a “Previous Page” button, and you can allow respondents to save their surveys and return later. You can even control how long the partial survey results are saved. Surveys can be protected with a password (so only certain respondents can take it) or open to the public. Surveys can be set to automatically open at a certain time and then automatically close at another time. Responses can even be “bucketed” into multiple response sets. At the end of the survey, a default message can be displayed or you can redirect the users to another web site, or display a message page you store in your library.

Incidentally, Qualtrics also has the capability to do one question polls that you can embed on your web site. Exploring the details of this feature goes beyond this already-too-long review, but suffice it to say that the feature exists.

Qualtrics offers all of the basic data collection features one would expect from an online survey software package. It offers several different ways to promote your survey, and all of them seem to be executed well. I have some concerns that the e-mail invitation function does not appeared to be structured to ensure deliverability, but at the same time it is no worse than most other online survey software packages and better than many. For most users, Qualtrics is as good as anything they are likely to find in other packages.

Qualtrics Reporting/Data Management Capabilities

Qualtrics has put some time into the reporting capabilities of the software package, and overall it appears to be as friendly as the survey development tools. Qualtrics data management and export features are fairly basic, but will probably get the job done for most users.

Qualtrics makes it fairly easy to review the individual records in your dataset. The software makes it easy to see how long each respondent took to answer the survey, which is a quick way of telling if some respondents are clicking through and not reading the questions. Respondents you don’t like can be easily deleted. Qualtrics gets extra points for making it fairly easy to importsurvey results into the system (so that you can presumably use the Qualtrics reporting engine to create reports from data collected outside of Qualtrics, or so that you can merge outside data with the data you’ve collected in Qualtrics).

The Qualtrics reporting system

You can look at the responses collected from individuals, although I have to say it is executed in kind of a funny way — when I pull up a report on respondent #R_6h4QhkGskGxeU4Y, for example, it shows me each of their responses in the form of multiple graphs (one per question) which looks OK, but seems a little wasteful (what is the point of seeing a graph of one person’s responses?). Nothing wrong with the way that this is implimented — it’s just kind of odd.

Data can be exported from Qualtrics in a variety of different formats including CSV, XML, HTML, and SPSS. Warning about SPSS data export: it doesn’t create .SAV files — instead, it exports a CSV file and a syntax file which creates all of the variable labels and the value labels. While this isn’t necessarily a bad way of doing things it does add a couple of extra steps and there are, frankly, lower level packages out there that actually export .SAV files. Given Qualtrics’ focus on the academic community, it is a little surprising that they haven’t taken the extra step to allow for true SPSS export.

I would describe the reporting capabilities of Qualtrics as “friendly, mid-level, but not super advanced” You can, for example, create graphs, charts and tables. You can embed them in any number of custom reports which can be saved and rerun at will. You can even make reports that you create public, which is a great feature if you either want to be able to monitor your data collection without logging into the software or if you have clients whom you want to be able to share your results with without letting them log in.

You can control which questions appear in each report. You can make a variety of different types of charts ranging from bar charts to pie charts to even spider charts. In tables, you can control which choices appear in each table and the order that the choices appear in. You can even create cross-tab tables that are regularly generated inside of your custom report. You can even add statistic tables for each question which include the mean, variance, standard deviation and total responses for each response choice. Reports can be exported to Microsoft Word, Microsoft Powerpoint, and Microsoft Excel. Qualtrics even lets you drill down in your responses, which is a pretty neat feature.

All of these are great reporting features and for those of you who tend to do your reporting directly from the survey software — who want something quick, easy and fairly powerful right out of the box — Qualtrics fits the bill.

If you’re looking for advanced analysis capabilities, such as those offered by a few other expensive packages, you aren’t going to find them here. But then, there really aren’t that many survey software packages that offer advanced analytics. And if you’re an advanced market research kind of a person, chances are you are going to use an external software package to analyze your data anyways.

Overall, I was very impressed by Qualtrics reporting and data management capabilities. I was a little disappointed by the fact that it doesn’t offer “true” SPSS export, but what it does offer is enough to make getting the data into SPSS manageable. The respondent editing capabilites are fine, and the report development system is excellent.

Qualtrics Security, Data Protection, Company Viability

Qualtrics is a privately held company located in Provo, Utah, just south of Salt Lake City. The company started a littled over ten years ago and has a very wide range of corporate clients. In my opinion, you shouldn’t have any concerns about working with this company.

Survey links are secure — there is no reasonably way for other people to guess the link to your survey based on links generated for other surveys. Furthermore, every invitation sent through the system has its own unique complex URL, which means that respondents can’t guess other respondent links and use it to get access to personal information.

Qualtrics allows each individual user to have their own account, and there are a number of different security options that can be assigned to each user. Surveys can be shared or not shared. Data can be shared or not shared.

Qualtrics maintains its data behind a firewall and all data can only be accessed only by the owners of the survey. I do not not whether survey data is encrypted on their server.

Pricing/Options

When you sign up for a Qualtrics account, you are generally allowed to have an unlimited number of users and can create an unlimited number of responses. A certain amount of service is included, and a certain number of custom templates will be included depending on the package that you purchase. Your account will generally be priced based on the number of responses you want to collect over the course of the year.

Qualtrics isn’t cheap — accounts tend to start at around $10,000 and go up depending on the number of responses you want to collect. Qualtrics will give you a quote based on what you’re looking to do, and my impression is that the company is pretty flexible in terms of being able to come up with a package that meets your needs at a price you can afford.

Qualtrics prides itself in offering expertise in addition to survey software — they have a lot of well educated research experts on their team and the organization has a very academic bent to it. If you’re looking for a survey system that will be able to provide in-depth consulting capabilities to your project, this might be the right survey software for you.

Conclusion

There is a lot to like about the Qualtrics survey software system. The questionnaire features and questionnaire development environment is among the best that I have seen in a web-based application. The reporting capabilities should please anyone who is into that sort of thing, and the system provides for both panel management and multi-user account capabilities.

For my own purposes, I struggle with the fact that Qualtrics doesn’t offers some of the advanced capabilities I like to have in a survey package. I miss the ability to edit the survey page structure or the CSS files. I wish I could write my surveys in Microsoft Word and import them directly into Qualtrics. I wish it offered some version of scripting/advanced programming capabilities. I’d also like it if the software allowed me to export data in true SPSS format. And for my own purposes, I wish that there was a way to install it on my own server instead of having to use it on the Qualtrics server.

But my personal approach tends to be much closer to that of a research firm than a corporate research team or an academic organization. And while I probably wouldn’t recommend this survey software for a major research firm, I would definitely recommend it for an academic organization or a corporate research department.

Overall, Qualtrics offers an extremely rich feature set and performs extremely well compared to other survey software systems in its price range.

• Qualtrics Web Site

PerkyPoll is a new “collaborative polling site” which allows you to create simple surveys that can easily be integrated into blogs, web sites, and presumably other collaborative sites such as facebook or Myspace. Although I probably wouldn’t suggest that you use it for serious research, it does have some unique elements that might make it especially entertaining if you’re looking for a simple way to collect and share survey results on a non-business critical topic.

PerkyPoll doesn’t offer a lot of question types — but it does have some of the important ones, including “pick one” (radio buttons); “pick many” (checkboxes); “ranked lists” (done with drop down boxes); and “Cumulative” (constant sum). It does not allow you to have open-ended questions, although there is a space to comment at the end of the polls.

In addition to adding questions to PerkyPoll polls, you also give the poll a name, a description, a period during which you want to collect responses, and tags which can be used to help people find your poll. That’s pretty much it for customizability.

Once a PerkyPoll is published, it can’t be changed unless you delete any responses that have been collected already. Once published, the poll can be added to a widget, which is a piece of code you add to your web site in order to have it display properly. Widgets can be customized for a couple of different sizes (300×250 or 160×300) or you can simply include a link to your poll on the PerkyPoll web site.

PerkyPolls are collaborative. That is, people can literally share the same polls on multiple web sites, and all of the results get combined. So if you create a PerkyPoll about who is going to win the 2008 election, all of your friends can include the same PerkyPoll on their web sites and all of you will share the results.

• PerkyPoll web site

Of course, it isn’t always possible to prove that this is what is going on. However, there are certain tell-tale signs:

• The same “review” appears on numerous web sites.
• The name of the reviewer isn’t published.
• The review appears on a blog that only features that review.
• The review appears on a blog that appears to have no history.
• Several survey software packages are reviewed, but only one has links.
• The review points to a landing page on the site it is reviewing, not the home page.
• The software gets a practically unblemished review.
• All of the other software reviewed on the site is compared to the one piece of software that received the practically unblemished review.
• Lots of unnecessary punctuation. Oh, they are so excited!!!!
• Lots of inexplicable links — people do this to boost themselves in the rankings.

The issue here isn’t that people shouldn’t do what they can to promote their software — but at the same time, it is absolutely wrong to create fake reviews and web sites that are “unbiased” for the purpose of manipulating people to visit your web site, and I would think twice before ever using such a company to conduct your important research.

SurveyMethods Questionnaire Features

SurveyMethods offers the traditional selection of question types, including single choice closed-ended questions (radio buttons or drop down boxes); multiple response questions (checkboxes); open-ended questions (essay, single line, multiple “form” questions); ranking questions and constant sum questions. It also offers the option of “points questions” which means simply that you can assign values to each response (red = 5, green =2, blue = 1, etc). The system also allows for basic matrix (table) questions.

Implementation of most of these question types in SurveyMethods isn’t detailed. Although in their marketing they say that they offer 20 different question types it seemed to me that many of these were just different variations on the question types listed above. The most advanced question type that is offered is a conjoint question, which sounds pretty impressive until you try to use it — besides being confusing, all it really does it let you rank various combinations of attributes (it doesn’t create an orthogonal design and it doesn’t generate utilities). If you’re serious about doing conjoint you’re much better off looking at the conjoint question option offered by SurveyZ or if you’re really serious, check out Sawtooth Software’s solution). Other “advanced” question types are limited compared to other survey systems as well – the ranking question, for example, makes you enter numbers into each box rather than letting you click or drag each option in order.

Example of SurveyMethod’s conjoint question.

The advanced options available in SurveyMethods for each question are limited. For example, I can choose to make a closed-ended single response question mandatory and I can randomize the order of the responses.  but I can’t add a reporting value for each option, I can’t assign “other” text boxes to each option (I can add one text box to the end as an “other” response), and I can’t lock items in place. If I have a lot of items, I can break them into sections. I can’t put the responses in multiple columns. I can’t select a default value or assign a unique CSS class to the question. You may not need to do any or all of these things, but if you do any detailed work in survey development you may miss them.

SurveyMethods does allow for some basic branch logic, that is very easy to understand but potentially frustrating if you have to do a lot of branching or if you want to create complex scenarios (you can’t, for example, have it branch based on the responses to two different questions). And while your survey can appear on multiple pages (which is a good thing) you can’t name the pages, which can make setting up the branch conditions somewhat annoying (when you branch you don’t tell it what question to branch to — you tell it instead what page you want to branch to, and your options are “Page 1″, “Page 2,” etc. SurveyMethods does not offer dynamic branching (which will hide questions on the page based on the responses to earlier questions on the page).

SurveyMethods does not offer the ability to “skip over” questions based on specific criteria other than through branching (although they say they offer skip logic, what they’re really offering is branching logic — skip logic is assigned to a particular question and says “only show me if certain criteria are met), which branch logic says “look at the answer to this question and depending on what it says take me to this other question.” Subtle differences, but there are times when it helps to have both types of logic available.

SurveyMethods does not allow you to randomize questions, groups of questions or pages of questions, although these aren’t common features offered by low end packages anyways.

Although SurveyMethods allows you to enter text blocks and images, you cannot enter blocks of HTML directly, nor can you include any special javascript headings or other code. There is no scripting language. You can’t include Flash or video.

Incidentally, SurveyMethods comes in three different flavors — Basic (free), Advanced, and Professional. The free version does not allow you to make questions mandatory, randomize answer choices or utilize branch logic.

Survey Designer Features in SurveyMethods

The survey designer is fairly basic — no AJAX here. The list of questions in your survey are displayed roughly as they will appear in real life, and one neat feature is the ability to change the “presentation style” of the survey on the fly through a drop-down box and instantly see the impact on your list of questions. But other than that, the look/feel of the survey designer isn’t particularly friendly (although it isn’t unfriendly either).

There are two methods for rearranging questions in SurveyMethods. One option is to click the little blue up and down arrows next to each question. The other option is to go to the move quesiton screen which lets you select from a drop down box which question you’d like to stick the question you’re moving in front of or behind.

SurveyMethods does not allow you to edit the response set of a closed-ended question once the survey has responses. This is a fairly common safety measure, but can be frustrating/annoying if you start the survey, collect fifty responses and realize you want to add an extra response. Your only option is to either delete all of the response data. Note that you also are not allowed to delete or move questions around once any response data is collected.

The SurveyMethods Survey Editor.

You are able to choose from a variety of different color schemes and the fonts, and if you like you can go ahead and create your own although with only a limited set of options to choose from (you can’t, for example, create your own CSS code). You don’t have a tremendous amount of control over the look/feel of your surveys — they all have pretty much the same structure/format. You can also upload a logo to appear at the top of your survey.

SurveyMethods offers online help, although I didn’t really like the way it was structured. Basically, you click on the “Help Center” icon and it takes you away from your survey (rather than opening the help in separate window) and presents all of the help options in the form of frequently ask questions which do not appear in any context sensitive manner.  If you’re able to find the subject you’re looking for (which can take a frustratingly long time) the posted answer is extremely short and not particularly helpful. And then when you’re done you have to hit your back button about four times to get back to the question that you’re working on.

Spell check is available, but not in the free version. The ability to add logos is also limited to the Advanced and Professional packages. Professional users can allso”collaborate” with others on survey development. It does not have a special test/debug mode for surveys.

SurveyMethods Response Collection Features

SurveyMethods offers an average set of features for collecting responses.  Users are limited in the number of responses they can collect each month based on their account level. Promotional methods are limited to e-mail invites and URL links, and only Professional level accounts can be configured to show a custom completion page or redirect to another site at the end of the survey — everyone else sees an add for SurveyMethods. The respondent experience (how fun/interesting it is to take a survey) is fairly limited.

SurveyMethods allows each account to have an unlimited number of surveys, although it limits the number of total responses an account can collect each month. A free account can collect 500 responses a month; an Advanced account can collect up to 1000, and a SurveyMethods Professional account can collect 5,000 responses. If you exceed the maximum number of responses that you’re entitled to on an Advanced or Professional account SurveyMethods will automatically add another 2,000 responses to your account and charge your credit card $5.00.

A sample SurveyMethods Survey in real life

I wasn’t completely impressed with my experience as a respondent. To ensure higher completion rates on voluntary surveys it is often helpful to make surveys interesting or entertaining — SurveyMethod surveys are somewhat plain to look at and if you had a lot of questions it is likely that the respondent would become bored faster than they would using a more interesting interface. Of course, this is just my subjective perception of the experience. To its credit, SurveyMethods does give you the option of providing several different types of progress indicators that may make the experence more pleasant. Respondents cannot save or print their responses, although if allowed by the designer they can go back to previous responses. It appears that respondents who quit the survey can re-enter their survey later to finish.

SurveyMethod surveys can be promoted in the usual ways, including web URL and e-mail invitations. One neat feature is the way in which a survey is “lauched” — it takes you through an easy to follow set of six steps that let you choose your method of publicizing your survey, setting up start pages, setting up completion options (if they’re available at your account level) and then launching of the survey. It does not allow you to embed surveys into your web site, nor does it allow you to embed the first page of your survey into an e-mail message. The e-mail invitation system is very basic, but it is also easy to use. E-mail addresses can be imported from Excel or typed in manually. If you have the Advanced or Professional version the system can also be configured to automatically send out reminders. The Advanced and Professional version of SurveyMethods can also be configured to stop collecting responses after a certain date or after a specific quota has been met.

SurveyMethods makes it possible to create a custom greeting page for your survey, but frankly even if you couldn’t you could accomplish the same effect by adding a text question to the beginning of the survey. At the end of the survey it appears that SurveyMethods will automatically take you to a built-in thank you message, presumably one that promotes SurveyMethods — except in the Professional version, where you have the additional options of creating a custom thank-you page (no HTML allowed), showing the respondent the results or redirecting respondents to a web page of your choice.

SurveyMethods Response Management / Reporting

SurveyMethods offers very limited reporting and analysis capabilities for free accounts — almost to the point of what I would consider worthless. Free accounts cannot export results, cannot generate reports, and cannot create charts and graphs. Even Advanced accounts are only allowed to generate reports and export data — they still can’t create charts and graphs or cross-tabs. All of these features are reserved for the $39 a month Professional account.

You are allowed (presumably at all levels) to view and delete individual respondents. You can also create basic summary reports of all respondents which do include horizontal bar graphs. It is a nice feature that you can choose to display results only from completed or not-completed surveys, and you can choose to have it display only certain questions or surveys that match certain criteria.

�
Sample SurveyMethods Summarized Data Report

One neat features of SurveyMethods is the ability to create reports featuring tables and charts. This charting options are fairly basic, but could be useful if you don’t want to take the time to create your own Excel charts (they look about as good as the basic Excel charts). You do not appear to have control over the order that the charts and tables appear in, although it appears that you can remove charts that you don’t want to have included. SurveyMethods allows you to export reports to Microsoft Powerpoint or PDF files, and you can manually e-mail reports to address you specifiy. Data can be exported into Excel, but oddly cannot be saved into a CSV file or SPSS format.

Survey response data is retained up to a year for the free accounts and for an unlimited period of time for the Advanced and Professional accounts.

Data Security, Company Stability, Etc.

The company does not indicate where/how the store their survey data and whether or not the data is encrypted. Nor, frankly, do they even provide any of the names of their owners on the web site or even which company they reside in. Even the SurveyMethods.com domain is registered privately so it is impossible to see who precisely owns the domain. The company’s phone number is, according to WhitePages.com, an unpublished landline located in Dallas, Texas (although with virtual phone numbers these days it is impossible to know whether the company is based in Dallas, London, or somewhere in India). Even the Terms of Use and privacy policy of the site offer no indication of who runs the company and where they are located.

SurveyMethods does not offer individual respondent authentication, but it does offer secure survey URLs (which means that there is no way to guess other survey URLs based on the URLS used to identify your survey). However, SurveyMethods doesn’t block any of their links from Google,which means it fairly easy to see a variety of studies that have been conducted on the site just by doing a domain search within Google.

SurveyMethods Summary Evaluation / Pricing

SurveyMethods comes in three subscription flavors: Basic (free), Advanced ($9 a month) and Professional ($39 a month) which dictate the number of responses you are allowed each month as well as which features you have access to. SurveyMethods offers a seven day trial account which gives you access to all of the features for free. In my opinion, the free account is fairly useless for most survey applications because it won’t allow you to make question mandatory, export data or redirect respondents to another web site after the survey is over. The Advanced option isn’t bad, although I’ve seen other survey systems that offer more at about the same price. SurveyMethods offers free Professional accounts to education and non-profit institutions — as long as they’re willing to post a link to SurveyMethods on thier web site and give credit research reports. If security matters to you, I might also be a little be leary of using

All in all, SurveyMethods is an OK system and is perfectly usable for many purposes. If this were a few years ago I probably wouldn’t be as critical of it as I am. However, the fact is that right now it does not offer nearly as many features as many of the other online survey systems that are out there. Off the top of my head I can think of at least three other systems that offer many more features as SurveyMethods at about the same price.

• SPSS Desktop Author, which was previously released in the fall, is now (apparently) officially included as part of the SPSS Dimensions suite. It allows you to create surveys fairly easily in a friendly, powerful interface.
• A new Question Wizard which makes it easy to build and rebuild questions with a consistent look. Using mrStudio is is also possible to build your own Question Wizards.
• Interviewer Player is a new Dimensions component that makes it possible to conduct in-person interviews over the internet or using a laptop. Surveys are collected using local software and the uploaded online or using a phone when a connection is available.
• SPSS Predictive Enterprise Repository “minimizes duplication and maintains research or corporate standards.” It is closely integrated with Clementine and other SPSS products.
• SPSS Reporter — which is different from SPSS Desktop Reporter — is a web-deployed interactive reporting tools that “allows users to quickly publish and share reports in visual form.” Users can droll down into the survey reports and aggregate results to better understand customer feedback. 

SPSS Dimensions 5.0 is available now from SPSS. SPSS Desktop Reporter (also part of Dimensions) seems pretty much exactly the same as 4.5.

Sam Winstanley has some great, more technical observations on his blog.

• Official SPSS Dimensions home
• SPSS Desktop Author 5 Recorded Demo
• SPSS Predictive Enterprise Repository details
• Sam Winstanley’s Blog

Let’s start at the beginning: the survey design interface. Beginning a new survey is as easy as selecting “Create Survey” from the main screen. You can either start with a blank survey, use one of the nine templates that are available (not a very big selection actually…and they’re pretty simple too), or you can copy one of your existing surveys (SurveyGizmo allows you to create an unlimited number of surveys, even if you’re using the free account).

SurveyGizmo’s survey development environment is clean, easy to navigate, and doesn’t require any training or instruction. For example, consider the blank questionnaire I’ve started above — the page even includes a green arrow telling me exactly what I need to click in order to add my first question. The entire interface is like that.

SurveyGizmo has a number of question types — and as far as I can tell new ones are regularly added. It of course includes all of the traditional questions such as single choice radio buttons, multi-choice checkboxes, drop down menus, text boxes and essays,but it also includes some really fancy question types that I’ve never seen anywhere else. For example, SurveyGizmo offers a star ranking table question (see below) that makes it easy and fun for respondents to rate items. It offers image-based single selection question (and makes them easy to implement). It allows for tables/matrixes of radio buttons, drop down menus and checkboxes and also supports a fairly friendly continuous sum and ranking question. SurveyGizmo also supports custom matrixes (table questions that allow for multiple question types in each row) and custom question “groups”, as well as file uploads — but these three features are only available to paying customers.

SurveyGizmo gives you a number of “advanced” option for each question type without making the process intimidating — thereby ensuring that you’ll have an easy time creating your survey but still have the ability to make it do exactly what you want. For example, I created a simple single-selection radio button question with three options. After I finished entering the basic question, I could then easily change the type of answers (checkbox, radio button or dropdown menus); change the orientation of the answers; set a default value; change the number of columns; randomize the options; indicate whether I wanted the question randomized on the page; hide the question based on criteria; number or not number the question; etc. I could also add a custom CSS class name. That’s a lot of options!

For text questions, validation is available to ensure that the responses fits a certain criteria. It is extremely flexible, and is based on regular expressions (or you can choose an option from the drop down list). It is very easy at any point to mark a question as required or not required. If you indicate that a question is a date, it will pull up a calendar when the respondent clicks on it — if you indicate that the field is an e-mail address, the system will do validate that the e-mail address is valid — not just by checking the format, but by contacting the respondent’s e-mail server to make sure it exists. Numeric fields can be automatically formatted as percentages, currency, or whole numbers (it won’t let the respondent enter decimals).

SurveyGizmo’s development environment is entirely AJAX based, which means that when you click a button or make a change you don’t have to wait for a postback — the change takes place immediately. Not a lot of survey packages offer this and it will greatly speed up your survey programming. Surveys can be previewed at any time.

It is very easy to add images or multimedia to your survey — however, only paid accounts are allowed to upload media for storage on the server, so if you’re using the free account you’re going to have to store your images on an account on a different system. Hard to complain though since it’s free…

SurveyGizmo allows branching, skipping and piping. Skipping can either be managed through a simple drop down box (show this question when some other question equals some value) or using advanced rules which have to be manually entered. Piping can be used to add responses to other questions to either the question text or the response area.

In addition to questions, SurveyGizmo allows for “Actions” to be inserted between questions. Some of these actions are about as advanced as any I’ve ever seen in a survey software package and allow you to do random branching, send mobile messages, allow for subscribers to log in, retrieve web content and send e-mails. Most of the actions are available only in the “Pro” version and the “Enterprise” version, but I think it is fair to say that many of the features available in SurveyGizmo aren’t even available in many of the paid packages I’ve tried.

SurveyGizmo allows you to choose from around 20 different “themes” which let you control the color scheme and the general look and feel of your survey. You can have your questions appear in whatever font you choose, and you can add your logo to the top — even for the free surveys. If that’s not enough customization for you, SurveyGizmo gives you direct access to the survey page’s underlying HTML template and the CSS file used to control how your page looks. Most survey packages make it difficult to get this level of control…

If you sign up for an account, you also get access to response quotas, auto-close on date, vote protection (de-duping), forward-only surveys, URL variable patterns, and IP-based geotracking. With the Enterprise version you can also eliminate the “powered by” icon that appears on the page.

Launching your survey is very easy and the system lets you go back and edit it at any time. Every survey has it’s own custom URL, and you have the option of creating your own custom survey URL as well. Surveys can easily be embedded into your web site and all of the code is provided for you. Special code is provided so that you can embed the entire first page of your survey into a MySpace page, an e-mail blast, or just into a page on your web site.

SurveyGizmo will send out survey invitations for Personal, Por and Enterpise accounts (not for free accounts). Upload a list, and the software makes it fairly easy to track who has taken your survey. You can also send out followup invitations.

Data can be exported in either text format or as a CSV file (no SPSS option except in the Enterprise version). One neat feature is that all data is saved on their server until you delete it (not automatically deleted after a certain period). SurveyGizmo offers an external database sync which allows you to send/receive information to outside websites and databases.

Reporting is extremely flexible — you can create multiple reports for each survey including charts, graphs and crosstabs. All account types can filter reports by date range, although only paid accounts can filder data by responses.

SurveyGizmo Pro and Enterprise allow for multiple user accounts (5 accounts for Pro, 20 accounts for Enterprise). Permissions can be user based, and team-based permissions (where groups of users can be partitioned) is also available. Surveys can also be secured to ensure privacy, although media (such as videos and pictures) cannot be secured using the software.

SurveyGizmo offers a variety of tutorials (some of them as videos) on how to use the software, ranging from how to create a new survey to personalizing your survey to details on some of the more unique advanced features. In addition, SurveyGizmo has a very active support forum with over 2000 posts one of which is dedicated to new feature requests.

SurveyGizmo is available is four flavors: “Free,” which gives you access to all of the basic features, the ability to create an unlimited number of surveys but only allows you to collect 250 responses a month; “Personal” which costs $19 and offers 1,000 responses a month plus some of the more advanced features; “Pro” which comes iwth 5,000 responses a month and “Enterprise” which comes with up to 50,000 responses a month (that’s a lot of responses).

I really like SurveyGizmo. I like the clean interface, the variety of features and the fact that it appears that they are constantly updating it with new capabilities. I can say with a great deal of confidence that I will be using this survey package for my own future projects and may even consider the enterprise version for larger applications. Certainly, if you’re doing a simple project for school or for your internal organization, I would recommend the free version of SurveyGizmo over either SurveyMonkey or any of the other free applications out there.

• SurveyGizmo