“Web-based surveys may appear to be less intrusive and easier to conduct, but without pen and paper or a good ‘old-fashioned’ telephone, organizations miss the opinions of many, including those without a computer, the forever and selectively computer illiterate and a large part of the senior population that simply missed the tech revolution.”

The release goes on to remind everyone that paper and phone survey are ”in many cases essential, if organizations expect to present clients with the most accurate and complete view of customer attitutes and opinions.”

It then proves its point by referring to a recent Pew report:

“In fact, the Pew Internet & American Life Project recently found 49 percent of Americans only occasionally use modern gadgetry and many others bristle at electronic connectivity — the Internet.”

Wow. Who would have thought that in 2007 more than half of the US population either doesn’t use and/or extremely dislikes the Internet. We all may need to rethink our online programs and go back to the phone banks, door-to-door solicitors and shopping malls many of us have mostly abandoned.

…but before we do, here are some stats not included in the SPSS release:

• Total US population is about 300 million people with 225 million of them over the age of 17. (US Census)
• There are 178.8 million web users in the US (comScore, June 2007)
• 71% of all adults are online (Pew)
• 87% of 18-24 year olds, 83% of 30-49 year olds, 65% of those 50-64 and 32% of those over 65 are online. (Pew)
• 73% of white, 62% of black, and 78% of English-speaking Hispanic are online. (Pew)
• 73% of people living in Urban/Surburban Environments and 60% living in rural areas are online. (Pew)
• 93% of those earning $75K+, 82% of those earning $50K-$74K, 69% of those earning $30K-$49K and 55% of those earning less than $30K are online. (Pew)
• Total number of households is 105.4 million (US Census)
• Almost 70% of US households have Internet access at home.  (Leichtman Research Group Q1 2007)
• 53% of US households have high-speed access (Leichtman Research Group Q1 2007)

The Pew study that SPSS refers to in their release is called “A Typology of Information and Communication Technology User.” The study measures not whether or not people have internet access (as implied by SPSS) but instead tries to categorize people by the degree to which information and communication technologies are utilized and enjoyed.

According to the report, only 15% of the population can be characterized as “Off the Network” — that is, individuals with neither cell phones nor internet connectivity. They tend to be in their mid-60s, nearly three-fifths are women. Only 7% have college degrees (vs. the US average of 27%) and only 4% earn over $75K a year (vs. the US average of 22%). They are the heaviest users of “old media” such as radio and TV but do not have the inclination to try new information and communication technology.

Obviously, not everyone is online and if you’re looking for a particularly special group you may want to revert to paper, pencils and phones. However, I’m thinking that for most purposes you’re going to be able to find who you’re looking for online.

But the implication of the SPSS release is that unless you use (expensive) multi-channel research techniques (provided by them?) you will be collecting bad information and misleading your clients. This isn’t true and is in fact extremely misleading.

• Read the SPSS news release
• Read Pew Report “Typology of Information and Communication Technology Users”

Sometimes it seems that many of the folks writing surveys — even the professionals — don’t seem to get it that the people taking the survey are for the most part volunteers (unless you’re paying everyone who takes your survey and not doing some kind of a drawing, almost all of your respondents are effectively volunteers) and if you don’t make the experience interesting and perhaps even fun then it is unlikely that anyone is going to go to the trouble of finishing the survey.

Not every survey can be fun. Some surveys are on boring topics. Some surveys use complex methodologies that make it difficult to create any kind of positive user experience. But it seems to me that it is important for us to at least try. If we’re going to ask our volunteers to give us their time and their opinions, it seems that the least we can do is try to make the experience at least somewhat entertaining and interesting.

To that end, in addition to Martin’s article below, I also present you with a link to a page on SurveyGalaxy which offers a list of the most highly rated (i.e., most interesting) surveys available on SurveyGalaxy as rated by respondents. Note that these aren’t always the prettiest survey in the world or the most interactive — but something about them has made respondents give them high ratings.

• Article: 20 Top Tips to Writing Effective Surveys
• SurveyGalaxy List of Best Public Surveys

Personally, when I’m conducting an online survey I tend to prefer a sample of 1,000. In an overall sense it is usually overkill, but it usually allows me to segment the results in a number of different ways — I can break the results down by age group, gender, income, etc — something I couldn’t necessarily do if I started from a much smaller sample. I suppose it is the luxury of having access to a large respondent base — I can afford to oversample. Believe me, if I were paying $10 a response (like what I sometimes have to do when I rent a panel of people in another country) I am much more conservative in my sample sizes and pay really close attention to my margin of error and the needs of the study.

There are some web sites out there that make margin of error more understandable. The Red River College Marketing Research blog recently pointed to an article at www.isixsigma.com entitled "Margins of Error Made Easy!" which I found to be worth reading.

There are also several sample size calculators out there (you can find them by typing "sample size calculator" into Google). One that appears especially handy is at dssResearch.com. Grapentime.com offers not just a sample size calculator, but also a sample size calculator for attribute ratings (in other words, it tells you the minimum sample sizes you need for different type of metric mesasurement scales).

Margins of Error Made Easy at Isixsigma.com
Sample size calculator at dssResearch.com
Sample Size Calculator for Attribute Ratings at Grapentine.com

My personal opinion is that there is nothing wrong with incentives — in fact, that they can be quite beneficial — as long as you are thoughtful about what you are offering and to who. For example, if I want to find out how interested people are in visiting a particular theme park, I probably should offer as my prize two tickets to the park in question — the only people who will take my survey are those who want the theme park tickets, and those will only be people who like theme parks — and so my results will, in fact, be pretty skewed. On the other hand, if I offer something more neutral — such as cash — then I should get a pretty broad based response since, as far as I can tell, liking cash has no bearing on whether or not you like theme parks.

If you need a more credible source than my personal opinion, a variety of studies have been conducted on the subject of incentives and the impact they have on research. I recently came across a literature review by Eleanor Simmons and Amanda Wilmont ("Incentive payments on social surveys: a literature review") in which they looked at incentives from many different angles, including the impact on response rates; monetary v. non-monetary incentives; value of incentives; differential incentives (which is when you only offer to pay incentives to people after they refuse to take your survey); the effect of incentives on interviewers; incentives and data quality; effects of incentives on sample composition; and the effects of incentives on long-term expectation effect.

Read "Incentive Payments and Social Surveys" here.

The article argues that it is impossible to get a true understanding of your customer’s satisfaction by simply using one of the three metrics, and that the best way to really understand what is going on is to use some combination of at least two, one of which will be the mean score.

In terms of determining whether it makes more sense to look at the top box or the bottom box, it suggests examining how changes in either box correlate with the overall outcome.

What is most fascinating and useful about the article is actually its description of "Non-Linear Effect Analysis" in which it describes how customer loyalty tends to rise not linearly, but exponentially. In other words, customers who give a "9" out of 10 instead of an "8" out of 10 are exponentially more loyal.

Read the full article at CustomerSat.com
See all of the online resources offered by CustomerSat.com

It identifies two types of online panels:

• Self selecting panels, which anyone can join — a "convenience sample" of the internet, and one that is likely to contain "professional respondents" and possibly even competitors trying to get insights into your secret plans;
• Probability-based Internet panels, which are (painstakingly) built by randomly selecting people, calling them, and then inviting them to join the survey panel.

One of the primary sources of data for the Knowledge Networks publication was a study conducted by the Stanford Institute for the Quantitative Study of Society entitled "Comparing the Results of Probability and Non-Probability Sample Surveys." The study begins by acknowledging that in general, the same survey conducted by two firms with the same methodology will usually yield comparative findings. However, most studies that led to this conclusion focused on surveys conducted in the same mode with comparable sampling methods. The folks at Stanford wanted to find out what would happen if the mode and sampling methods differed.

Nine data collection firms participated in the study: seven of whom use a self-selecting, volunteer sample (self selecting panel) and the other two who used probability-based panels (one that used a probability based telephone sample, the other (Knowledge Networks) that used an Internet-based probability panel). Each data collection firm asked their respondents the same set of questions, and the results were compared against benchmark probability-based responses.

The findings of the folks at Stanford were that the results were "remarkably comparable" across the board. Knowledge Networks had the most accurate findings, followinged by SRBI (telephone survey) and Harris Interactive (volunteer sample) who tied for second place (all of the others were about equally as accurate).

A few questions led to bigger differences between the self-selecting and the probability based methods: for example, volunteer respondents tended to be more comfortable using computers than probability-based respondents. Otherwise, however, it would appear based on the results of the study that a volunteer sample base will ultimately lead to results that are closely comparable to the more expensive probability-based sample.

Read the Knowledge Networks publication.
Read the press release announcing the publication.
Read the results of the 2005 Stanford study.

SPSS Dimensions isn’t so much an individual software package as much as it is a platform of several independent software packages that are able to work together in a relatively seamless fashion. Sort of like how each of the programs within Microsoft Office (Word, Excel, Powerpoint, Access, etc) can work independently (and be purchased independently) but also work very well together. Like Office, most of the packages that work with Dimensions are published by SPSS — although the platform has been designed to accommodate the integration of software written by 3rd party developers (and several such packages do exist).

Some of the programs that work with Dimensions include:

• SPSS mrDialer – Automated dialing for phone surveys
• SPSS mrInterview – Create and execute online surveys
• SPSS mrInterview CATI – Create and execute phone surveys
• SPSS mrPaper – Create and execute paper surveys
• SPSS mrScan – Scan paper surveys
• SPSS mrStudio – Manage and manipulate data
• SPSS Desktop Reporter – Create tables from local data
• SPSS mrTables – Interact with tables on your desktop
• SPSS mrTranslate – Manage translations of surveys and reports
• Techneos Entryware – Collect data using handheld devices
• SPSS Base – Analyze data
• Clementine – Data mining
• SPSS Text Analysis for Surveys – text analysis & categorization

At the core of SPSS Dimensions is the Dimensions Data Model, a set of components (openly documented and supported) which allow for accessing information about questionnaires and respondent data. It also deals with keeping track of changes to the questionnaire (versioning), translating both questions and data from one format to another, and managing data stored in multiple formats and platforms.

Visual representation of the Dimensions Data Model
(from the SPSS presentation "Using the SPSS MR Data Model")

The table above describes the role of the data model well. Data can be collected from multiple sources. Instead of each collection program storing the data in its own database, it instead sends it to the Dimensions Data Model which puts it into its own special format. When another program, such as a data processing program or a data analysis program needs the data it requests it from the Dimensions Data Model using standardized request formats (that just about any program can use).

Consider a project in which you need to collect data using three different survey techniques including a phone survey, a web-based survey and a paper survey. Even though you’re going to ask the same basic questions in each survey, you are still going to have to develop three completely different questionnaires in order to compliment each of the mediums, which further means you’re going to have to program the survey three different times (perhaps four, if you consider that you may be using scanning software to read some of your paper surveys). 

Even though the question is the same, it needs to appear different
across modes and across functions (from the SPSS presentation
"Improving Government Programs with Comprehensive Data Collection")

After you’ve finished collecting the data (using three separate data collection tools, all of which store the data in their own, separate proprietary format that exports into the frustratingly simple CSV format, you’ll then have to combine all of the data into one file which you’ll then need to clean and prepare for analysis. Following analysis, you’ll export your results into yet another program.

Using the SPSS Dimensions Suite (or more specifically, software that is integrated into the Dimensions Suite) makes the process go much faster by optimizing the mechanics of designing and fielding your questionnaire and analyzing and reporting on the data.

Dimensions reduces the time it takes to conduct a complex research project
(Source: SPSS presentation "Discover it with Dimensions")

SPSS Dimensions has been developed based on the notion of "Design Once, use Many" so once you have created your initial questionnaire (either using a simple, graphic user interface found in mrInterview or the more advanced script driven interface provided by mrStudio — either package will allow you to import the text of your survey from MS Word), you can then quickly (and easily) set it up to deploy using multiple modes (paper, web, CATI, etc).

Perhaps one of the most exciting features of SPSS Dimensions is its multi mode deployment capabilities. Most surveys today require some amount of programming to deal with skipping, piping, the incorporation of outside data, and other advanced options. Ordinarily, each mode would require its own programming. SPSS Dimensions is designed so that you only need to write the script once and it will work the same in each context.

SPSS Dimensions allows you to program your survey once
and have it work on multiple platforms. (from the SPSS presentation
"Improving Government Programs with Comprehensive Data Collection")

Dimensions not only helps you design and execute your survey, it also manages security, translating the survey into multiple languages, and manages multiple versions of your survey as well.

External databases containing participant details can be added at any time, and it can be used both in the survey and during analysis. Data from outside sources can be reviewed during the survey to check for inaccuracies, and it can even be updated based on responses given in the survey.

All of the data that is collected, regardless of how it is collected, goes into one SPSS Dimensions database where it can then be analyzed and reported on. Although Dimensions is an open platform that will allow analysis to be conducted in any program (it will export data into a variety of formats for other programs to use), the suite is optimized to work with several SPSS-published programs, such as mrStudio, mrTables, SPSS for Windows and Clementine. Results can then be automatically turned into interactive web-based reports or analyzed using Excel, Word or PowerPoint. Dimensions integrates all of the major capabilities provided by SPSS’s various data analysis packages, including SPSS Base for statistical analysis; Clementine for data mining, and SPSS Text Analysis for Surveys for text analysis and categorization as well as a variety of SPSS and 3rd party data collection and reporting tools.

Reasons to Consider SPSS Dimensions

• Powerful interviewing engine
• Open architecture
• Web-based user interface
• Easy to create surveys
• Write the survey once, use in multiple modes (phone, web, etc)
• Write the scripting/programming once
• Write the survey once, use in multiple modes (phone, web, etc)
• Write the scripting/programming once
• Easy to program (similar to VB Script)
• Common data storage format/interface
• Translation capabilities
• Faster development and analysis time
• Works with (some) third party applications
• Scriptable (write your own scripts to work with data)
• Integrates well with SPSS
• Integrates well with Excel

Reasons not to use SPSS Dimensions

• Expensive
• Limited to Dimensions compatible tools
• Complicated to set up and integrate with existing systems
• Requires lots of IT support

Learn more at the SPSS web site.

• Who is doing it, how common is it?
• Why are they doing it?
• Why are some other people not doing it?
• Which modes work best together?
• When does it make sense to switch modes?
• What impact does it have on the data?
• What are the technical requirements?

One really neat concept I hadn’t thought much about was the idea of having a respondent start the survey using one mode (perhaps paper or the phone) and then have them finish the survey in another mode (usually the web). This has been found to help out when it is hard to retain respondents in one mode using a particular data collection method (perhaps they don’t want to hang out in your store for 20 minutes, or maybe they just want to get off the phone).

Multi-mode data collection will become especially useful as we adopt more mobile survey solutions — perhaps have the user start with a WAP based survey and finish up with a web based survey when they get home.

Read Tim’s full article at Meaning (PowerPoint).

The hour long presentation, led by Larry Nichter (EVP of Restaurant.com) and Brian Koma (VP of Services for Websurveyor) will include:

• Incentive best practices
• When to use incentives
• What types of incentives are most effective for increasing response rates, maximizing completion and improving data quality
• How to put incentive strategies in place that don’t blow the budget

Sign up for the October 25 webcast
Sign up for the October 31 webcast
Sign up to learn about future Perseus Webcasts

Next, a live Zoomerang moderator conducts your focus group online using their own proprietary focus group/chat software. You can watch the focus group as it proceeds from your own computer. Note that the Zoomerang Online Focus interface is fairly interactive and allows participants to write, type and draw on a white board that everyone can see and further allows them to participate in exercises devised by the moderator.

Shortly after the focus groups are over, Zoomerang sends you a final report with recommendations and conclusions, as well as a transcript of all chat discussions, recording of online comments and full audio feedback.

Pricing for Zoomerang Focus Groups is as follows: one group costs $5900 plus incentive ($240 – $800). If you want to use members from Zoomerang’s panel, the cost is around $1200. For two sets of focus groups, the price drops to $11k + incentive ($240-$800) + optional Zoomerang panel ($1.6k).

MarketTools is certainly not the first company to offer online focus groups — other companies include e-Focusgroups, iTracks, GMI, and others.