Quantitative Research Considerations
Surveys are the primary method of quantitative research – research with some claim to statistical accuracy. There are several types of surveys – and several key considerations within each. This segment will discuss two important factors in surveying – sampling and return rate – and give short descriptions of survey types – with pros, cons, and cautions. The next segment will discuss the questionnaire design.
Types of Surveys
There are four basic types of surveys: 1) mail, 2) telephone, 3) online, and 4) in person. In addition, some of these might be self-administered or done by interviewers. There are also “hybrid” techniques. Each format is the most appropriate in a given circumstance.
Mail Surveys. Mail surveys are paper and pencil instruments that are mailed to respondents. They are self-administered by the recipient, which means there is little control over the feedback. However, they are the most convenient for respondents, who can complete them in the place and time of their choosing. Mail surveys are best for the collection of sensitive information because they provide anonymity for the respondent. They provide the best opportunities for both random samples and targeted random samples. They are the least expensive way to collect data from large numbers of people.
Telephone Surveys. Surveys by telephone might be conducted by trained interviewers or by automated systems. Data collected through telephone surveys usually have minimal missing or erroneous data, primarily because it offers the opportunity for personal assistance. New automated random dialing systems increase the “randomness of the sample,” although only people with telephones are included in the sample.
Telephone surveys offer a good opportunity to reach “low incidence” respondents – populations of people that are very small within the general population. They also allow for relatively quick data collection. New IVR (Interactive Voice Response) provides researchers with the opportunity to branch – take respondents to questions based on previous responses – and otherwise customize the survey.
Computer/Online Surveys. Surveys can also be administered by computers and the Internet. All provide the potential to conduct complicated research because “help menus” can assist respondents through the survey. You can also include visual aids or images as part of these surveys. And perhaps most importantly, they are the least expensive format and have the quickest speed of data collection and reporting.
In addition, they offer technical advantages, such as control of order bias, etc. The most convenient type of computer/online survey is the Disk by Mail DBM) survey. These are self-administered, with respondents pre-recruited. They allow respondents to work at their own pace and to find answers to questions, such as brand names or the number of utensils, as needed. Surveys (CATI) with similar features can be administered by computer – with keyboard, touch screen, electronic pen, or voice-activated response. Computerized surveys administered from a central location (CASI) offer all the same benefits.
The downside of computer/online surveys is skewed or limited sampling. Only participants with access to computers outside the work environment can be reasonably be expected to respond. This sample is further limited by technophobes, who either won’t respond or who have so many problems their data is unusable.
Hybrid Methods. You can combine any of the methods – and additional technologies – to help you get better, faster, and more responses. The most common ones are Telephone – Mail – Telephone (TMT), in which you recruit, screen, instruct respondents by phone and then send them a survey. They can either mail the questionnaire back or call an interviewer.
The same method can be used with a fax machine or computer. Online bulletin boards are another hybrid method. Respondents are recruited, screened, and instructed by phone and then respond online – often to comments by other respondents as well as survey questions.
Selecting the Best Method
A number of factors will determine the type of survey you choose to conduct. Among the most important factors are how best to communicate with respondents (collect the best information), length and complexity of survey instrument, sample size, timing, and budget.
Computer/Online Surveys. Here is a very general approach to determining what type of survey will best meet your needs.
- Identify your objectives. What do you want to find out?
- Determine your sample. To find out what you want to know, who should respond to the survey (be in the sample)?
- Determine sample size/confidence. How many responses do you need to feel confident in the results?
- Determine a time frame and schedule. When do you need the results?
- Determine how much you can budget for the survey or surveys.
- Develop several sample questions. Write your questions with the best-case scenario in mind — you can ask the most complex question you can think of. Any very complex questions or objectives might determine the type of survey you use.
- Prioritize the four key factors – sample, time frame, budget, and complexity. If sampling is the most important consideration, then a telephone survey might be best. If you need responses very quickly, computer/online surveys are the only choice. If you need a large number of responses, mail – or a combination mail and telephone – a survey might be the best choice. If cost is the key factor, mail or computer surveys might be your choice. If the questions require respondent choices and interaction, telephone or computer surveys are best. Of course, there are other considerations – and my factors are usually some combination of these.
Sampling and Sampling Errors
The crucial factor in making a survey successful, which I define as getting actionable results that you are confident reflect the feelings of your market, is reducing “error.” Survey error is the term used to describe any reasons that interfere in collecting “perfect results.” You have heard public opinion polls discuss “margin of error.” The determination of margin of error is a mathematical formula – which is beyond this essay. This discussion will focus on general ways to reduce errors.
There are two types of survey error: a) non-sampling error and b) sampling error. Both can be controlled. The non-sampling error results from poor questionnaire construction, low response rates, noncoverage (missing a key part of the market), and processing weaknesses. More on that later.
The other type of error is sampling error. Sampling is the process of deciding what portion(s) of your market you will survey, including who and how many. The goal of sampling techniques is to reduce (or eliminate) sampling error. In the ideal world, you wouldn’t need sampling, and there would be no sampling error. You would (and could) survey all customers (called a census), and they would all respond.
The benefit: a high level of confidence that you know your customers. However, that is only possible if you have a limited number of customers. If your pool of respondents is large, hard-to-reach, or otherwise problematic, your only approach is to use a sampling technique. The following chart compares the benefits of sampling and census techniques.
Benefits of Sampling and Census
Proper sampling means reaching the right audience in large enough numbers to be confident that you know what your market feels and thinks. Sloppy sampling causes sampling errors which, in turn, will render the most expensive and exquisite survey plan and questionnaire worthless because the “wrong people” responded (or didn’t respond).
There are two basic types of sampling errors – systemic and random. Systemic errors occur when the sample selected reflects a bias, in other words, it does not reflect the range of findings for the whole market. The systemic error can be greatly reduced by carefully estimating the market – what the key segments are and the relative sizes of them. Random error is the other sampling error – and the most common.
It relates directly to the size of the sample – and is basically a mathematical predictor of precision. A general rule of thumb: as sample size increases, random sampling error decreases. Of note, quantitative researchers and statisticians claim that a carefully selected sample may yield lower total error than a census – attempting to survey the entire population. A carefully selected small sample can be more accurate than a less-carefully selected large sample.
Selecting a Sample
Here are the basic steps in selecting a sample.
- Define the universe. Who do you want to get information from? Decide the units (malls over 100,000 square feet), the elements (purchasing agent for a security company), the extent (purchased any or your company’s product), and time (in the last six months). These factors put a limit on the survey. You could try to survey all the purchasing agents for your companies for all malls – but it is a daunting task.
- Develop a “sampling frame.” Who are the folks that make up the group(s) you want to survey? Always use the most updated list available. In the above example, your sales manager can provide a list of customers who fit the profile. In more general cases you might need to use the information provided by specialists or industry sources – Dunn and Bradstreet, subscription lists, etc.
- Specify the sampling unit and element. What specific segment(s) will get you the information I need? All security company purchasing agents or your company users?
- Specify the sampling method. What selection criteria will you use: probability vs. non-probability, simple random, cluster, or stratified (I call it targeted)? (There are several other technical criteria that you might consider as well.)
- Probability sampling means that every segment of the population will most likely be included in a typical sample. Non-probability is a selection based on the researcher’s judgment or convenience. All my national surveys are non-probability surveys, sent to prospects in selected states, demographics, etc. but not to specific segments of those customers (because those segments don’t generally matter to my research). Non-probability selection is faster, far less expensive, and helps control non-sampling errors. The major benefit of probability sampling is that the results are projectable to the population segments with confidence.
- Simple random sampling is usually computer-generated, selecting every x name from the list to build a group of the desired size using a random number table. Cluster sampling selects names randomly with know segments of the market. Stratified sampling identifies “the best” candidates within each element, based on the researcher’s judgments.
- Determine sample size. Compute the ideal sample size using one of the formulas – or your own judgment. Several factors, including the study specs, population variability, analysis considerations, and cost, combine to determine sample size. The most common formula for computing the size of a simple random sample with a confidence level (CL) of 95 percent is:
- n is the sample size
- d is the desired precision/margin of error
- Z is the value of corresponding the desired confidence level obtained from a normal distribution table (usually 95%)
- P is the proportion being estimated
Using this formula, you can compute the approximate sample size for various margins of error. The following chart gives the basic sample sizes.
- Detail the sampling plan.
- Specify the procedures for selecting the sample.
- Select the sample.
More on Sampling Methods
Here is a list of sampling methods. Each has lists of pros and cons. (The most common determinants for me are usually cost and convenience). You can find easily understood descriptions in a basic statistics book.
- Convenience Sampling
- Purposive (Targeted) Sampling
- Simple Random Sampling
- Systematic Random Sampling
- Stratified Sampling
- Cluster Sampling
One of the factors influencing the sample size is the return rate. Unfortunately, return rates have slowly dwindled in recent years. Today most general surveys have a rate of return between 2 – 10 percent. That means to get 1,000 returns, you might have to distribute 10,000 to 20,000 surveys. Obviously, it’s important to increase response rates as much as possible. The following actions have proven to increase response rates to survey (above standard expected rate for survey type) compared to just sending the survey. The possible percentage increase is in parentheses.
- Length of the survey. Never more than 4 printed pages or 10 minutes on telephone or computer; some researchers claim 2 pages is the limit.)
- Pre-notification. A letter/postcard/phone call alerting them of the survey, recruiting respondents, etc.(+20%)
- Money. $1 – $10 (+19%) ($2 bill recommended for uniqueness). Money upfront, with the survey, is much more effective than a promise to pay, (i.e., enter your name in a drawing).
- Collateral. Mouse pads, pens, inexpensive but nice collateral (especially something unusual but useful) (+10%)
- Credible university sponsorship (+ 8%)
- Personalization (+ 2 – 8%)
- real stamp
- addressed to person by name
- letter signed by person
- Follow-up. Callbacks, follow-up mailing (at least 2 follow-ups) (+ 5%)
- 1st follow-up (postcard with Thank You and Reminder) (+ 3%)
- 2nd follow-up letter with survey (+ 2%)
- Deadline. A date with raffle (when offering a drawing include a deadline date for participation) (+ 2%)