Qualitative Research Considerations
There are multiple types of qualitative research including formal and informal focus groups, round tables, feedback forms, Webinars, seminars, usability studies, etc. These meetings can include internal participants (sales, management, customer service, engineering, etc.) or external participants such as analyst, press, existing customers, prospects, governing committees, and others.
This section will concentrate on focus groups. As additional content is added it will cover the other qualitative approaches.
All research methods have benefits and limitations. This is especially true of qualitative methods because the research tends to be less structured and the results harder to interpret than those of quantitative techniques. Here are some pros and cons of focus groups, followed by the decisions you will need to make once you’ve decided they fit your plan.
Benefits of Focus Groups
- The study can be created and conducted relatively easily and results reported quickly.
- The method allows flexibility. For example, the Discussion Guide can be modified easily during the project when findings spawn new questions.
- A relatively large number of topics can be researched during the study.
- Through probing and group interactions, participants usually offer in-depth insights and context.
- Segments of the market that may be hard to reach through other methods can be included.
- Confidentiality can be protected because focus groups are done in facilities where all aspects of the experience are controlled.
- Decision-makers can observe sessions – live attendance or live broadcast. Sessions can be videotaped and shared with all interested parties in the organization.
Limitations of Focus Groups
- Focus groups are seductive. People tend to assign greater importance to results than warranted. (Observing live participants often leaves a much greater impression than reading a quantitative summary.)
- Related to the first point, people often ascribe quantitative analysis to the results – counting the number of first-place votes, etc. – despite the very small sample size.
- People often “hear what they want to hear.”
- Due to the nature of the results, it is hard to analyze and interpret focus group results.
- Focus groups are all about the interactions of people. Unless they are moderated by a skilled moderator, there are numerous opportunities for problems, e.g., inappropriate participants, dominating members, leading questions, other biases – and improper interpretation of results.
- Some researchers do not take focus group results serious because they are very small samples and somewhat difficult to interpret.
- When used effectively, focus groups are quite cost-effective. When not, they are expensive for the returns. A very large survey (thousands of contacts) can be cheaper than five focus groups.
Decisions About Focus Groups
- How many? You want enough to feel you’ve heard enough to act and take in the range of possibilities. When you continue to hear the same answer repeatedly, then you’ve typically hit the limit. But they can be expensive.
- Where and how many cities? You’ll want to collect information from your target markets – and maybe explore a few other locales.
- How large? In most cases, 8-10 participants work best.
- How long? Depending on the objectives, 1-2 hours is best. You’ll meet diminishing returns after 2 hours.
- What type of facility? For discussion groups, you’ll want a regular facility, usually with a two-way mirror, videotaping capabilities, etc. These facilities exist in most cities. For special requirements – taste testing, computer software, etc. – you will need to find specialized facilities – or set up your own.
- Internal or external moderator? I favor external moderators because no one can claim bias and people are often more willing to listen to constructive criticism from an outsider. On the other hand, politically you might have to use internal folks. The benefit is knowledge to help with probing, etc.
Obviously the first – and perhaps most important – consideration is when and why to use focus groups. I’ve found the most useful, and the literature concurs, at two key points: the start (concept) stage and the prototype stage. At the concept stage, you have the opportunity to hear your customers (or other participants) describe and discuss, answer questions, etc. It’s a great opportunity to become familiar with their words and hear them think out loud about key issues. At this point, the groups often spawn additional questions or ideas. The information also provides a context for the development teams. At the prototype stage, you will get reactions to specific models, executions, etc. Again you hear the customer’s words.
One of the other benefits of focus groups is that the research is confidential – the participants do not know who is doing the research or why. This is especially important for new product development.
NOTE: To ensure absolute confidentiality I also make it a point not to deal with the facilities directly. I let the moderator handle all the direct interaction with facilities, recruiters, etc. If you want more direct control, you might consider creating a DBA for the research department.
Preparing for the Groups
Here is the process I use to prepare for focus groups.
- Determine the purpose and objectives of the research. After meeting and discussion, this can usually be stated fairly clearly. At this meeting, I always set a follow-up meeting a month to two weeks before the groups. The purpose of that meeting is to make sure nothing has changed about the purpose, materials, etc.
- Develop questions and lists of assumptions. I ask the key players to develop lists of questions would like asked – and more importantly – lists of assumptions they have. These lists will help you frame the research and focus it very tightly. At the same time, develop a schedule for the development of any materials you will need for the groups. Remember that in some cases you might want to send materials to participants before the groups so build in time to do so.
- Hire a moderator. This can actually be done anywhere early in the sequence. Initially, you may want the moderator to participate in the planning meetings so that he/she can help clarify the purpose and has a clear idea of it. As you get more comfortable with the process, this isn’t necessary (which saves money).
- Contact the moderator early. Most good moderators fill their calendars quickly, so it’s a good idea to get on their schedule as soon as possible. As a rule of thumb, I’d say to contact the moderator 2-3 months before the dates of the groups. On the other hand, there are lots of moderators so you can usually find one for your project.
- Identify the participants. This is the most important aspect of the process. Recruiting the “right” participants who can express themselves is essential for useful information. There may be several segments of your customers. You need to decide who you want represented in the research. I review the assumptions and query the key players to make sure I recruit participants with the “right” attributes.
- Determine the group size. Most moderators like groups of 8-10 participants. If the group is larger, some will have very limited opportunities to speak. If the group is smaller, there are fewer chances for group interactions. Of course, the number in the group could depend on what they will do, the size of the room, etc. Recruiters will recruit 12-13 for 8-10 for the group. This will ensure at least 8-10 folks, with some opportunity to select the “best” candidates.
- Identify Sites/Target Markets. Depending on the purpose and objectives, you’ll want to select what cities (countries, etc.) in which you do the focus groups. For the general population, it doesn’t make much difference where you go. Most midsize to large cities have facilities for groups. (Of course, larger cities have more and better facilities.) The more specific the recruiting requirements, the more important to go to target areas where those characteristics can be found – or to the largest cities where population segments are largest. (For example, if you are looking for input from Mexican Americans on an ad campaign, you’ll probably do research in Texas, Arizona, and California – unless there are other characteristics as well. For senior citizens, you might visit Florida and Arizona.) Your sales staff and the moderator can help identify the best sites if needed.
- Book the facility. The best facilities can get booked up a couple of months in advance. If the location of the facility is important to you – near the airport, in the neighborhood you want to research, large enough for your group, recommended by the moderator, etc., you should have the moderator book it as soon as possible. In most large cities there are several competent facilities, but in smaller venues, there might be only one or two. In those cases, it’s important to get the room booked – or be flexible about your research schedule (or find a different but similar site).
- Create a screener. Most recruiting is done either by facilities where the groups will be held or the moderator’s office. In either case, the recruiters will have very limited knowledge of you, your product, or your customers. So you need to create a screening document that helps them find the “best” participants. This is called a screener and is simply a list of questions the recruiters ask to focus in on the folks you want. Questions can include age, income, profession, familiarity with the product, etc. Screeners are easy to create—but I often have to tweak them once they’re in use because additional insights emerge. FOR example, you might be looking for parents of children between 12 and 21. After a few recruiters ask the question you realize you only want parents of children 12-21 that live at home – so you add that to the screener.) I’ll send some samples of simple screeners.
- Follow-up with recruiting. In most instances, the moderator will monitor the recruiting. I have this done on a weekly basis at first and then daily as the groups get closer. The moderator updates me, makes suggestions on refinements, etc. Invariably recruiting will not go as well as expected somewhere. You’ll need to make decisions about changing the site (or canceling that location), changing the “requirements” to qualify for the groups (generally not a good idea unless you change them everywhere), or other recommendations offered by the facility or moderator (increase the incentive, change the date or time, etc.).
- Send a screener early. Also, most recruiters won’t start the bulk of the recruiting until about two weeks before the dates of the groups. However, I try to prepare and send the screener at least a month before – especially if there are lots of parameters. I encourage the recruiters to do a practice run to gauge how hard it will be and to see if they have questions. (Sometimes I’ll even have a local facility do a pretest if I know I need to refine the screener.)
At this point, you have made all the external preparations for the focus groups. You have a purpose, a plan, a moderator, places to hold the sessions, and participants. Now you need to make sure that the internal players are up to speed. You may wish to use the following process:
- Meet with key players. As recruiting and final planning is about to get underway, it’s a good idea to meet with the key players to make sure everything is still on target. There usually aren’t major changes at this point but the might be an additional question or conjecture that has arisen. It’s a good idea to add those to the research model before the groups rather than trying to fit them at the last minute. On occasion, the new ideas have prompted a modification in the recruiting screener – or even the locations of the groups.
- Create the Discussion Guide. The Discussion Guide is the “script” the moderator will follow as he/she conducts the groups. The moderator will create the guide – complete with time estimates – and send it to you for review. Depending on your situation, you might want to meet with the key players to review the DG or handle it yourself. To help the moderator prepare the DG, I send him as much detail as possible. I include a statement of the purpose and objectives of the research, a section I call background information – which might be information on the market, including market share and key competitors, on past history of the product (if pertinent), etc., and a list of materials that will be used in the groups. (I usually send all this information along with the recruitment parameters but sometimes wait until the internal status is finalized.) The most common question from moderators at this point is helping them prioritize the objectives – if that isn’t clear – and what to eliminate if there isn’t enough time.
- Prepare and publish the Schedule/Itinerary. I invite all the key players – and strongly encourage the project manager – to travel the circuit with me. I publish the schedule of the groups and my itinerary. I develop the schedule/itinerary so that it works best for me since I have to live by them week in and week out. For example
- Determine times. Typically I do four nights of groups (Monday-Thursday) – two groups a night of two hours each (4:30-6:30 and 7:00-9:00). I’m restricted by those times because most of my research is with teachers, who are at work until late afternoon. You may be able to build a different schedule depending on when your participants are available. I have found, in general, that groups that run much after 9:30 meet with diminishing returns, especially if participants have worked that day. Saturday groups are possible but cost additional incentives.
- Find personal lodging. I try to stay at or near the airport so that I don’t get caught in morning traffic.
- Determine the best arrival times. I fly east to west, leaving as late as possible to still arrive around noon. (West to the east with the time zones means very early flights and tighter schedules.) I also check to make sure there is at least one more flight to my destination in time to get me to the groups if my original flight is canceled. (Monday – Raleigh; Tuesday – Chicago; Wednesday – Dallas; Thursday – San Diego; Friday – home)
- Coordinate itineraries. I encourage travelers to follow my itinerary as much as possible. It’s more convenient – and can be cost-effective – if people share cabs or rental cars, stay at the same hotel, etc.
- Coordinate the attendee list. I request that anyone from the company attending groups let me know. You’ll need to know how many attendees to make sure the client rooms, food orders, etc. are large enough. You also want to alert the facility staff that you expect other folks – so they don’t mix your people with participants, etc. If they do, your confidentiality can be compromised.
- Ensure proper attendees. I strongly urge the product manager to attend all the groups. In addition, you really want to avoid colleagues who attend just one night (or one group) and consider decisions on that limited feedback. NOTE: Most facilities are equipped to broadcast the groups by video or the Internet if you need it. (My experience with the broadcasts is lots of enthusiasm but little staying power. People will focus on the first group, then start to drift away – home, back to work, etc. – so I’m not sure if you get your money’s worth. However, it can be politically powerful to offer the managers the opportunity to look in.
- Consider videotaping. Remember that the groups will be videotaped (if you wish). These tapes are available for review as needed. If your project extends more than a week, consider sending the tapes back for viewing after each night. If possible, have an associate you’ve briefed schedule a daily showing so that he/she can answer questions and remind folks that the study isn’t finished until all the groups have been completed.
- Coordinate material delivery. Be sure you have the materials and ways to get them to the facilities. The final aspect is to pull together whatever materials you need. You may need to ship them ahead so build in time to do so. If you are taking them with you, make sure you have a convenient method of conveyance.
The Groups Themselves
Here are some of the things to expect from the groups themselves – including some cautions and anomalies.
- Plan to get to the facility early. I try to arrive at least an hour before the first group starts so that I have time to review the setup and talk with the moderator. During this time you can make onsite decisions.
- Re-screen participants. The facility will provide “profile” sheets with recruits answers to screening questions. Based on these responses you might decide to keep or eliminate specific people. You might also decide that you want folks re-screened to clarify answers to a question.
- Evaluate the facility. Your host/hostess will show you to your viewing room. Depending on the size of your group or other variables, you might need other accommodations, more or different furniture, etc.
- Tweak the timing. Lots of circumstances can affect the overall time of the group – late (or early) arrival of participants, added topics or questions, etc. Each night you might need to work with the moderator to tweak the time spent on specific topics.
- Order food. I generally serve a meal to the people traveling with me, especially if the groups are going to run until 9:30 or 10:00 at night. When you arrive the facility might ask you to order the food so that it will arrive in a timely fashion.
- Plan for the inevitable. Expect the unexpected. Traffic jams, power outages, weather – almost anything out of your control could affect the smooth plans for the groups.
- Distribute meeting details. Make sure your team knows what to expect. I send out a memo to all company travelers with information about the facilities and the groups, along with maps and other pertinent information. Here are some of the things I tell my folks:
- Arrive early. Plan to arrive at the facility at least an hour before the first group. This is especially important for sales personnel or others that participants might recognize. You want to avoid having your people walk through the room of waiting participants. Also, plan to stay until the last participants have left the facility – and avoid them in the parking lots.
- Be discreet. Don’t wear, carry, or drive anything with the company name. Don’t give the person at the desk of the facility your card. Ask for the project by the name of the moderator (“The Tracy groups” or the project “The beer taste test.” I always give my people this information in the memo.) Do everything to maintain anonymity and confidentiality.
- Come to observe. Don’t plan to get work done during most of the group. The room you are in is dark and quiet. It might have a phone but people shouldn’t plan to use it while others are trying to hear the participants. There are usually workrooms available in the facility – but why come to the groups if you aren’t going to listen to the information from the participants and/or discuss with the observers.
- Expect the participants to NOT meet your expectations. For the most part, people, in general, are not as conversant with the topics of discussion as we are. They will say “dumb” things and seem to have trouble grasping what seem to be simple ideas. Folks need to listen beyond that – or realize that this disconnect could be part of the problem.
- Understand the time frame. The typical group lasts about two hours – although I have done some that lasted two and a half. Participants might be interacting and discussing topics throughout that time, so your folks need to be ready to take notes and otherwise focus for that long.
- Words of caution. When the groups are finished there will be lots to discuss – and lots of questions. (This is a good thing.) Two words of caution.
- Avoid the temptation for folks to sum up the research each night. Until you’ve heard all the groups you really don’t know what the trends are or what the cumulative effects are. Try to keep people from drawing conclusions based on nightly feedback – especially if they won’t be viewing all the groups. (This is one reason I don’t usually invite our sales personnel to the groups. Often they will focus on a negative comment and have trouble overlooking it. That comment can then be communicated among the sales reps until it causes concern.)
- Avoid the temptation to tinker with the Discussion Guide very night. To be able to draw some conclusions from the accumulated information, you want the groups to be as consistent as possible. (This doesn’t mean you shouldn’t add a question for clarification, especially if something new arises.) Of course, if the groups aren’t getting at the information you need (and you’ll know this pretty quickly) you and the moderator will need to go back to the drawing board.
Because there are several factors to consider in reporting your findings, there is a separate section on reports. Here are a few “rules of thumb” I use.
- The report should fit the importance of the purpose – or the audience. New product research probably rates a dog-and-pony show; confirmational research might get a written report and meeting.
- If you have the facility to do so, it can be very powerful to include snippets of video from group proceeding – especially if someone says something profound.
- I think a good report should include recommendations or action items – and there should be a follow-up “Action Planning Meeting” after the debriefing (so people have time to digest the findings and consider what to do next).
More on reports and debriefings later.