Keys to an Effective Survey: Q&A

In a recent teleconference on survey design, the following questions and answers were discussed.

Q1. Is it better to have few open-ended questions?

A1: Survey design is driven by the objectives of the survey. The challenge with open-ended questions is that you end up with many varied/different responses. They are useful if you are looking for general information and if you want many broad ranging ideas about a topic. While open-ended questions aren’t recommended in general, it is good to have comment fields in your survey so respondents have a place to provide input that may not be covered by the survey questions. The reason open-ended questions are not recommended is that they are rarely actionable since not all respondents complete them.

Q2. What is a good response rate (in percent) for web surveys?

A2: The response rate depends on the connection the target audience has with the survey sponsor, which, in turn, dictates the respondent’s interest. If the target audience is customers or employees, the response rate will generally be considered good at about 50%, above 50% would be very good. If there were no specific connection with the target audience – for example a market survey on a new product – a response rate of 25% or more would be considered good.

Q3. Can you give examples of effective wording to help the audience be more definitive in their decision response instead of answering middle of the road to “play it safe”?

A3: I take issue when someone indicates they want to “force” respondents to answer one way or another. There is no wording that can force a respondent to be more definitive. They either are or they are not and the survey’s purpose is to measure that. When you remove a neutral (or middle of the road) response you are muddling the data (mixing those that are definitive with those that are middle of the road). The survey should be able to mirror real, which we all know is not always black and white.

Q4. What time of day and day of the week is best for sending an online survey?

A4: This depends on if you are sending a B2B or B2C survey. Email research shows that for B2B Tuesday through Thursday is most effective, as Friday emails can get overlooked and if it is missed it then becomes part of Monday’s email. Time of day has not been shown to be critical.

B2C emails are most effective on nights and weekends because most people work and will not use work time to answer a non-work related survey.

Q5. Do you think it is a good thing or a bad thing to end a focus group with a written survey given that they have already been asked all of the questions in the group?

A5: It is a good thing, particularly for a focus group. Groupthink often clouds individual thoughts. The loudest voices are often the only ones heard. We have sometimes even provided a written survey ahead of the focus group. This provides participants a chance to make notes and see and think about the questions ahead of time.

Q6. Are customers less likely to respond to an online survey if they know we’ll know who they are by their email address?

A6: This can be an issue but in general it is not. Of course the more sensitive the topic the bigger issue it is, but 99% of the surveys I do have very little confidentiality issues.

Q7. How do you decide when to keep a survey blind or non-blind? Of course, a customer satisfaction survey would be by definition a non-blind survey. But what about some of the more attitudinal/behavioral, competitive and brand – equity related surveys?

A7: The only time to keep a survey blind is when you’re going to lose something by using a non-blind survey. A non-blind survey is when you include the name of the company doing the survey and a blind survey is when the respondents don’t know who is sponsoring the survey.

For example, if you are Nike and you want to know how you compare to another company (you want unbiased, honest feedback). The feedback may not be as honest and forthcoming if the respondent knows Nike sponsors the survey. On the hand if you are doing a customer satisfaction survey the respondent needs to know Nike is sponsoring the survey which shows they are interested in what customer think.

Q8. A follow – up question to the blind vs. non-blind — how does this affect response rate? Does it help the response rate if not blind? By blind, I mean … not disclosing who is conducting/sponsoring the survey.

A8: The response rate has nothing to do with blind, only with non-blind. A non-blind survey helps the response rate because of the connection the target audience feels with the company.

Q9. What is the best time to send a survey after an event so people have time to digest the event and can thoughtfully respond?

A9: In general, sooner is better however sometimes it is not practical to survey participants immediately. For multi-day events it would be great to have an email waiting in their in-box when they return home because people will forget the details over time. The survey objectives can also dictate the timing. If the event is a training and the objective is to get input on the value of the training in practice then you need to provide enough time for the participant to apply what they have learned.