Hors-Série IA 2020

Web surveys: How to optimize your answers rate?

Web-surveys

Online surveys imposed themselves as the most efficient way and the fastest way to collect data with each kind of public: consumers, customers, collaborators…
But with the request multiplication to answer questionnaires on Internet, we assist to the same phenomenon than the one that happened for the surveys by phone: a regular increase and constant refusal and abandon rates during a questionnaire, and the most troubling, the random answers or voluntarily wrong answers rate.
The explanation of these phenomena is essentially linked to the structure and to the management mode of web surveys. The challenge for all online questionnaires authors is to adopt the most efficient techniques in order to optimize the answer rate and the quality of the collected answers.

Here are ten advices coming from the best experts in order to optimize your online surveys…

Rule n° 1 : Shorten your questionnaires

First rule consists in doing a short questionnaire. This can seem so obvious when we look at the interviewed people side and when we think about these endless questionnaires that we often get and rarely finish.
But when we are in the other side and that we have to design a questionnaire, big is the temptation to talk about everything by multiplying questions, precision demands and precision comments. Some surveys responsible persons are only interested in their aim and their problem and they forget the fact that they face people and not machines to answer their questions. Those who pay the respondent people sometimes tend to consider the payment or the bonus as a way to give them the right to ask anything to people and to provide long questionnaires. In this case, respondent people will perhaps fill the whole questionnaire but with the high temptation to give bad quality answers.
A survey from the American organism AFR (Advertising Research Foundation) perfectly showed it by having established a long questionnaire that clearly was counterproductive and even risky. This association was mandated by 17 huge surveys institutes and by many big companies (Unilever, Coca-Cola…) in order to establish the FoQ (Foundation of Quality) in the online surveys field. Its survey, that had a $1 million budget, proved that a 30 minutes questionnaires multiplies the risk to obtain bad quality answers per 6 compared to a questionnaire about the same theme but expected to only last 15 minutes.
Our advice goes further: limit the time of your questionnaires and make them last only few minutes (5 or 10 maximum) by concentrating on only one clear issue and by resisting the converse temptations and resisting the demands of the ordering persons. Whether the number of researched elements does not enable to do a short questionnaire, the solution is to divide the questionnaire in several shorter questionnaires. Give clear and exact information to respondent people about the real time of the questionnaire. Also use methods and advices from the next slides in order to give to the respondent the feeling of having a short questionnaire (conditional displays and pages, compact layout, cursors utilization…).

Rule n° 2: Take care of the appearance

Today, internet users are used to have a visual ergonomy and quality level to respect in the questionnaires made for them.
Of course, it is not about making a too sophisticated and too original questionnaire with special effects, animation or too artistic layout that could perturb the respondent and harm the good run of the survey. But you must not design dull and bad presented questionnaires, such as the one in the left example. The right example, that has exactly the same question, presents a sober structure but professional. Some online surveys tools, such as the Net-Survey software that is used here, proposed standards quality layouts by enabling you to easily adapt the graphical interface of your company.
Let us here point to importance of keeping pages of the questionnaires as light as possible. A page having big size images takes more time to be loaded. This can make the questionnaire slower, especially during the survey launching, if the surveys responsible person gets the bad idea to request thousands of respondents at the same time. During the first hours, many simultaneous connections bring a leverage of the heaviness for the server, that is why it is important to minimize the load to be treating by conceiving weight optimized pages for the questionnaire.

Rule n° 3: Well thought-out ideas are those that are explained clearly

In order to have correct and useful answers, you should ask clear questions without any ambiguity. The internet user, who is alone behind her screen, must perfectly understand what is asked to her and what choices are proposed without having to do any particular efforts.
The questions (and the proposed answers) should exclusively use words and expressions coming from the everyday language and understood by everyone in the same way. The technical jargon, abbreviations and complicated words must be avoided. If it is essential to use them, you should give a clear explanation of their meaning (but short). Sentences formulation also is important. The negative forms or negative-interrogative are more difficult to be understood and should be avoided. Also, conciseness is a needed quality, both for not losing the respondent person and for giving a dynamic and short appearance to the questionnaire. Questions should actually be expressed by short and precise sentences. Ambiguous terms that could end to wrong answers absolutely must be avoided. So, it is needed to replace terms such as “Often” or “Rarely” by more precise scales such as “More than 3 times a week” or “Less than once a month”, in order to avoid different interpretation from one person to another.
In the same clarity idea, it is needed to deal with only one subject at the time in one question. By asking the internet user judgment about the “quality and prices of our products” you will be sure to get some biased responses (is it about quality, price or quality/price relationship?). Such a question must be divided in two or expressed in another way if it is about evaluating the judgment of the quality/price relationship.
The logic in questions and responses proposals also is essential. Unfortunately, it is common to have respondent people who find themselves at an impasse to answer because of some questions that are not adapted to their context and for which they actually do not find an answer, or because of the incomplete list of responses that actually do not cover their case. Temptation with such a situation is to simple close the window (or to answer with any random response whether the survey is paid).

Rule n° 4: Limit mandatory questions

This is not because you decided that the respondent absolutely must answer to all of your questions that you will have what you want. Conversely, the mandatory questions utilizations often make people bored, especially if the questionnaire is badly designed and that proposed responses do not cover all possible cases. Facing the impossibility to continue without answering the question whereas none of them is suitable, the only way out is the abandon (or answering with any random response, what more often happens than what we think). Rather than propose only mandatory questions, such we can often (too often) see, it is better to reserve mandatory question only to essential questions (the questions that cannot be avoided because the questionnaire would not be sensible). For example, this is the case with questions enabling the connection to different parts of the questionnaires. But even with this case, it is needed to provide a derivative response that will result in the end of the questionnaire (with a thank you message).

Rule n° 5: Optimize your tables

It is common to see many questions on tables in online questionnaires. These questions follow on without break for the respondent person. Usually, each table represents one question for the surveys responsible person. And when it is mentioned at the beginning that the questionnaire will propose around 10 questions, hundreds of responses are actually expected from the respondent person.
Besides the importance to limit the global number of questions in the survey, it is essential to avoid grouping tables with too many successive questions. Actually, whether the structuration into tables enables to make the evaluation easier, the multiplication of the lines results in the weariness and the temptation to quickly answer, in a less precise way.
We advise to limit on one hand the number of tables (no more than 3 or 4 in a survey) and to limit the number of items in a table (around ten in each table). Whether the items consist in several lines sentences (what you should avoid to do), their number must be reduced in order to obtain a table that people can entirely see on a medium resolution screen (today: 1024 x 768). This is actually important to have the row titles visible for the respondent when she ticks the answer.
In order to make the finding easier on this table, it is essential to alternate 2 different colors for pair and odd lines.
Some online surveys tools such as Net-Survey enable to make the table easier by replacing the responses rows by rolling lists or, even better, by imaging cursors. Beside the playful aspect, this last presentation has the advantage to enable opposite presentations on tables with many aligned items with a cursor for each of them (above and below) by giving a feeling of simplicity and density.

Rule n° 6: Move from the general to the specific

The questionnaire organization is important to optimize the feedbacks. Actually, a questionnaire beginning with complex or too much engaging questions can discourage respondent people. In order to reassure interviewed people and encourage them to answer, a “funnel-shaped” questionnaire is needed, by starting with the easiest and less engaging questions before moving to the most difficult and the most personal ones. Unless you need to check quotas for the signing questions, these questions would be better in the middle of the questionnaire than at the beginning of the survey.
First questions need to be simples and clears. We can ideally begin with questions giving a simple choice and having an obvious response for the interviewed person, in order to make her feel confident. After this, we can deal with factual questions (habits, behaviors) and then the opinion questions (evaluation, satisfaction, recommendation).
About the satisfaction questions, it is advised to group the different criteria to be evaluated into reasonable size tables (cf last slide). In the idea of moving from the general to the specific, we need to place a question about the global satisfaction before placing satisfaction items. Besides the fact that this question is essential for some advanced statistical analyses (ex: calculation of the importance matrix compared to the satisfaction in Stat'Mania), its position before the exposition of detailed elements to be evaluated enable to obtain a global evaluation that is more exact and less subject to a modification by the respondent person according to her responses to the different items (whether they were placed before).

Rule n° 7: Guide the respondent people

In order to optimize the feedbacks, it is important to well welcome and well guide the respondent person among the questionnaire.
First screen should present a greetings message and a brief but clear explanation about what we want the respondent to do. It is also important to precise the estimated time of the questionnaire by giving a clear and sincere indication.
If you directly address to people that with who you already had a contact (customers, prospects, collaborators…), you should tell about the aim of the survey from the beginning and reassure about the anonymous aspect of the results utilization. The surveys made on the staff (360 degrees, social climate, stress at work, payment…) usually are the most sensitive concerning these aspects. The use a faithful third party (surveys institute or advising company) to design the survey, or at least, to host it on external servers to the company, enables to reassure collaborators and to get more reliable answers.
In the running of the questionnaire, it is advised to regularly give indications about the way to fill questions (précising that the respondent can tick several boxes for multiple questions for instance).
Some filling mistakes should also be clearly indicated rather than simply block the respondent people on the page by giving her the temptation to abandon during the filling. If, for instance, mandatory questions were not filled, it is needed to highlight them in a clear way after having displayed a matching message (generally we circle them and put a star or we change the color).
Finally, it is important to thank at the end of the questionnaire and to give a link that enables to quit the questionnaire by automatically closing the navigator window.

Rule n° 8: Place an advancement indicator

When a questionnaire exceeds 2 pages, it is important to give the respondent person a precise idea about the screens number that she will still have to display. To do this, the usual method consists in placing on each screen an advancement bar that will indicate the percentage of elements to be seen.
It is essential to avoid cheating on the given indication. However, in some questionnaires that adapt to the respondent context, the advancement is difficult to judge because pages to be passed depend on the answers of the interviewed person. In this case, the indicated advancement cannot exactly be proportional but can be optimized according an average with lower or faster evolutions on some screens. However, you absolutely must avoid an advancement bar that steps backwards.

Rule n° 9: Test the questionnaire before publishing it

Generally, all serious surveys responsible people test the whole questionnaire before diffusing it by imagining all different filling cases and by verifying the successions, the conditional masking, the rotations and other automatic fillings. However, we do not always think about testing the questionnaire on different navigators. And so we sometimes risk being surprised, even on small random displays but that differently behave according to the navigator.
We advise to systematically verify your questionnaires on navigators likely to be used by the target population (by also thinking about testing them on different versions of each navigator that are still used).
In France, the most used navigators (January 20132) are: Chrome (32,15%), Firefox (28,72%), Internet Explorer (27,17%), Safari (9,72%) and Opera (1,01%). You can consult actualized statistics and for the geographical area that you want on the American Statcounter website.

Rule n° 10: Correctly request your penpals

This step is essential to the success of the survey. At this level, you must both take care of the message and of the timing.
About the message, an appeal to be a part of the survey should comprise the invitation text itself with a direct link to the survey. Whether the survey is anonymous or not, the link should be personalized in order to directly connect the requested person to a questionnaire identified by a code. This is the only efficient way to avoid a person to not answer several times to the same survey by clicking on the same link. If a later click happens, the system will be able to position the respondent on the same questionnaire or to indicate her that she finished to answer it. The alternative system that consists in using cookies is to be prohibited because it only works for users who authorized cookies on their device.
Message should of course be clear and have a well appearance. But this is not necessarily useful to integrate images that risk to not being able to be displayed on the screen. A simple text properly designed and disposed often seems to be more serious and less “commercial”.
About the timing, it is useful to choose appropriate moment to address the emails. A message sent to professional on a Friday night could not be read on Monday morning because of the huge quantity of other emails and tasks of the new beginning week. You should also avoid sending email on Monday. For this population, Tuesday, Wednesday and Thursday should be favored. For private individuals, Saturday is the day where opening rate is the most important.
The spread of requests is highly recommended in order to avoid overloads of servers that host the survey. Actually, the addressed messages are usually opened within the minutes or hours following the sending, and rarely after this time. If thousands of respondent people are requested at the same time and get to the survey at the same time, slowing down should be expected, whatever the powerful of the system. Some surveys platforms such as Survey-Manager intend the spacing out of the messages sending (a spacing out that is about some seconds fractions) in order to avoid the overload as well as the risk of becoming a spam…