7.4 Stakeholder Engagement and Consultation

Questionnaire Surveys

Invented by Sir Francis Galton, a questionnaire is a research instrument consisting of a set of questions (items) intended to capture responses from respondents in a standardized manner. Questions should be designed in such a way that respondents are able to read, understand, and respond to them in a meaningful way. Surveys are commonly used to gather information from stakeholders because they are efficient in gathering lots of data, are of low cost, and, when designed well, can reveal valuable information.

Here are the steps involved applying a survey in your study:

Constructing a survey questionnaire is an art. Numerous decisions must be made about the content of questions, their wording, format, and sequencing, all of which can have important consequences for the survey responses. In the process, you must use the ethical principles discussed in Chapter 7.5 to guide you. Primarily, you would want to avoid doing anything that would cause physical or emotional harm to your participants. For example, carefully word sensitive or controversial questions and avoid inserting unintended bias or asking leading questions. You want to design questions to get meaningful and accurate responses rather than ambiguous information that is impossible to quantify or analyze. As a result, constructing a survey questionnaire is not a linear straightforward process. Instead, it is an iterative process, in which you would most probably need to produce several versions of the questionnaire and revise them a few times before coming up with a final version of your questionnaire.

Where to start? To write effective survey questions begin by identifying what you wish to know. In other words, refer to the research questions (general and specific) that have guided your research process to ensure that you collect all relevant data through the survey.

Let’s say you want to understand how students make a successful transition from high school to college. Perhaps you wish to identify which students are comparatively more or less successful in this transition and which factors contribute to students’ success or lack thereof. Let’s suppose you have set up the following general research question: Which factors contribute to students’ success or failure in the process of transition from high school to university? To understand which factors shape (un)successful students’ transitions to university, you’ll need to include questions in your survey about all the possible factors that could affect this transition. Consulting the literature on the topic will certainly help, but you should also take the time to do some brainstorming on your own and to talk with others about what they think may be important.

Perhaps time or space limitations won’t allow you to include every item you’ve listed, so rank your questions so as to include those that are the most important.

LLMs can assist in drafting survey questions when they are provided with specific information about your project. Remember that the LLM output will often produce generalized information and content, so a careful review of the questions and revision for your specific research goals is essential.

1. Dichotomous response, where respondents are asked to select one of two possible choices, such as true/false, yes/no, or agree/disagree. An example of such a question is: Do you think that the death penalty is justified under some circumstances? (circle one): yes / no.
2. Nominal response, where respondents are presented with more than two unordered options, such as: What is your industry of employment?— manufacturing / consumer services / retail / education / healthcare / tourism & hospitality / other
3. Ordinal response, where respondents have more than two ordered options, such as: What is your highest level of education?—high school / college degree / graduate studies.
4. Interval-level response, where respondents are presented with a 5-point or 7-point interval scale, such as: To what extent do you agree with the following statement [insert statement]: strongly disagree / disagree / neither agree nor disagree / agree / strongly agree.
5. Continuous response, where respondents enter a continuous (ratio-scaled) value with a meaningful zero point, such as their age, weight or year of birth (How old are you? What is your weight in kilograms? Which year you were born?). These responses generally tend to be of the fill-in-the blanks type.
6. Rank ordering the response options, such as “Rank the five elements that influenced your choice of the university program from the most to the least influential; assign number 1 to the most influential element and number 5 to the least influential element; properly assign numbers 2-4 to the rest of the elements.”

Generally, respondents will be asked to choose a single (or best) response to each question you pose, though certainly it makes sense in some cases to instruct respondents to choose multiple response options. One caution to keep in mind when accepting multiple responses to a single question, however, is that doing so may add complexity when it comes to analysing your survey results. Nevertheless, for each closed-ended question, clearly instruct the respondents on the number of response options they are required to choose.

Guidelines for Designing Response Options

Here are a few guidelines worth following when designing the response options.

Ensure that your response options are mutually exclusive. In other words, there should be no overlapping categories in the response options. In the example about the frequency of alcohol consumption, if we ask “On average, how many times per week did you consume alcoholic beverages during your first semester of college?”, we may then provide the following response options: a) less than one time per week, b) 1-2, c) 3-4, d) 5-6, e) 7+. Do you notice that there are no overlapping categories in the response options for these questions?

Create response options that are exhaustive. In other words, every possible response should be covered in the set of response options. In the example shown in the paragraph above, we have covered all possibilities: those who drank, say, an average of once per month can choose the first response option (“less than one time per week”) while those who drank multiple times a day each day of the week can choose the last response option (“7+”). All the possibilities in between these two extremes are covered by the middle three response options. When you are unsure about capturing all response options, you can add the option “other” to the list to enable the respondents to specify their response in their own words. This is particularly useful in case of nominal unorder response options.

If there is a reason to believe that not all respondents would be able to select from the given response options, then add the “not able to answer” or “not applicable to me” response to the response options list, in order to ensure the validity of collected data.

Use a question matrix when answer categories are identical. Using a matrix is a nice way of streamlining response options. A matrix is a question type that lists a set of questions for which the answer categories are all the same (Table 7.4.2). If you have a set of questions for which the response options are all the same, create a matrix rather than posing each question and its response options individually. Not only will this save you some space in your survey but it will also help respondents progress through your survey more easily and quickly.

Table 7.4.2 Sample Question Matrix

Ensure that response options are aligned with the question wording and vice versa. For example, if you have a yes/no question type, then you should only provide the yes and no response options (e.g., “Do you support the Prime Minister in his endeavours to foster respectful communication?”–yes/no). If you want to know about the frequency of an event (let’s say the frequency of alcohol consumption during college), then the yes/no question (e.g., “Did you consume alcohol during college?”) would be inappropriate. It would also be inappropriate to have a yes/no question type and ordered response options (e.g. every day in a week, once per week etc.). This is where attention to detail and proofreading skills come into play.

Questionnaire Design

In addition to constructing quality questions and posing clear response options, you’ll also need to think about how to present your written questions and response options to survey respondents. Designing questionnaires takes some thought, and in this section, we’ll discuss what you should think about as you prepare to present your well-constructed survey questions on a questionnaire.

In general, questions should flow logically from one to the next. To achieve the best response rates, questions should flow from the least sensitive to the most sensitive, from the factual and behavioural to the attitudinal, and from the more general to the more specific.

One of the first things to do once you’ve come up with a good set of survey questions is to group those questions thematically. In our example of the transition to college, perhaps we’d have a few questions asking about study habits, others focused on friendships, and still others on exercise and eating habits. Those may be the themes around which we organize our questions. Or perhaps it would make more sense to present any questions we had about precollege life and habits and then present a series of questions about life after beginning college. The point here is to be deliberate about how you present your questions to respondents.

Once you have grouped similar questions together, you’ll need to think about the order in which to present those question groups. Most survey researchers agree that it is best to begin a survey with questions that will want to make respondents continue (Babbie, 2010; Dillman, 2000; Neuman, 2003). In other words, don’t bore respondents, but don’t scare them away either.

There’s some disagreement over where on a survey to place demographic questions such as those about a person’s age, gender, and race. On the one hand, placing them at the beginning of the questionnaire may lead respondents to mistake the purpose of the survey. On the other hand, if your survey deals with some very sensitive or difficult topic, such as workplace racism or sexual harassment, you don’t want to scare respondents away or shock them by beginning with your most intrusive questions. Generally, it’s advisable to put demographic questions in the end of the questionnaire, unless it is required to have them at the beginning, for instance if particular survey questions serve as filter questions.

In truth, the order in which you present questions on a survey is best determined by the unique characteristics of your research—only you, the researcher, and in consultation with colleagues, can determine how best to order your questions. To do so, think about the unique characteristics of your topic, your questions, and most importantly, your sample. Keeping in mind the characteristics and needs of the potential respondents should help guide you as you determine the most appropriate order in which to present your questions.

Also consider the time it will take respondents to complete your questionnaire. Surveys vary in length, from just a page or two to a dozen or more pages, which means they also vary in the time it takes to complete them. How long to make your survey depends on several factors. First, what is it that you wish to know? Wanting to understand how grades vary by gender and year in school certainly requires fewer questions than wanting to know how people’s experiences in college are shaped by demographic characteristics, college attended, housing situation, family background, college major, friendship networks, and extracurricular activities. Even if your research question requires a good number of survey questions, do your best to keep the questionnaire as brief as possible. Any hint that you’ve thrown in unnecessary questions will turn off respondents and may make them not want to complete your survey.

Second, and perhaps more important, consider the amount of time respondents would likely be willing to spend completing your questionnaire? If your respondents are college students, your survey would be taking up valuable study time, so they won’t want to spend more than a few minutes on it. The time that survey researchers ask respondents to spend on questionnaires varies greatly. Some advise that surveys should not take longer than about 15 minutes to complete (cited in Babbie, 2010), others suggest that up to 20 minutes is acceptable (Hopper, 2010). This would be applicable to self-completion surveys, while surveys administered personally face-to-face by an interviewer asking questions, may even take up to an hour or more. As with question order, there is no clear-cut rule for how long a survey should take to complete. The unique characteristics of your study and your sample should be considered in order to determine how long to make your questionnaire.

A good way to estimate the time it will take respondents to complete your questionnaire is through pretesting (piloting). Pretesting allows you to get feedback on your questionnaire so you can improve it before you actually administer it. Pretesting is usually done on a small number of people (e.g., 5 to 10) who resemble your intended sample and to whom you have easy access. By pretesting your questionnaire, you can find out how understandable your questions are, get feedback on question wording and order, find out whether any of your questions are unclear, overreach, or offend, and learn whether there are places where you should have included filter questions, to name just a few of the benefits of pretesting. You can also time pre-testers as they take your survey. Ask them to complete the survey as though they were actually members of your sample. This will give you a good idea about what sort of time estimate to provide respondents when it is administered, and whether you are able to add additional questions/items or need to cut a few questions/items.

Perhaps this goes without saying, but your questionnaire should also be attractive. A messy presentation style can confuse respondents or, at the very least, annoy them. Use survey design tools like Google Forms, Microsoft Forms to create a consistent design and to automate the collection and visualization of the data collected.

In sum, here are some general rules regarding question sequencing and questionnaire design:

• Brainstorm and consult your research to date are two important early steps to take when preparing to write effective survey questions.
• Start with easy non-threatening questions that can be easily recalled.
• Never start with an open-ended question.
• If following an historical sequence of events, follow a chronological order from earliest to latest.
• Ask about one topic at a time (group the questions meaningfully around common topics). When switching topics, use a transition, such as “The next section examines your opinions about …”.
• Make sure that your survey questions will be relevant to all respondents and use filter or contingency questions as needed, such as: “If you answered “yes” to question 5, please proceed to Section 2. If you answered “no” go to Section 3.” Automated surveys will guide respondents to the next question based on the filters provided.

Always pretest your questionnaire before administering it to respondents in a field setting. Such pretesting may uncover ambiguity, lack of clarity, or biases in question wording, which should be eliminated before administering to the intended sample.

Qualitative Interviews

Qualitative interviews are sometimes called intensive or in-depth interviews. These interviews are semi-structured; the researcher has a particular topic about which he or she would like to hear from the respondent, but questions are open-ended and may not be asked in exactly the same way or in exactly the same order to each respondent. In in-depth interviews, the primary aim is to hear from respondents about what they think is important about the topic at hand and to hear it in their own words. In this section, we’ll take a look at how to conduct interviews that are specifically qualitative in nature, and use some of the strengths and weaknesses of this method.

Qualitative interviews might feel more like a conversation than an interview to respondents, but the researcher is in fact usually guiding the conversation with the goal of gathering information from a respondent. A key difference between qualitative and quantitative interviewing is that qualitative interviews contain open-ended questions without response options. Open-ended questions are more demanding of participants than closed-ended questions, for they require participants to come up with their own words, phrases, or sentences to respond. In qualitative interviews you should avoid asking closed-ended questions, such as yes/no questions or questions which are possible to be answered with short replies (e.g., in a couple of words or phrases).

In a qualitative interview, the researcher usually develops a guide in advance that he or she then refers to during the interview (or memorizes in advance of the interview). An interview guide is a list of topics or questions that the interviewer hopes to cover during the course of an interview. It is called a guide because any good interview is a conversation, and one topic can lead to another that is not on your list. Interview guides should outline issues that a researcher feels are likely to be important, but because participants are asked to provide answers in their own words, and to raise points that they believe are important, each interview is likely to flow a little differently. While the opening question in an in-depth interview may be the same across all interviews, from that point on what the participant says will shape how the interview proceeds. Here, excellent listening skills, knowledge of your goals, and an intuitive approach work together to obtain the information you need from the interviewee. It takes a skilled interviewer to be able to ask questions; listen to respondents; and pick up on cues about when to follow up, when to move on, and when to simply let the participant speak without guidance or interruption.

Like in case of a questionnaire surveys, the interview guide should be derived from the research questions of a given research. The specific format of an interview guide (a list of topics or a detailed list of general and additional interview questions) might depend on your style, experience, and comfort level as an interviewer or with your topic. If you are well-experienced in qualitative interviewing and you are highly familiar with the research topic, then it might be enough to prepare the interview guide as a list of potential topics and subtopics to be covered during the interview. But if you are less experienced in qualitative interviewing and/or you are not that familiar with the research topic, then it would be much better to prepare a detailed set of interview questions following research into the relevant issues.

Begin constructing your interview guide by brainstorming all the topics and questions that come to mind when you think about your research question(s). Once you’ve got a pretty good list, you can pare it down by cutting questions and topics that seem redundant and group like questions and topics together. If you haven’t done so yet, you may also want to come up with question and topic headings for your grouped categories. You should also consult the scholarly literature to find out what kinds of questions other interviewers have asked in studies of similar topics.

Also, when preparing the interview guide, it is a good idea to define and separate the general questions from a few extra additional probing questions for each general question. You may use these probing questions if required, such as in instances when the respondent has difficulties answering a given general question, if he/she needs additional guidance, or if you want to dig deeper into the topic. Often, interviewers are left with a few questions that were not asked, which is why prioritizing the questions in a list of most important to least is good practice.

As with quantitative survey research, it is best not to place very sensitive or potentially controversial questions at the very beginning of your qualitative interview guide. You need to give participants the opportunity to become comfortable with the interview process and with you. Finally, get some feedback on your interview guide. Ask your colleagues for some guidance and suggestions once you’ve come up with what you think is a pretty strong guide. Chances are they’ll catch a few things you hadn’t noticed.

In terms of the questions you include on your interview guide, there are a few guidelines worth noting:

Avoid questions that can be answered with a simple yes or no, or if you do choose to include such questions, be sure to include follow-up questions. Remember, one of the benefits of qualitative interviews is that you can ask participants for more information—be sure to do so. While it is a good idea to ask follow-up questions, try to avoid asking “why” as your follow-up question, as this particular question can come off as confrontational, even if that is not how you intend it. Often people won’t know how to respond to “why,” perhaps because they don’t even know why themselves. Instead of “why,” say something like, “Could you tell me a little more about that?” This allows participants to explain themselves further without feeling that they’re being doubted or questioned in a hostile way.

Avoid phrasing your questions in a leading way. For example, rather than asking, “Don’t you think that most employees would rather have a four day work week?” you could ask, “What comes to mind for you when you hear that a four day work week is being considered at our company?”

Collecting and Storing Interview Information

Even after the interview guide is constructed, the interviewer is not yet ready to begin conducting interviews. The researcher next has to decide how to collect and maintain the information. It is probably most common for qualitative interviewers to take audio recordings of the interviews they conduct. Interviews, especially conducted on meeting platforms like Zoom can also now be captured using AI-assisted tools that will provide a video, transcription, and summary of the interview.  For in person interviews, recordings can be captured by applications designed for this purpose, such as Otter.ai.

Recording interviews allows the researcher to focus on her or his interaction with the interview participant rather than being distracted by trying to take notes. Of course, not all participants will feel comfortable being recorded and sometimes even the interviewer may feel that the subject is so sensitive that recording would be inappropriate. If this is the case, it is up to the researcher to balance excellent note-taking with exceptional question asking and even better listening. Whether you will be recording your interviews or not (and especially if not), practicing the interview in advance is crucial. Ideally, you’ll find a friend or two willing to participate in a couple of trial runs with you. Even better, you’ll find a friend or two who are similar in at least some ways to your sample. They can give you the best feedback on your questions and your interviewing performance.