In general, questions should measure precisely what they are intended to answer. They should be as clear and accessible as possible, especially in places with low levels of education and literacy. Wording and phrasing, therefore, are crucial.
A useful guide is to look at the work of respected analysts in an area, such as voter turnout, and to examine the types of questions they are asking. What do other people internationally ask in order to measure efficacy, or likely voter turnout?
Asking questions that have been asked elsewhere also enables one to compare one's results with what has been found in other places and at other times. This is crucial. Any given result from a survey - such as that 34% of South Africans are not interested in politics - may mean something on its own. On the other hand, it may mean more if we knew that it was much higher, much lower, or about the same as people in other countries.
At the same time, a particular context might demand a uniquely worded question. Thus, there is a fine line between developing questions that are meaningful in a specific context, and producing results whose comparability helps enhance one's understanding of local dynamics.
Beyond these general comments, there are a number of potential pitfalls the question designer should understand.
Open-ended Questions
These questions allow the respondent to answer spontaneously, on their own terms. Rather than asking people to rate the importance of several possible reasons for voting on a scale of "very important" to "not important at all"' for example, one might ask them: "What are the most important reasons to vote?" Then, they have the advantage of not presuming what needs to be proven.
Open-ended questions, however, are very expensive. Most survey companies will allow for only three to four in their normal quotations. A typical question, such as "What are the most important problems facing the country?", may get dozens of unique responses. All of them have to be examined and categorized or "coded" into broader categories that are considered useful. This is a very time consuming process and drives up labour costs considerably.
Closed-ended questions, those that have a closed set of responses from which people may choose, also present a range of potential problems that are discussed in greater detail below.
Framing
Framing refers to how important issues are presented or "framed" in a survey question. Which aspects of a larger issue should be tapped? Which set of policy alternatives should be offered to respondents? Should a question on the location of a parliament tap costs and efficiency? Should South Africa have one administrative capital in Pretoria, for example, and one legislative one in Cape Town? Or should they be put together in one city? Or, should the question ask about changing the status quo? Again, as an example, should parliament be kept in Cape Town , where it presently is, or should it be moved to Pretoria or somewhere else? These question may yield quite different results, with very different political implications.
While different frames can yield significantly different results, decisions about which frames to use are almost impossible to resolve and will almost always invite criticism from some part of the political spectrum.
Question Order
Question order can shape responses by altering the larger context in which respondents think about an issue. Because answers to one question can be shaped by answers to previous ones, questions that are themselves fairly unbiased may create a very different effect when asked in combination. Questions on likely voter turnout, for example, may be biased in favour of higher potential participation if those questions were preceded by items asking people about people's duty to vote, thus reminding them of that duty.
Response Order
The order in which possible responses are listed may also have important effects on results. When extreme response items are placed before a more moderate response - the "contrast" effect - the preceding extreme responses increase the likelihood of choosing the following, more moderate response.
Order effects also differ according to interview method. With phone surveys or personal interviews that are read out, there is a "regency effect" where respondents tend to choose latter options from a list because they have more time to think about them. In contrast, visual presentations like show cards or mail questionnaires may have a "priming effect" where earlier alternatives tend to be chosen because people are more likely to think about the first alternative.
One-sided vs. Forced-choice Questions
"One-sided" questions ask people to agree or disagree with a statement, to favour or oppose some position, or to state some degree of an opinion. With "forced-choice" questions, the researcher attempts to provide balanced alternatives, such as, "Do you favour the government doing X policy, or should it pursue Y policy?"
"Agree" or "disagree" response sets tend to bias results in favour of the "agree" response, especially when knowledge is low. Less educated respondents with little political experience may be especially susceptible to these effects. When people have given little thought to a topic, they are less likely to develop counter arguments against one-sided statements and are more likely to acquiesce.
The typical solution is to offer respondents a second, or even a third, substantive alternative - a forced choice. This provides respondents with a counter argument. This generally decreases the number of people favouring the first alternative in a single-sided format and also changes the distribution of opinion. Yet the strength of the arguments and alternatives presented is important - not all are equally effective. Creating a second substantive alternative also places researchers in the awkward position of shaping public opinion by deciding which alternatives to include as well as the substance of those alternatives.
Double-barrelled Questions
One pitfall typically to be avoided is the "double-barrelled" question. Here a proposed alternative is coupled with a solution. An example would be, "Do you approve of a tax increase to end the budget deficit?" Respondents may not be clear about what their response will mean.
Does a "yes" mean they approve of the tax increase, for example, or getting rid of the deficit, or both?
A "one-and-a-half-barrelled question" contains qualifications that lead respondents toward choosing a specific alternative.
Key Words
Another potential area of difficulty is the actual wording used to describe the object or referent of a given proposition. Is government funding designed to deal with "drug addiction" or "drug rehabilitation", "assistance to the poor" or "welfare," "assistance to the poor" or "improving conditions of the poor"? Are respondents being asked to approve of the "president's policy" or his "handling of the policy"?
What words describe the actual choice respondents are asked to make? Are they asked to "approve," "support," or "favour" something, or rate it on a scale of "excellent," "quite good," "only fair" or "poor"?
A special form of this problem is the association of "buzzwords", or words used to evoke emotional responses from respondents, with response alternatives. These words may inject partisan or ideological calculations into responses, as well as allow less-educated responses. In the United States, the word "communist" was well known for its effects on responses to foreign policy questions. The mention of the president almost always has a large impact on the results, usually in favour of whatever the White House has done. In South Africa, one might well obtain significantly different results if people are asked to compare their lives now to twenty years ago, that is, if they were asked to compare to "life under apartheid."
"Don't Knows"
Question format also affects the number of people who offer an opinion. Increased levels of "don't know" (DK) are obtained by agree/disagree questions, questions on remote and abstract issues, and ones with greater task difficulty (e.g., ones that require long explanations or require respondents to make future-oriented projections).
The level of DK can also be affected by the use of a "filter", such as, "Or haven't you had a chance to think about this?" Filters usually increase the absolute number of DK responses by legitimizing non-response. But filters can also affect the actual substantive distribution of opinion. Those who are likely to give an opinion when they really don't have one do not come from the various response options in a random way. The problem is made more difficult because "floaters" (those who give differing responses to different types and forms of questions) are difficult to predict and don't seem to be characterized by any single trait.
All of the potential difficulties reviewed in this section arise because of the way human beings think. The way in which humans process information is greatly affected by how that information is presented. This is no less true of the survey environment. People do not conduct exhaustive searches for representative instances of an opinion or an attitude in long-term memory. Rather, they look for the most accessible information from either the environmental context of recent history or experience, or from the immediate context of the questionnaire and interview.
There are no easy answers. A question's meaning is always partially dependant on where it is placed in the questionnaire. The way we consider our response is always partially dependent on what the response alternatives are and how they are presented to us.
Besides a few well-worn "rules of thumb," therefore, there appear to be no obvious solutions to many wording problems. While we may try to avoid "contaminating" questions, "clean" questions stripped of political context may be unrealistic and irrelevant. Buzzwords and double-barrelled associations are often what give a survey questions political realism.
