When I teach Survey Design Boot Camp, I like to remind students that open ended questions are like a box of chocolates: you never know what you’re going to get.
I once had a client who wanted to ask 1,000 Americans: “Where do flowers come from?” She wanted to prove that people didn’t know that the US was a major source of flowers, resulting in a catchy headline: “Only x% of Americans know that the US is a major flower producer.”
“So you want to know what COUNTRY people think most flowers come from?” I asked. “We could ask that as a closed ended question. We’ll give people a list of countries and ask which one they think is the biggest flower producer. You’ll get bigger percentages of people giving each response. Plus it will cost you less, since we won’t have to do any coding, where we go through 1,000 responses by hand and group them into categories.”
“Oh, no, I don’t want to restrict people’s thinking. I really want to get their top of mind thoughts without leading them.”
“It’s not really leading if we give a diverse list. And it will be easier for people to answer from a list when it comes to a topic they don’t think about a lot.”
“No, no, I really want to keep it open.”
“Can we ask ‘What country do you think most flowers come from?’”
“No, I just want to say “where.” People will understand that we mean ‘what country’.”
Fast forward two weeks. We got 1,000 responses to the question “Where do flowers come from?” including:
“From the ground”
“From the garden”
“From the florist”
I never tell my clients “I told you so,” but I like to think this one learned two valuable lessons: first, if you want to know “what country,” don’t ask “where;” and second, open ended questions can be good if the research is more exploratory (eg, you don’t have any idea what people are going to say), but if we are trying to prove a hypothesis and get a headline, a closed ended question is generally a better choice.