BBQ Sauce, or How to Design Survey Questions

When I moved to the US, I tried BBQ sauce and decided I didn’t like it. Too sweet. Later I learned there are two families: sugar-based and vinegar-based. The vinegar kind, I loved.

So at restaurants, before ordering pulled pork or anything with BBQ sauce, I’d ask the server: “Is your BBQ sauce sugar based?” The answer was almost always yes, and I never ordered. After enough of these, it got hard to believe that every restaurant used sugar-based sauce. So I tried asking with a disgusted face, and the answers flipped to a definitive no. Same restaurants, same sauce, different face.

Then I switched the question: “Is your BBQ sauce vinegar based?” Now the answer was almost always yes.

The problem was that my questions leaked what I was hoping for. Servers are, reasonably, trying to keep a customer happy and ordering. Whichever way my face and voice leaned, the pleasing answer was the easy answer. So I changed the question again, this time with a flat poker face: “Is your sauce sugar based or vinegar based?”

That does two things. An either/or question can’t be answered with a reflexive yes or no, so the server has to think about it. And with no emotional cue to read, there’s no preferred answer to drift toward. The most common reply became “let me ask the chef,” which is exactly the answer when the server doesn’t know. I’ve used it ever since.

It also reminded me of a survey we ran with friends at my undergrad alma mater, as the execs of the student scientific club’s math and CS branch. The survey covered everything at the department level that affected a student’s academic life. We spent hours predicting every possible answer to each question, rewording anything that could lead the respondent, and designing follow-up questions for each plausible interpretation of the results, so we’d have the evidence for a complete read instead of a single headline number. The results made a lot of noise, and many things changed in the department over the following years.

Now I work with experiments and data, and this is the same failure mode as leading survey questions and biased user interviews. People tend to give the answer they think you want. Agents do it too, and so do the people they talk to. When an LLM agent asks a user a clarifying question, the same rules apply:

• Offer the real options, not a confirmation. Replace “Is it X?” with “Is it X or Y?” A question that can be answered with a reflexive yes will be.
• Cover the whole answer space. Include “other” and “not sure,” so nobody is forced into a wrong pick or a guess.
• Hide your preference. Keep your preferred answer out of the wording, the order, and the tone, and don’t explain why you’re asking if the reason reveals what you hope to hear.
• Don’t presuppose. “What would you change?” assumes something needs changing, even with “if anything” attached. Open-ended doesn’t mean neutral: name “nothing needs to change” as an option, and ask for details only after the person picks.

The skill has a few more (one question at a time, balanced option order, a cheap way to say “I don’t know”), plus before/after examples, a short process, and a checklist. This is an early version and I’ll add to it as I collect my thoughts and experiences. Give it a try: the skill on GitHub.

If you try it, I’d like to hear where it fails, especially for questions where the neutral version feels awkward or too long, or leading in unexpected ways.