Designing market research to learn what we want from participants is already hard enough. Why do we have compound the challenge by introducing “advanced methods” or “analytics”? Finish writing one report on a survey with the feeling “we still don’t really know which actions to take,” and you’ll sense the answer. Survey research does a terrific job of gathering descriptive information but, even with regression and correlation, it has a hard time confidently revealing what causes what.

Faced with this apparent defect of surveys, it’s tempting to retreat from the survey to a series of one-off “marketing experiments.” Let’s try this home page. No, let’s try that. How about this change? But even if one of these marketing experiments gets lucky and leads to a jump in some performance metric, how do you generalize the result to other pages? You are left with the nagging feeling that you still don’t know enough about the “why we succeeded” to apply the learnings elsewhere. Suddenly the survey doesn’t look so bad because, by randomly sampling a target population or set of market choices, at least its results are generalizable.

The solution, as others have argued, is neither the descriptive survey nor a series of simple experiments, but research design that permits surveys to behave like generalizable experiments.