OpinionPath » 2009 » April

Reading on integrating experimental and observational research design

Chris Stetson — Wed, 22 Apr 2009 03:02:02 +0000

Because people stumble into the career of market research from so many other fields, many books on research and survey design need to be written at a basic level.

The books listed below, though, are different. Many leap to a level of mental gymnastics that some of us haven’t experienced since college. But the exercise can be good for you! Even if you initially only understand 10% of it, that 10% can set you free to rise above the mundane.

Unfortunately, since I haven’t reading these books myself, I can’t yet give you a full mapping yet of which fit together, which are too tough, and which might be skipped over.

Leslie Kish, Survey Sampling (1965) [Classic text for dealing with imperfect sampling (non-response, lack of coverage) and complex sampling (multi-stage and multi-level)]
Kerlinger and Lee, Foundations of Behavioral Research (1999) [Comprehensive text that reaches from the past almost into the present]
Shadish, Cook, and Campbell, Experimental and Quasi-Experimental Design for Generalized Causal Inference (2001) [The best book, and the one that new methods aspire to beat]
Paul R. Rosenbaum, Observational Studies (2002) [The new generation: propensity-scoring...]
Donald B. Rubin, Matched Sampling for Causal Effects (2006) [Key thinker of new generation]
Judea Pearl, Causality: Models, Reasoning and Interference (2000, 2009) [Coming out of the field of computer science, Pearl writes almost as a mathematical philosopher, challenging both the new generation and the old, and giving you a new understanding of structural equation modeling]

Merging Surveys and the Experimental Method

Merging Surveys and the Experimental Method

Chris Stetson — Wed, 22 Apr 2009 02:35:57 +0000

Designing market research to learn what we want from participants is already hard enough. Why do we have compound the challenge by introducing “advanced methods” or “analytics”? Finish writing one report on a survey with the feeling “we still don’t really know which actions to take,” and you’ll sense the answer. Survey research does a terrific job of gathering descriptive information but, even with regression and correlation, it has a hard time confidently revealing what causes what.

Faced with this apparent defect of surveys, it’s tempting to retreat from the survey to a series of one-off “marketing experiments.” Let’s try this home page. No, let’s try that. How about this change? But even if one of these marketing experiments gets lucky and leads to a jump in some performance metric, how do you generalize the result to other pages? You are left with the nagging feeling that you still don’t know enough about the “why we succeeded” to apply the learnings elsewhere. Suddenly the survey doesn’t look so bad because, by randomly sampling a target population or set of market choices, at least its results are generalizable.

The solution, as others have argued, is neither the descriptive survey nor a series of simple experiments, but research design that permits surveys to behave like generalizable experiments.

New Media and Content-Marketing Collide

Chris Stetson — Tue, 14 Apr 2009 16:38:28 +0000

An increasing share of content on technology-media sites is either produced by technology vendors (e.g., white papers) or produced at their behest (e.g., some webinars and mini-sites).

Some of this “vendor content” (as distinct from user-generated content) is excellent. Some is not.

But the question is, why is so much of it appearing on media sites rather than just on the corporate sites of the vendors? There are many possible explanations but, after some careful examination, most of those explanations seem peripheral to the deeper causes.

The link below points to my presentation on the topic last year.

http://www.slideshare.net/cstetson/new-blends-of-advertising-and-content-in-the-web-age

The deck closes with a discussion of how editors and journalists might develop strategies to respond to these changes.

Market Research Groups on LinkedIn

Chris Stetson — Thu, 09 Apr 2009 13:39:13 +0000

Members of LinkedIn have created a slew of “Groups” (moderated bulletin boards). Unfortunately, the marketing and market-research groups with the largest memberships seem to attract the most entries from those just trying to win business rather than to raise substantive issues or share ideas.

So far, I’ve found three LinkedIn groups to be potentially useful: Consumer Insights, Next Gen Market Research, and Marketing Science.

Consumer Insights has, well, some geniunely insightful people. I am especially pleased to see the interest not just in anthropological approaches, but in behavioral economics.

Next Gen seems to have strong enough management to fend off the promotional entries that clutter other larger groups. If it keeps up the good work, it could become the strongest.

Marketing Science could get into research-design analytics better than the other marketing and research groups on LinkedIn. (The “Marketing ROI and Effectiveness” group could be good, but seems to attract too much clutter. The “Marketing Experiments” group has potential, but its tight connection to MarketingExperiments.com may somehow hinder flourishing discussion.)

Outside of LinkedIn, the “Marketing Research Roundtable” (forum.researchinfo.com) often contains excellent professional thinking and expertise, especially on the statistical end of things.

Market segmentation for the Web