Predicting and Understanding Consumer Behaviour
Sometimes big data approaches are effective; other times not so much. One challenge is that big data models are usually inscrutable (i.e., black boxes). One solution is combine big data with existing theory. In this talk, I marry theory from the behavioural sciences concerning how people make decisions with big data to understand how consumers make choices in the supermarket. The approach successfully predicts when a shopper is open to switching to a new product (e.g., Ariel vs. Fairy). In the second part of the talk, I consider what makes two products similar, modelling people's judgments using a psychologically informed LDA approach. In both examples, practical lessons are highlighted, such as the need for permutation testing and to understand the basic properties of models to ensure convergence. The basic ideas from this talk should apply to other domains.