Sundsøy, P., Bjelland, J., Iqbal, A. M., & de Montjoye, Y. A. (2014, April). Big data-driven marketing: how machine learning outperforms marketers’ gut-feeling. In International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction (pp. 367-374). Springer, Cham.
This paper shows how big data can be experimentally used at large scale for marketing purposes at a mobile network operator. We present results from a large-scale experiment in a MNO in Asia where we use machine learning to segment customers for text-based marketing. This leads to conversion rates far superior to the current best marketing practices within MNOs.
Using metadata and social network analysis, we created new metrics to identify customers that are the most likely to convert into mobile internet users. These metrics fall into three categories: discretionary income, timing, and social learning. Using historical data, a machine learning prediction model is then trained, validated, and used to select a treatment group. Experimental results with 250 000 customers show a 13 times better conversion-rate compared to the control group. The control group is selected using the current best practice marketing. The model also shows very good properties in the longer term, as 98% of the converted customers in the treatment group renew their mobile internet packages after the campaign, compared to 37% in the control group. These results show that data-driven marketing can significantly improve conversion rates over current best-practice marketing strategies.
Identifying the Dataset
This study examines the effectiveness of text-centric ad campaigns by Mobile Network Operators (MNOs) in Asia. Because the reach of texts and mobile service is so widespread, marketing strategy often relies on “gut feeling.” In this paper, the authors hope to provide a big data oriented approach to streamlining and enhancing marketing efforts.
For the big data campaign, the team calculated the total number of texts sent and received per month, the revenue generated by each user, and prioritised prepaid customers. There are a total of 50,000 users in the control group. The dataset considers 350+ metadata features and 40 social features as an impact factor for product adoption (between customers interacting more than 3 times a month). The training dataset consisted of natural converters (non-mobile) totalling to 50,000 natural users and 100,000 non-internet users (this includes people who might be interested in using the internet and who would maintain their mobile internet use). Six months of metadata is used to train the models.
The authors note that the characteristics of customers who are likely to adopt mobile internet can only be determined by natural conversion data and rates. They are aware that the model may have received more accurate training if data on previous campaigns was recorded and available. After testing a variety of modeling algorithms (SVM, Neural networks), the authors landed on a bootstrap aggregated decision tree where performance evaluation is measured via stability and accuracy. The final model included a minority, approximately 20, of the previously identified 350+ metadata features.
The treatment group chosen for the experiment consists of the top 20% of the treatment group - the top 200,000 customers which is representative of 1% of the customer base. The experiment plays out as follows: a text offering a 15 MB internet plan over 15 days for half price is sent. The customer must activate the offer by sending a text to a number specified in the text itself. According to the model created by the authors, the conversion rate was 6.42% whereas the normal MNO approach had a conversation of 0.5%. Similar to the stark difference in initial conversion rates, the renewal rate is also significantly higher with the data-driven model, 37% vs. 98% renewal rates.
Key Assumptions Stated by Authors
Reflecting on the chosen features which informed their model, the authors recognise that discretionary income, timing, and social learning served as key contributing factors. In terms of spending, rather than looking at overall income, the authors claim that understanding total spending on mobile services is a more helpful indicator. In terms of social factors, the total spending on data by one’s neighbour is also predictive of one’s individual spending - otherwise known as network externality effects.
In conducting this research, the authors realise the necessity of considering privacy concerns, especially regarding sensitive data.
This paper confines privacy concerns to data storage - even then, the data is not anonymised and is made available to the marketing team of a company. Moreover, simply focusing on conversion and renewal rates sets aside considerations regarding the quality of services and simplifies customer satisfaction.
The results of this experiment are evaluated against MNO’s current ad campaign practices and will compare the conversion rates of groups receiving texts that follow normal MNO practices, and groups receiving data-informed texts. Experiments show, according to the authors, that data-driven approaches yield higher conversion rates than MNOs normal marketing strategy. Moreover, the authors show that in cases where big data isn’t available, historical consumer uptake/adoption data is sufficient. Unlike many other papers that focus on Global North contexts, this takes the Asian context into account while focusing on a less affluent portion of the population. That said, the particularities and vulnerabilities of populations that are studied in these models are not fully considered.