Methodology and Results

Customer churn is referred as the un-subscription of a product or service by customers for a particular brand or company and accepting same product or service from another company. Hence, churning customers are not good for any business as it shows customers are not satisfied with the product or service of the current company and thus are forced to associate with a different brand of same industry. Churning reduces the brand trust in new customers also as they are influenced by the comments and activities of old customers who have stopped using company services or products. Hence, every brand needs to investigate the reason of churn and more importantly target the customers who are likely to churn with attractive offers or other strategies to keep those customers associated with company. This is important as it is relatively easy to maintain existing customers than to acquire new customers in terms cost of business promotion and involvement of company resources. Hence, like any other sector in telecommunication sector it is important to know which customers are churning and predicting who are likely to churn based on analysis for targeted retention planning of customers which will make the business sustainable in the long run (Ahmad, Jafar & Aljoumaa, 2019). Now, analysis of customer can be done manually by inspecting customer data thoroughly and then deciding the critical factors that influences customer churn, however, this method is very time consuming and thus inefficient for analyzing large customer bases of the telecom company. Hence, machine learning solution to automatically detect the customers who are likely to churn are much recommended as the process is fast and when ML algorithm is well trained then good accuracy can be produced (Jain, Khunteta & Srivastava, 2021). In this research using a sample customer churn dataset obtained from Kaggle repository a decision tree model is built which is evaluated on a small section of the dataset and performance of the algorithm is assessed and more over the important attributes that are related with churn are found. The dataset contains a total of 21 attributes of 7043 customers of the telecom industry and last column of the dataset is the Churn column with binary values Yes or No for churning and non-churning customers (“Telco Customer Churn”, 2022). Some attributes are removed from dataset before doing analysis and fitting model which is described in details in the later section. The dataset is saved in an online MongoDB platform and then retrieved from it in google Colab, where the entire pre-processing and model fitting is performed with necessary python coding utilizing Pyspark MLlib, matplotlib, pandas, NumPy, sci-kit learn modules.

Handling of Missing Values

After loading the dataset in Colab as per instruction the attributes MonthlyCharges, OnlineSecurity, StreamingTV, InternetService and Partner are removed and analysis and model fitting is performed with the modified version of data. Additionally, the customer ID is not a feature and assumed to have no influence in determining the class and thus removed from the dataset.

In the pre-processing the missing values are investigated and no missing instances are found. However, in the attributes TotalCharges some invalid values are found which are coerced by NAN by pandas numeric conversion method and then those instances are imputed with mean of valid values of the attribute (Raja and Thangavel, 2020). This reserves all instances of the loaded dataset.

There are three numeric variables in the dataset for which the descriptive statistics are computed as given below.

 

It can be seen that the on an average most of the customer are not senior citizen, the average tenure of customers are about 32 months and average total amount charged to customer is about €2283 (Betancourt and Chen, 2019).

Now, the categorical features are analyzed displaying the frequency of categories in bar charts and it is found that only gender distribution is uniform while the distribution of other categorical features are imbalanced.

 

The class variable churn shows that there are high number of non-churning customers in the dataset that the churning customers in the collected sample. Hence, assuming the sample represents the entire customer base of the telecom company, it can be seen that company is able hold a large proportion of customers and only some customers are churning which are needed to be identified by ML methods.

Now, the categorical features are one-hot encoded for fitting the model and then appended with the numeric features to create a features column containing values of all features and then features class label are extracted to fit the decision tree model (Seger, 2018). However, before that the train-test splitting of the selected features and columns are performed in 80:20 ratio with a specific random state for reproducibility of splitting. The decision is model with maximum depth of 3 is then fitted to the train set and then used for prediction in the test set (Nokeri, 2022).

First 10 rows of prediction along with gender and senior citizen attribute values:

 

Evaluation results in test set:

 

Hence, it can be seen that developed decision tree model correctly predicted more non-churning customers than churning customers and thus accuracy of model is observed to be about 79.38% and f1, recall and precision scores are also enough high (Thambawita et al., 2020). Hence, the results of fitted decision tree model for churn detection can be considered satisfactory.

Statistical Analysis of Attributes

Now, the importance scores of features are gathered from decision tree as given below.

 

It can be seen that contract with month-to-month category, tech support with No category and tenure has some significant positive importance scores and for the rest the scores are close to zero and thus are not important. Now, the most important feature contract has no missing instances and missing value replacement is not required (Raff, Sylvester & Mills, 2018). Now, to find the insights about customers who are churning the churn attribute is plotted against these three in three different plots.

 

It is clearly observed that churning customers are the ones who are in month-to-month contract basis as from the bar plot the count for same are significantly higher than one-year or two-year contract (Pajankar, 2022).

 

Hence, the churning customers are the ones who does not need tech support from company as the count for the same is significantly higher than churn count for customers who does not have internet or those who needs support (Pajankar, 2022).

 

It is observed that most of the customers who are churning have tenure mostly under 30 months whereas significant percentage of non-churning customers have tenure over 30 months which goes over 70 months (Pajankar, 2022). There are very few churning customers who have tenure over 30 months.

Conclusion:

Hence, in conclusion it can be stated that decision tree model is developed using necessary machine learning techniques in python with built-in libraries and model shows good performance to detect churn and non-churning customers based on their historical attributes. The number of missing values in dataset is very small which is observed in total charges attribute only and thus a different imputation or removal is unlikely to improve model performance significantly. However, a different types of pre-processing like normalization of attributes or feature selection by some algorithm like L1 or L2 based SVM feature selector or tree-based feature selection methods can improve the accuracy of model and thus overall churn analysis in a significant way which is left to be investigated as a future scope of this research. The churning customers are mainly found to be under 30 months tenure, requires no technical support and are on month-to-month contract basis and thus telecom company needs to target customers with these attributes and develop suitable strategies (like giving new exciting offers, promotional phone calling) to retain this segment of customers.

References:

Ahmad, A. K., Jafar, A., & Aljoumaa, K. (2019). Customer churn prediction in telecom using machine learning in big data platform. Journal of Big Data, 6(1), 1-24.

Betancourt, R., & Chen, S. (2019). pandas Library. In Python for SAS Users (pp. 65-109). Apress, Berkeley, CA.

Jain, H., Khunteta, A., & Srivastava, S. (2021). Telecom churn prediction and used techniques, datasets and performance measures: a review. Telecommunication Systems, 76(4), 613-630.

Nokeri, T. C. (2022). Tree Modeling and Gradient Boosting with Scikit-Learn, XGBoost, PySpark, and H2O. In Data Science Solutions with Python (pp. 59-74). Apress, Berkeley, CA.

Pajankar, A. (2022). Visualizing Data with Pandas and Matplotlib. In Hands-on Matplotlib (pp. 211-241). Apress, Berkeley, CA.

Raff, E., Sylvester, J., & Mills, S. (2018, December). Fair forests: Regularized tree induction to minimize model bias. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 243-250).

Raja, P. S., & Thangavel, K. J. S. C. (2020). Missing value imputation using unsupervised machine learning techniques. Soft Computing, 24(6), 4361-4392.

Seger, C. (2018). An investigation of categorical variable encoding techniques in machine learning: binary versus one-hot and feature hashing.

Telco Customer Churn. Kaggle.com. (2022). Retrieved 31 March 2022, from https://www.kaggle.com/datasets/blastchar/telco-customer-churn.

Thambawita, V., Jha, D., Hammer, H. L., Johansen, H. D., Johansen, D., Halvorsen, P., & Riegler, M. A. (2020). An extensive study on cross-dataset bias and evaluation metrics interpretation for machine learning applied to gastrointestinal tract abnormality classification. ACM Transactions on Computing for Healthcare, 1(3), 1-29.