Credit Card Fraud Detection and Cost Minimization

Using Calibrated Probabilities to Minimize Cost Associated With Credit Card Fraud

Executive Summary

Historical transaction data was used to predict fraudulent credit card transactions, followed by calibration of predicted probabilities to better approximate real world values. The motivation is to use calibrated probabilities to make classification decisions that minimize the loss incurred by fraudulent purchases.

Analysis: The models used in analysis include Gaussian Naive Bayes (NB), feed forward Neural Networks (NN) and Random Forests (RF). NB was used as a baseline, while the other models were used in actual prediction and calibration. Performance was measured using precision, recall, F1, and precision-recall area under curve (PR-AUC).

Results and Conclusions: For the data used in this analysis, the best RF model shows a better calibrated reliability curve and achieves a higher PR-AUC value — 0.88 vs. 0.78 for the NN. The performance differences between the two models were small when evaluated using risk minimization.

Cost analysis shows that the the choice of model depends on the internal cost in dollars (C_in) of acting on potential fraud. Except for intermediate C_in values (appx. 1.5-5), the RF appears to be the better model, but the NN offers better performance within that range. Using these models, total loss can be reduced by as much as 83%, varying according to C_in.

Recommendations:

• A larger dataset with more fraud instances would enable better analysis and calibration.
• Determine internal cost of acting on predicted fraud.
• Select a model based on the results of cost analysis, using the determined internal cost.

Outline

1. Introduction
2. Questions
3. Data
4. Analysis & Modeling
5. Risk Minimization
6. Results
7. Conclusions
8. Limitations

INTRODUCTION

This report outlines the process of modeling historical transaction data (approximately 285,000 transactions) to predict credit card fraud, using Naive Bayes (NB), neural networks (NN), and random forest classifiers (RF). Modeling was followed by calibration of predicted probabilities, to better approximate real world values. The motivation behind the calibration is to make classification decisions that minimize the loss incurred by fraudulent purchases; in order to compute minimized risk correctly, it is necessary to have predicted probabilities that are as close as possible to real-world values.

QUESTIONS

The questions that motivated this study are:

1. Can fraudulent credit card transactions be predicted from the available data with reasonable performance?
2. How well can fraud be predicted using different models?
3. Will there be a large trade off between precision and recall?
4. How well do the top models minimize cost for different C_in values?
5. What can be done to further minimize cost?

DATA

The dataset used in this analysis was obtained from Kaggle datasets and is structured as follows:

• 284,807 transactions with 492 classified as fraudulent (positive)
  • positives account for only 0.17% of the set (extreme imbalance)
• 31 columns (Time, 28 anonymized numerical features, Amount, and Class)

Time is a generic integer value that conveys little more than the order in which the transactions occur. The anonymized features are mean-centered numerical floats, each denoted by Vi, where i is the feature number (from 1-28). Amount refers to the amount of the transaction in dollars. Class has two possible values, zero or one, where a value of one indicates fraud.

ANALYSIS

Preprocessing

This step was nothing more than splitting the data into three sets: training, validation, and test, followed by standardizing all features in the training set as ~N(0, 1) — then transforming the validation and test sets accordingly. This split was the only method of cross validation used for this project.

Exploratory Analysis

The anonymized features were grouped by class, and individually plotted as scatter and kernel density estimates (see figures 1 and 2), such that each plot contained one feature with both classes for comparison. The motivation behind this was to gain insights for feature selection: features with some separation between class distributions are likely to perform better for classification, while those with little or no separation are likely to add noise to the models. The latter is less of a problem for NN and RF, but eliminating features can cut down on training time. Also, distributions that are highly irregular should typically be excluded from NB models.

Many features are approximately normal, though some look like Skellam distributions and some range from slightly to extremely irregular. Those that would likely be excluded from NB models due to irregularity have a lot of overlap between classes, and will already be removed when selecting by separation of distributions.

Feature Selection

Based on the findings in the last section, only one method of feature selection was used: filter out those with too much overlap between classes. This was done by computing summary statistics for each feature by class, and selecting those where the medians of each class had a minimum separation of max(std class0, std class1). The features selected using this method were V[4, 11, 12, 14].

Resampling

Because the data is extremely unbalanced (only a very small proportion of transactions are labeled as fraud), a few resampling methods were used to help offset this, and these methods were tested for each classifier. The methods used were random under sampling and generative over sampling.

For random under sampling, the set was separated by class, and a random sample of 50,000 was selected from the portion labeled as legitimate transactions. This sample was recombined with the fraudulent transactions to form a complete, resampled training set. Reducing the number of legitimate transactions to below 50,000 begins to cause large decreases in precision, and so smaller values were not used.

The generative oversampling selected only the fraudulent transactions from the original training set, and computed estimated distributions for all features. Fraudulent transactions were then generated by randomly sampling from these estimates, and class labels were added. This synthesized sample was recombined with the legitimate transactions. Four sets were generated using this method: 500, 1,000, 2,000, and 5,000 generated transactions.

Scoring Method

For all models, including NB, performance was measured using precision, recall, and F1. For the final models, precision-recall area under curve (PR-AUC) was also measured. Accuracy was recorded, but because of the imbalance in the data, this value has no importance: predicting everything as being legitimate transactions will get only a fraction of one percent of the predictions wrong, giving a minimum accuracy greater than 99%.

Baseline Models: Naive Bayes

To get a baseline for comparison against the NN and RF models, a Gaussian Naive Bayes classifier was chosen. Using both the raw data (all features) and only the features selected above, NB models were fitted to the original and resampled training sets, and scored on the validation set. The scores are tabulated in table 2 below.

The table above shows that the chosen feature selection method greatly improves performance over the raw data for the NB classifier, and the generative resampling method works well for smaller resampling sizes, but loses precision as the number of samples generated increases. The generative resampling with 1,000 additional samples was chosen as the resampling method to be tested with the NN and RF models.

Model Selection

To select the best models among NN and RF classifiers, the algorithms for each were run numerous times — with different hyper-parameters — on four training sets (raw, feature selected, and both of these resampled), and evaluated using the validation set. This was done with automated processes described in the next few paragraphs, where the details of modeling are discussed for each classifier.

Neural Nets

The program used for constructing, training, and evaluating neural networks can be found on my GitHub. Three network architectures were tested on the raw set: one with no hidden layer, and two with a single hidden layer of 14 and 29 nodes. Two architectures on the feature-selected training set without resampling: one with no hidden layer, and one with a single hidden layer of 4 nodes. The output layer for each was a softmax layer with two nodes that output probabilities for each class.

These models were evaluated using an automated hyper-parameter (hypers) search algorithm built into the program, using a random search approach. This iteratively trained on 150 combinations of hypers for each model, at 40 epochs per combination. Results from printouts and plots showing epoch scores were used to choose final model architecture.

The two best performers used a single hidden layer with with 14 nodes (no feature selection), both resampled (gen 1K) and original sample. The resampled set gave slightly better recall scores on the validation set (0.778 vs. 0.741) at the expense of precision (0.764 vs 0.842), what appears to be an overall reduction in performance for the gen 1K set. However, the no-resample model achieved an approximate 0.03 bump in precision after what appeared to be some over-training. Because of this and a slight preference for higher recall, the final model used the gen 1K resampled data.

The final NN model used the concatenation of the resampled training set and validation set. Performance was tracked against this combined data for fine-tuning, and not the test data (see table 3); scoring predictions on the test set were saved until after tuning.

Random Forests

The scikit-learn random forest classifier was used for these models, using the standard Gini index. Several random forests were evaluated for each training set (no-resample, and gen 1K), using different numbers of estimators (forest size). The models were evaluated by computing PR_AUC. The best model resulted from the no-resmaple set, using 1000 estimators. The difference in scores between the two sets was marginal: a PR-AUC using 1,000 estimators of 0.70 vs. 0.69.

The final RF model was trained on the concatenated version of the original training and validation sets, and scored using precision, recall, and F1. See Results for details.

Calibrating Probabilities

Because the predicted probabilities from different machine learning models are not always true to real-life values, something that is important to have for risk minimization, predicted values should be analyzed and likely calibrated.

Analysis of the probabilities had two parts: comparing the mean predicted value from each model to the proportion of positives in the test set, and visual inspection of reliability curves. The former is done simply by computing the mean of the predicted values from each model, while the latter is beyond the scope of this report. The comparisons can be found in table 3 and figures 4 and 5.

Predicted probabilities were calibrated using Platt’s scaling, which trains a logistic regression (LR) model on probabilities predicted from the training set. Test set predictions from the models of interest (NN and RF) were then used as inputs to the LR models, producing outputs as calibrated probabilities. Outputs from neural networks are typically well calibrated on their own, but this particular dataset poses some challenges in this regard (see Limitations). Mean predicted values (MPV) and reliability diagrams from before and after calibration are shown below. For the reliability diagrams, only 5 bins were used because of the limited number of positives.

The RF shows a notable improvement in the diagrams and MPV. The NN almost appears worse, but the large dip in the diagram is due to a lack of predicted probabilities in that range; the shape is otherwise better aligned with the diagonal, and there was a marked improvement in MPV.

RISK MINIMIZATION

While the trained classifiers can do a good job of correctly predicting fraudulent transactions, they are not perfect. Instead of making a decision based solely on the class with the highest predicted probability, risk minimization uses Bayesian decision making — involving conditional probabilities and the losses associated with possible decisions to minimize total loss. This can be described more completely and efficiently with a mathematical definition of risk and decision making:

$$R(\omega_i | x) = \sum_{j=0}^c\lambda(\omega_i | \omega_j)P(\omega_j | x)$$

Where \(\lambda(\omega_i | \omega_j)\) is the loss incurred from deciding \(\omega_i\) when the true state is \(\omega_j\). More compacty, \(\lambda(\omega_i | \omega_j)\) will be referred to as \(\lambda_{ij}\).

Now, in the case of predicting fraud, we have two class conditional risk: $$R(\omega_0 | x) = \lambda_{00}P(\omega_0 | x) + \lambda_{01}P(\omega_1 | x)$$ $$R(\omega_1 | x) = \lambda_{10}P(\omega_0 | x) + \lambda_{11}P(\omega_1 | x)$$ Decision making: if \(R(\omega_0 | x) < R(\omega_1 | x)\) choose \(\omega_0\), else choose \(\omega_1\).

The conxtual definitions for \(\lambda_{ij}\) are:

\(\lambda_{00}\) = Zero ($): the cost of deciding the transaction is legitimated when it is.
\(\lambda_{01}\) = Amount of transaction ($): the cost of deciding a transaction is legitimate when it is not.
\(\lambda_{10}\) = C_in ($): the cost of deciding the transaction is fraudulent when it is not.
\(\lambda_{11}\) = C_in ($): the cost of deciding the transaction is fraudulent when it is.

The value C_in is some internal (administrative) cost associated with acting on the decision that a transaction is fraudulent. Because the internal cost is both unknown, and may not always be the same, the first goal of cost analysis was to compare the losses of the top two models as C_in varies, relative to the total loss if no fraud detection is used. The second goal was to evaluate the overall predictive performance of the two models under the same varying conditions, but without minimum risk decision making (i.e. making decisions based solely on model predictions).

This analysis was conducted in multiple parts:

1. Using risk minimization
   • with calibration
   • without calibration
2. Using only model predictions
   • with calibration only

For each version listed above, total loss was computed for each model for 100 values of C_in. Using ranges [0, 3) and [0, 20). These values may seem small, and that is primarily because automated systems make contacting a customer about suspected fraud extremely inexpensive.

RESULTS

Test Scores

In pure predictive performance, the RF outperformed the NN when using 1000 estimators. The test scores and are tabulated in table 4, and the PR curves with AUC can be seen in figure 5:

While the two models can achieve the same level of recall, there is a large difference in precision, meaning the NN is generating many false-positives in order to achieve high recall. The performance difference is also reflected in the AUC values. The PR-AUC was unchanged after calibrating probabilities.

Cost Analysis

With and Without Risk Minimization

Two figures are presented for evaluation of the performance of the calibrated models: Figure 6 shows total loss in dollars (left y-axis), and the decision truth ratio (right y-axis) as functions of C_in. The decision/truth ratio is the number of transactions decided to be fraudulent, divided by the number of actual fraudulent transactions — this is a simplified way of knowing how the minimum risk is making decisions as C_in varies. Figure 7 shows a wider range for C_in and compares losses of classifications with and without using minimum risk (Bayesian decisions vs. purely prediction-based). Making decisions based on minimum risk result in smaller losses compared to decisions based purely on model predictions.

A comparison of losses incurred using risk minimization from calibrated vs. uncalibrated probabilities are presented in figures 8 and 9 (below). Making minimum risk decisions without probability calibrations resulted in greater losses and and many more false positives for the range of C_in plotted.

 

CONCLULSIONS

A fraud detection system that uses Bayesian minimum risk can lead to significant savings when compared to using model predictions alone, and it is important to have properly calibrated probabilities when doing so. Both the NN and RF models appear to have very similar loss for most C_in values when using this method, but the RF has slightly lower loss for much of the range. Regardless, choosing a model should be determined after knowing the precise value of C_in, as well as how many false negatives and false positives are acceptable.

LIMITATIONS

The dataset was relatively small with very few instances of fraud, which means the performance of models could have high variance when predicting new data. The small number of positive instances also imposes some limitations on probability calibrations, possibly hindering the performance of risk minimization.