A classifier is only as good as the data that's used for training. One of the most common problems we face in the real world is the quality of data. For a classifier to perform well, it needs to see equal number of points for each class. But when we collect data in the real world, it's not always possible to ensure that each class has the exact same number of data points. If one class has 10 times the number of data points of the other class, then the classifier tends to get biased towards the first class. Hence we need to make sure that we account for this imbalance algorithmically. The following program shows how to do that:
import sys
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from utilities import visualize_classifier
# Load input data
input_file = 'data_imbalance.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
# Separate input data into two classes based on labels
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
# Visualize input data
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='black',
edgecolors='black', linewidth=1, marker='x')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.title('Input data')
# Split data into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=5)
# Extremely Random Forests classifier
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
if len(sys.argv) > 1:
if sys.argv[1] == 'balance':
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0, 'class_weight': 'balanced'}
else:
raise TypeError("Invalid input argument; should be 'balance'")
classifier = ExtraTreesClassifier(**params)
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
# Evaluate classifier performance
class_names = ['Class-0', 'Class-1']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train), target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")
print(classification_report(y_test, y_test_pred, target_names=class_names))
print("#"*40 + "\n")
plt.show()
Our program begins with importing all the required packages. I've used the data in the file data_imbalance.txt for our analysis, you may replace this with your data file. First we load the data. Each line in this file contains comma-separated values. The first two values correspond to the input data and the last value corresponds to the target label. We have two classes in this dataset. Let's load the data from that file:
input_file = 'data_imbalance.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
Then we separate the input data into two classes:
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
Next we visualize the input data using scatter plot:
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='black',
edgecolors='black', linewidth=1, marker='x')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.title('Input data')
Now we split the data into training and testing datasets:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=5)
Next, we define the parameters for the Extremely Random Forest classifier. Note that there is an input parameter called balance that controls whether or not we want to algorithmically account for class imbalance. If so, then we need to add another parameter called class_weight that tells the classifier that it should balance the weight, so that it's proportional to the number of data points in each class:
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
if len(sys.argv) > 1:
if sys.argv[1] == 'balance':
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0, 'class_weight': 'balanced'}
else:
raise TypeError("Invalid input argument; should be 'balance'")
Now build, train, and visualize the classifier using training data:
classifier = ExtraTreesClassifier(**params)
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
Then we predict the output for test dataset and visualize the output:
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
Finally we compute the performance of the classifier and print the classification report:
class_names = ['Class-0', 'Class-1']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train), target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")print(classification_report(y_test, y_test_pred, target_names=class_names))
print("#"*40 + "\n")
plt.show()
When we run the code, we will see a few screenshots as output. The first screenshot shows the input data:
The second screenshot shows the classifier boundary for the test dataset:
The preceding screenshot indicates that the boundary was not able to capture the actual boundary between the two classes. The black patch near the top represents the boundary. We see the following output on our Terminal:
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 1.00 0.01 0.01 181
Class-1 0.84 1.00 0.91 944
accuracy 0.84 1125
macro avg 0.92 0.50 0.46 1125
weighted avg 0.87 0.84 0.77 1125
########################################
########################################
Classifier performance on test dataset
C:\Users\python\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn
\metrics\classification.py:1437: UndefinedMetricWarning: Precision and F-score a
re ill-defined and being set to 0.0 in labels with no predicted samples.
'precision', 'predicted', average, warn_for)
precision recall f1-score support
Class-0 0.00 0.00 0.00 69
Class-1 0.82 1.00 0.90 306
accuracy 0.82 375
macro avg 0.41 0.50 0.45 375
weighted avg 0.67 0.82 0.73 375
########################################
------------------
(program exited with code: 0)
Press any key to continue . . .
We see a warning because the values are 0 in the first row, which leads to a divide-by-zero error (ZeroDivisionError exception) when we compute the f1-score. Run the code on the terminal using the ignore flag so that you do not see the divide-by-zero warning:
>>> -W ignore class_imbalance.py
Now if we want to account for class imbalance, run it with the balance flag:
>>>class_imbalance.py balance
Now the classifier output should look like:
The output terminal should look like :
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\python>class_imbalance.py balance
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 0.44 0.93 0.60 181
Class-1 0.98 0.77 0.86 944
accuracy 0.80 1125
macro avg 0.71 0.85 0.73 1125
weighted avg 0.89 0.80 0.82 1125
########################################
########################################
Classifier performance on test dataset
precision recall f1-score support
Class-0 0.45 0.94 0.61 69
Class-1 0.98 0.74 0.84 306
accuracy 0.78 375
macro avg 0.72 0.84 0.73 375
weighted avg 0.88 0.78 0.80 375
########################################
C:\Users\python>
By accounting for the class imbalance, we were able to classify the data points in class-0 with non-zero accuracy.
import sys
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from utilities import visualize_classifier
# Load input data
input_file = 'data_imbalance.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
# Separate input data into two classes based on labels
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
# Visualize input data
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='black',
edgecolors='black', linewidth=1, marker='x')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.title('Input data')
# Split data into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=5)
# Extremely Random Forests classifier
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
if len(sys.argv) > 1:
if sys.argv[1] == 'balance':
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0, 'class_weight': 'balanced'}
else:
raise TypeError("Invalid input argument; should be 'balance'")
classifier = ExtraTreesClassifier(**params)
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
# Evaluate classifier performance
class_names = ['Class-0', 'Class-1']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train), target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")
print(classification_report(y_test, y_test_pred, target_names=class_names))
print("#"*40 + "\n")
plt.show()
Our program begins with importing all the required packages. I've used the data in the file data_imbalance.txt for our analysis, you may replace this with your data file. First we load the data. Each line in this file contains comma-separated values. The first two values correspond to the input data and the last value corresponds to the target label. We have two classes in this dataset. Let's load the data from that file:
input_file = 'data_imbalance.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
Then we separate the input data into two classes:
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
Next we visualize the input data using scatter plot:
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='black',
edgecolors='black', linewidth=1, marker='x')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.title('Input data')
Now we split the data into training and testing datasets:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=5)
Next, we define the parameters for the Extremely Random Forest classifier. Note that there is an input parameter called balance that controls whether or not we want to algorithmically account for class imbalance. If so, then we need to add another parameter called class_weight that tells the classifier that it should balance the weight, so that it's proportional to the number of data points in each class:
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
if len(sys.argv) > 1:
if sys.argv[1] == 'balance':
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0, 'class_weight': 'balanced'}
else:
raise TypeError("Invalid input argument; should be 'balance'")
Now build, train, and visualize the classifier using training data:
classifier = ExtraTreesClassifier(**params)
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
Then we predict the output for test dataset and visualize the output:
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
Finally we compute the performance of the classifier and print the classification report:
class_names = ['Class-0', 'Class-1']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train), target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")print(classification_report(y_test, y_test_pred, target_names=class_names))
print("#"*40 + "\n")
plt.show()
When we run the code, we will see a few screenshots as output. The first screenshot shows the input data:
The second screenshot shows the classifier boundary for the test dataset:
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 1.00 0.01 0.01 181
Class-1 0.84 1.00 0.91 944
accuracy 0.84 1125
macro avg 0.92 0.50 0.46 1125
weighted avg 0.87 0.84 0.77 1125
########################################
########################################
Classifier performance on test dataset
C:\Users\python\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn
\metrics\classification.py:1437: UndefinedMetricWarning: Precision and F-score a
re ill-defined and being set to 0.0 in labels with no predicted samples.
'precision', 'predicted', average, warn_for)
precision recall f1-score support
Class-0 0.00 0.00 0.00 69
Class-1 0.82 1.00 0.90 306
accuracy 0.82 375
macro avg 0.41 0.50 0.45 375
weighted avg 0.67 0.82 0.73 375
########################################
------------------
(program exited with code: 0)
Press any key to continue . . .
We see a warning because the values are 0 in the first row, which leads to a divide-by-zero error (ZeroDivisionError exception) when we compute the f1-score. Run the code on the terminal using the ignore flag so that you do not see the divide-by-zero warning:
Now if we want to account for class imbalance, run it with the balance flag:
>>>class_imbalance.py balance
Now the classifier output should look like:
The output terminal should look like :
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\python>class_imbalance.py balance
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 0.44 0.93 0.60 181
Class-1 0.98 0.77 0.86 944
accuracy 0.80 1125
macro avg 0.71 0.85 0.73 1125
weighted avg 0.89 0.80 0.82 1125
########################################
########################################
Classifier performance on test dataset
precision recall f1-score support
Class-0 0.45 0.94 0.61 69
Class-1 0.98 0.74 0.84 306
accuracy 0.78 375
macro avg 0.72 0.84 0.73 375
weighted avg 0.88 0.78 0.80 375
########################################
C:\Users\python>
By accounting for the class imbalance, we were able to classify the data points in class-0 with non-zero accuracy.
nice article !
ReplyDeleteThanks!
DeleteNice article
ReplyDelete