A Random Forest is a particular instance of ensemble learning where individual models are constructed using Decision Trees. This ensemble of Decision Trees is then used to predict the output value. We use a random subset of training data to construct each Decision Tree. This will ensure diversity among various decision trees. One of the best things about Random Forests is that they do not overfit, a problem we encounter frequently in machine learning.
By constructing a diverse set of Decision Trees using various random subsets, we ensure that the model does not over fit the training data. During the construction of the tree, the nodes are split successively and the best thresholds are chosen to reduce the entropy at each level. This split doesn't consider all the features in the input dataset. Instead, it chooses the best split among the random subset of the features that is under consideration. Adding this randomness tends to increase the bias of the random forest, but the variance decreases because of averaging. Hence, we end up with a robust model.
Extremely Random Forests take randomness to the next level. Along with taking a random subset of features, the thresholds are chosen at random too. These randomly generated thresholds are chosen as the splitting rules, which reduce the variance of the model even further. Hence the decision boundaries obtained using Extremely Random Forests tend to be smoother than the ones obtained using Random Forests.
In the following program we'll see how to build a classifier based on Random Forests and Extremely Random Forests. The process to construct both classifiers is very similar, so we will use an input flag to specify which classifier needs to be built. See the code below:
import argparse
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import classification_report
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from utilities import visualize_classifier
# Argument parser
def build_arg_parser():
parser = argparse.ArgumentParser(description='Classify data using \
Ensemble Learning techniques')
parser.add_argument('--classifier-type', dest='classifier_type',
required=True, choices=['rf', 'erf'], help="Type of classifier \
to use; can be either 'rf' or 'erf'")
return parser
if __name__=='__main__':
# Parse the input arguments
args = build_arg_parser().parse_args()
classifier_type = args.classifier_type
# Load input data
input_file = 'data_random_forests.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
# Separate input data into three classes based on labels
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
class_2 = np.array(X[y==2])
# Visualize input data
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='s')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.scatter(class_2[:, 0], class_2[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='^')
plt.title('Input data')
# Split data into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=5)
# Ensemble Learning classifier
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
if classifier_type == 'rf':
classifier = RandomForestClassifier(**params)
else:
classifier = ExtraTreesClassifier(**params)
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
# Evaluate classifier performance
class_names = ['Class-0', 'Class-1', 'Class-2']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train), target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")
print(classification_report(y_test, y_test_pred, target_names=class_names))
print("#"*40 + "\n")
plt.show()
Our program starts with importing the required packages. Next we define an argument parser for Python so that we can take the classifier type as an input parameter. Depending on this parameter, we can construct a Random Forest classifier or an Extremely Random forest classifier:
def build_arg_parser():
parser = argparse.ArgumentParser(description='Classify data using \
Ensemble Learning techniques')
parser.add_argument('--classifier-type', dest='classifier_type',
required=True, choices=['rf', 'erf'], help="Type of classifier \
to use; can be either 'rf' or 'erf'")
return parser
The next step is to define the main function and parse the input arguments:
if __name__=='__main__':
# Parse the input arguments
args = build_arg_parser().parse_args()
classifier_type = args.classifier_type
In the program I am using the data from the data_random_forests.txt file but you may chose your data source file. Each line in this file contains comma-separated values. The first two values correspond to the input data and the last value corresponds to the target label. We have three distinct classes in this dataset. Let's load the data from that file:
input_file = 'data_random_forests.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
Next we separate the input data into three classes:
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
class_2 = np.array(X[y==2])
Now we visualize the input data:
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='s')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.scatter(class_2[:, 0], class_2[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='^')
plt.title('Input data')
Next step is to split the data into training and testing datasets:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=5)
We then define the parameters to be used when we construct the classifier. The n_estimators parameter refers to the number of trees that will be constructed. The max_depth parameter refers to the maximum number of levels in each tree. The random_state parameter refers to the seed value of the random number generator needed to initialize the random forest classifier algorithm:
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
Now depending on the input parameter, we either construct a random forest classifier or an extremely random forest classifier:
if classifier_type == 'rf':
classifier = RandomForestClassifier(**params)
else:
classifier = ExtraTreesClassifier(**params)
After the classifier is constructed we train and visualize it:
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
Next we compute the output based on the test dataset and visualize it:
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
Finally we evaluate the performance of the classifier by printing the classification report:
class_names = ['Class-0', 'Class-1', 'Class-2']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train),
target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")
print(classification_report(y_test, y_test_pred,
target_names=class_names))
print("#"*40 + "\n")
When we run the code with the Random Forest classifier using the rf flag in the input argument, we will see a few figures pop up. The first screenshot is the input data.
Run the following command on your Terminal:
>>> random_forests.py --classifier-type rf
The first screenshot is the input data as shown below:
In the preceding screenshot, the three classes are being represented by squares, circles, and triangles. We see that there is a lot of overlap between classes, but that should be fine for now. The second screenshot shows the classifier boundaries:
Now let's run the code with the Extremely Random Forest classifier by using the erf flag in the input argument. Run the following command on your Terminal:
>>> random_forests.py --classifier-type erf
We will see a few figures pop up. We already know what the input data looks like. The second screenshot shows the classifier boundaries:
If we compare the preceding screenshot with the boundaries obtained from Random Forest classifier, we see that these boundaries are smoother. The reason is that Extremely Random Forests have more freedom during the training process to come up with good Decision Trees, hence they usually produce better boundaries.
Now we'll estimate the confidence measure of the predictions. If we observe the outputs obtained on the terminal, we will see that the probabilities are printed for each data point. These probabilities are used to measure the confidence values for each class. Estimating the confidence values is an important task in machine learning. In the same program file, add the following line to define an array of test data points:
test_datapoints = np.array([[5, 5], [3, 6], [6, 4], [7, 2], [4, 4], [5, 2]])
The classifier object has an inbuilt method to compute the confidence measure. Let's classify each point and compute the confidence values as shown in the code below:
print("\nConfidence measure:")
for datapoint in test_datapoints:
probabilities = classifier.predict_proba([datapoint])[0]
predicted_class = 'Class-' + str(np.argmax(probabilities))
print('\nDatapoint:', datapoint)
print('Predicted class:', predicted_class)
Next we visualize the test data points based on classifier boundaries:
visualize_classifier(classifier, test_datapoints, [0]*len(test_datapoints), 'Test datapoints')
Now when we run the code with the rf flag, we get the following output:
The terminal window should contain the following output:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\python>random_forests.py --classifier-type rf
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 0.91 0.86 0.88 221
Class-1 0.84 0.87 0.86 230
Class-2 0.86 0.87 0.86 224
accuracy 0.87 675
macro avg 0.87 0.87 0.87 675
weighted avg 0.87 0.87 0.87 675
########################################
########################################
Classifier performance on test dataset
precision recall f1-score support
Class-0 0.92 0.85 0.88 79
Class-1 0.86 0.84 0.85 70
Class-2 0.84 0.92 0.88 76
accuracy 0.87 225
macro avg 0.87 0.87 0.87 225
weighted avg 0.87 0.87 0.87 225
########################################
Confidence measure:
Datapoint: [5 5]
Predicted class: Class-0
Datapoint: [3 6]
Predicted class: Class-0
Datapoint: [6 4]
Predicted class: Class-1
Datapoint: [7 2]
Predicted class: Class-1
Datapoint: [4 4]
Predicted class: Class-2
Datapoint: [5 2]
Predicted class: Class-2
C:\Users\python>
For each data point, it computes the probability of that point belonging to our three classes. We pick the one with the highest confidence. If we run the code with the erf flag, we will get the following output:
The terminal window will contain the following output:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\python>random_forests.py --classifier-type erf
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 0.89 0.83 0.86 221
Class-1 0.82 0.84 0.83 230
Class-2 0.83 0.86 0.85 224
accuracy 0.85 675
macro avg 0.85 0.85 0.85 675
weighted avg 0.85 0.85 0.85 675
########################################
########################################
Classifier performance on test dataset
precision recall f1-score support
Class-0 0.92 0.85 0.88 79
Class-1 0.84 0.84 0.84 70
Class-2 0.85 0.92 0.89 76
accuracy 0.87 225
macro avg 0.87 0.87 0.87 225
weighted avg 0.87 0.87 0.87 225
########################################
Confidence measure:
Datapoint: [5 5]
Predicted class: Class-0
Datapoint: [3 6]
Predicted class: Class-0
Datapoint: [6 4]
Predicted class: Class-1
Datapoint: [7 2]
Predicted class: Class-1
Datapoint: [4 4]
Predicted class: Class-2
Datapoint: [5 2]
Predicted class: Class-2
C:\Users\python>
We can see that the outputs are consistent with our observations.
By constructing a diverse set of Decision Trees using various random subsets, we ensure that the model does not over fit the training data. During the construction of the tree, the nodes are split successively and the best thresholds are chosen to reduce the entropy at each level. This split doesn't consider all the features in the input dataset. Instead, it chooses the best split among the random subset of the features that is under consideration. Adding this randomness tends to increase the bias of the random forest, but the variance decreases because of averaging. Hence, we end up with a robust model.
Extremely Random Forests take randomness to the next level. Along with taking a random subset of features, the thresholds are chosen at random too. These randomly generated thresholds are chosen as the splitting rules, which reduce the variance of the model even further. Hence the decision boundaries obtained using Extremely Random Forests tend to be smoother than the ones obtained using Random Forests.
In the following program we'll see how to build a classifier based on Random Forests and Extremely Random Forests. The process to construct both classifiers is very similar, so we will use an input flag to specify which classifier needs to be built. See the code below:
import argparse
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import classification_report
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from utilities import visualize_classifier
# Argument parser
def build_arg_parser():
parser = argparse.ArgumentParser(description='Classify data using \
Ensemble Learning techniques')
parser.add_argument('--classifier-type', dest='classifier_type',
required=True, choices=['rf', 'erf'], help="Type of classifier \
to use; can be either 'rf' or 'erf'")
return parser
if __name__=='__main__':
# Parse the input arguments
args = build_arg_parser().parse_args()
classifier_type = args.classifier_type
# Load input data
input_file = 'data_random_forests.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
# Separate input data into three classes based on labels
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
class_2 = np.array(X[y==2])
# Visualize input data
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='s')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.scatter(class_2[:, 0], class_2[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='^')
plt.title('Input data')
# Split data into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=5)
# Ensemble Learning classifier
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
if classifier_type == 'rf':
classifier = RandomForestClassifier(**params)
else:
classifier = ExtraTreesClassifier(**params)
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
# Evaluate classifier performance
class_names = ['Class-0', 'Class-1', 'Class-2']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train), target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")
print(classification_report(y_test, y_test_pred, target_names=class_names))
print("#"*40 + "\n")
plt.show()
Our program starts with importing the required packages. Next we define an argument parser for Python so that we can take the classifier type as an input parameter. Depending on this parameter, we can construct a Random Forest classifier or an Extremely Random forest classifier:
def build_arg_parser():
parser = argparse.ArgumentParser(description='Classify data using \
Ensemble Learning techniques')
parser.add_argument('--classifier-type', dest='classifier_type',
required=True, choices=['rf', 'erf'], help="Type of classifier \
to use; can be either 'rf' or 'erf'")
return parser
The next step is to define the main function and parse the input arguments:
if __name__=='__main__':
# Parse the input arguments
args = build_arg_parser().parse_args()
classifier_type = args.classifier_type
In the program I am using the data from the data_random_forests.txt file but you may chose your data source file. Each line in this file contains comma-separated values. The first two values correspond to the input data and the last value corresponds to the target label. We have three distinct classes in this dataset. Let's load the data from that file:
input_file = 'data_random_forests.txt'
data = np.loadtxt(input_file, delimiter=',')
X, y = data[:, :-1], data[:, -1]
Next we separate the input data into three classes:
class_0 = np.array(X[y==0])
class_1 = np.array(X[y==1])
class_2 = np.array(X[y==2])
Now we visualize the input data:
plt.figure()
plt.scatter(class_0[:, 0], class_0[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='s')
plt.scatter(class_1[:, 0], class_1[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='o')
plt.scatter(class_2[:, 0], class_2[:, 1], s=75, facecolors='white',
edgecolors='black', linewidth=1, marker='^')
plt.title('Input data')
Next step is to split the data into training and testing datasets:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=5)
We then define the parameters to be used when we construct the classifier. The n_estimators parameter refers to the number of trees that will be constructed. The max_depth parameter refers to the maximum number of levels in each tree. The random_state parameter refers to the seed value of the random number generator needed to initialize the random forest classifier algorithm:
params = {'n_estimators': 100, 'max_depth': 4, 'random_state': 0}
Now depending on the input parameter, we either construct a random forest classifier or an extremely random forest classifier:
if classifier_type == 'rf':
classifier = RandomForestClassifier(**params)
else:
classifier = ExtraTreesClassifier(**params)
After the classifier is constructed we train and visualize it:
classifier.fit(X_train, y_train)
visualize_classifier(classifier, X_train, y_train)
Next we compute the output based on the test dataset and visualize it:
y_test_pred = classifier.predict(X_test)
visualize_classifier(classifier, X_test, y_test)
Finally we evaluate the performance of the classifier by printing the classification report:
class_names = ['Class-0', 'Class-1', 'Class-2']
print("\n" + "#"*40)
print("\nClassifier performance on training dataset\n")
print(classification_report(y_train, classifier.predict(X_train),
target_names=class_names))
print("#"*40 + "\n")
print("#"*40)
print("\nClassifier performance on test dataset\n")
print(classification_report(y_test, y_test_pred,
target_names=class_names))
print("#"*40 + "\n")
When we run the code with the Random Forest classifier using the rf flag in the input argument, we will see a few figures pop up. The first screenshot is the input data.
Run the following command on your Terminal:
>>> random_forests.py --classifier-type rf
The first screenshot is the input data as shown below:
In the preceding screenshot, the three classes are being represented by squares, circles, and triangles. We see that there is a lot of overlap between classes, but that should be fine for now. The second screenshot shows the classifier boundaries:
Now let's run the code with the Extremely Random Forest classifier by using the erf flag in the input argument. Run the following command on your Terminal:
>>> random_forests.py --classifier-type erf
We will see a few figures pop up. We already know what the input data looks like. The second screenshot shows the classifier boundaries:
Now we'll estimate the confidence measure of the predictions. If we observe the outputs obtained on the terminal, we will see that the probabilities are printed for each data point. These probabilities are used to measure the confidence values for each class. Estimating the confidence values is an important task in machine learning. In the same program file, add the following line to define an array of test data points:
test_datapoints = np.array([[5, 5], [3, 6], [6, 4], [7, 2], [4, 4], [5, 2]])
The classifier object has an inbuilt method to compute the confidence measure. Let's classify each point and compute the confidence values as shown in the code below:
print("\nConfidence measure:")
for datapoint in test_datapoints:
probabilities = classifier.predict_proba([datapoint])[0]
predicted_class = 'Class-' + str(np.argmax(probabilities))
print('\nDatapoint:', datapoint)
print('Predicted class:', predicted_class)
Next we visualize the test data points based on classifier boundaries:
visualize_classifier(classifier, test_datapoints, [0]*len(test_datapoints), 'Test datapoints')
Now when we run the code with the rf flag, we get the following output:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\python>random_forests.py --classifier-type rf
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 0.91 0.86 0.88 221
Class-1 0.84 0.87 0.86 230
Class-2 0.86 0.87 0.86 224
accuracy 0.87 675
macro avg 0.87 0.87 0.87 675
weighted avg 0.87 0.87 0.87 675
########################################
########################################
Classifier performance on test dataset
precision recall f1-score support
Class-0 0.92 0.85 0.88 79
Class-1 0.86 0.84 0.85 70
Class-2 0.84 0.92 0.88 76
accuracy 0.87 225
macro avg 0.87 0.87 0.87 225
weighted avg 0.87 0.87 0.87 225
########################################
Confidence measure:
Datapoint: [5 5]
Predicted class: Class-0
Datapoint: [3 6]
Predicted class: Class-0
Datapoint: [6 4]
Predicted class: Class-1
Datapoint: [7 2]
Predicted class: Class-1
Datapoint: [4 4]
Predicted class: Class-2
Datapoint: [5 2]
Predicted class: Class-2
C:\Users\python>
For each data point, it computes the probability of that point belonging to our three classes. We pick the one with the highest confidence. If we run the code with the erf flag, we will get the following output:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\python>random_forests.py --classifier-type erf
########################################
Classifier performance on training dataset
precision recall f1-score support
Class-0 0.89 0.83 0.86 221
Class-1 0.82 0.84 0.83 230
Class-2 0.83 0.86 0.85 224
accuracy 0.85 675
macro avg 0.85 0.85 0.85 675
weighted avg 0.85 0.85 0.85 675
########################################
########################################
Classifier performance on test dataset
precision recall f1-score support
Class-0 0.92 0.85 0.88 79
Class-1 0.84 0.84 0.84 70
Class-2 0.85 0.92 0.89 76
accuracy 0.87 225
macro avg 0.87 0.87 0.87 225
weighted avg 0.87 0.87 0.87 225
########################################
Confidence measure:
Datapoint: [5 5]
Predicted class: Class-0
Datapoint: [3 6]
Predicted class: Class-0
Datapoint: [6 4]
Predicted class: Class-1
Datapoint: [7 2]
Predicted class: Class-1
Datapoint: [4 4]
Predicted class: Class-2
Datapoint: [5 2]
Predicted class: Class-2
C:\Users\python>
We can see that the outputs are consistent with our observations.
0 comments:
Post a Comment