Decision Trees and Random Forests

What Is a Decision Tree?

A decision tree is a supervised learning algorithm that makes predictions by applying a sequence of conditions to input features.

It consists of:

• Root node: The first decision in the tree.
• Internal nodes: Conditions used to split the data.
• Branches: Outcomes of the conditions.
• Leaf nodes: Final predictions.

For example, a simplified customer churn model might examine contract type, usage frequency, and customer support history to estimate whether a customer may leave.

Decision trees can model nonlinear relationships and interactions between features.

How Does a Decision Tree Learn?

During training, the algorithm evaluates possible splits in the data and selects splits that improve the separation of target values or classes.

For classification, common split criteria include Gini impurity and entropy. For regression, trees can use criteria based on reducing prediction error or variance.

The algorithm continues splitting until a stopping condition is reached, such as a maximum depth or a minimum number of observations in a leaf.

Building a Decision Tree Classifier

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
iris = load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42,
stratify=y
)
model = DecisionTreeClassifier(
max_depth=3,
random_state=42
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))

The max_depth parameter limits how many levels the tree can grow. This can help control model complexity, although the best value depends on the dataset.

What Is a Random Forest?

A random forest combines multiple decision trees to produce a prediction.

It introduces randomness during training, commonly through bootstrap samples of the training data and randomized feature selection at each split.

For classification, predictions are aggregated across trees. For regression, the outputs are commonly averaged.

Using multiple trees can improve predictive performance and reduce the instability associated with an individual decision tree.

Building a Random Forest Classifier

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
model = RandomForestClassifier(
n_estimators=100,
random_state=42
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))

The n_estimators parameter specifies the number of trees in the forest.

Increasing the number of trees can improve stability, but it also increases computation. More trees do not automatically guarantee better generalization.

Decision Tree vs Random Forest

Random forests are useful for many tabular problems, but a well-tuned decision tree or another algorithm may be preferable depending on interpretability, latency, and accuracy requirements.