Activation Functions Explained

An activation function decides whether a neuron's output should be passed forward to the next layer, and by how much, based on the value it receives.

Without activation functions, a neural network would only be able to learn simple, straight line relationships, no matter how many layers it had, since combining linear operations still produces a linear result.

Here are some of the most commonly used activation functions in neural networks.

Sigmoid Function

output = 1 / (1 + e^(-x))

Sigmoid squashes any input into a value between 0 and 1, which makes it useful for problems where the output represents a probability, like binary classification.

ReLU (Rectified Linear Unit)

output = max(0, x)

ReLU passes positive values through unchanged, but converts any negative value to zero. It helps networks train faster and is one of the most commonly used activation functions in hidden layers.

Tanh Function

output = (e^x - e^(-x)) / (e^x + e^(-x))

Tanh squashes input into a range between -1 and 1, centering outputs around zero, which often helps the model learn more efficiently than sigmoid in hidden layers.

Softmax Function

output_i = e^(x_i) / sum(e^(x_j)) for all j

Softmax converts a set of raw output values into probabilities that sum to 1, making it especially useful in the final layer of multi-class classification problems.