Activation Functions and Network Architecture

What Are Activation Functions?

An activation function determines how a neuron transforms its calculated input before passing the result to the next layer.

Without nonlinear activation functions, a network composed only of ordinary linear transformations would behave like a single linear transformation, regardless of how many layers it contains.

Common activation functions include:

1. ReLU (Rectified Linear Unit)

ReLU returns zero for negative inputs and the input itself for positive inputs.

[
f(x) = \max(0,x)
]

It is commonly used in hidden layers because it is simple and computationally efficient.

2. Sigmoid

The sigmoid function maps values into a range between 0 and 1.

[
f(x) = \frac{1}{1+e^{-x}}
]

It is often used in the output layer for binary classification when the output represents a probability.

3. Softmax

Softmax converts a vector of scores into values that sum to 1. It is commonly used for mutually exclusive multiclass classification.

4. Tanh

Tanh maps values into a range between -1 and 1. It is used in some neural network architectures, including certain recurrent networks.

Choosing an Output Layer

The output layer depends on the problem being solved.

Choosing an appropriate output activation and loss function helps the network learn the intended task.