What are Neural Networks?

Before you write a single Keras line for this intermediate level, you need to understand what a neural network actually is, not just as code, but as an idea. The good news is that the core concept is beautifully simple once you strip away the jargon.

What is a Neural Network?

A neural network is a computational system loosely inspired by the human brain. It is made up of layers of interconnected nodes called neurons. Each connection carries a weight, a number that controls how strongly one neuron influences another. During training, the network adjusts these weights to produce the correct output for a given input.

Think of it like learning to ride a bicycle. The first attempt is a disaster. With every fall, your brain adjusts, subtly shifting your balance intuition. A neural network does the same thing mathematically: it makes a prediction, measures how wrong it was (the loss), and nudges its weights in the direction that reduces the error.

Biological vs Artificial Neurons

The biological neuron receives electrical signals from other neurons through dendrites, processes them in the cell body, and if the combined signal is strong enough, fires an output signal down the axon to the next neuron. The artificial neuron mimics this idea with numbers:

• Inputs (like dendrites): each incoming value x is multiplied by a weight w.
• Summation (like the cell body): all weighted inputs are added together, along with a bias term b.
• Activation (like the firing threshold): a non-linear function is applied to the sum; if the result exceeds a threshold, the neuron 'fires'.

The key difference is that biological neurons are analogue, massively parallel, and energy-efficient in ways we still do not fully understand. Artificial neurons are discrete, mathematical, and need significant compute but they are far more flexible and programmable.

Components of a Neural Network

Every neural network, regardless of size, contains the same core building blocks:

• Neurons: the basic computational units. Each neuron computes a weighted sum of its inputs and passes it through an activation function.

• Weights: learnable parameters that determine the strength of connections. These are the knobs training turns.

• Biases: additional learnable parameters added to the weighted sum, allowing neurons to shift their activation threshold.

• Activation functions: non-linear transformations applied at each neuron. Without them, stacking layers would be equivalent to a single linear transformation — useless for learning complex patterns.

• Loss function: measures how far the network's predictions are from the correct answers. Training tries to minimise this.

• Optimiser: the algorithm (e.g., Adam, SGD) that updates weights using the gradient of the loss.

Activation Functions (ReLU, Sigmoid, Softmax)

Activation functions introduce the non-linearity that makes deep networks powerful. Without them, no matter how many layers you stack, the network reduces to a single linear equation. Here are the three you will use most often:

Did You Know?

ReLU (Rectified Linear Unit) was a breakthrough when Hinton et al. showed it dramatically outperformed Sigmoid and Tanh for deep networks by avoiding the vanishing gradient problem, where gradients become so tiny in early layers that weights barely update at all.

Forward Propagation and Backpropagation

Training a neural network is a two-step loop repeated for many iterations:

• Forward propagation: data flows from the input layer through each hidden layer to the output, producing a prediction. At each layer, the weighted sum is computed and the activation function applied.

• Loss calculation: the prediction is compared to the true label using the loss function (e.g., cross-entropy for classification). The result is a single number — how wrong the network was.

• Backpropagation: the loss is propagated backwards through the network using the chain rule of calculus. Each weight receives a gradient — the rate at which the loss changes when that weight changes slightly.

• Weight update: the optimiser uses these gradients to nudge every weight in the direction that reduces the loss.

# This is what Keras does internally when you call model.fit()

# Understanding it manually helps when you need custom training loops

with tf.GradientTape() as tape:

predictions = model(x_batch, training=True)     # forward pass

loss = loss_fn(y_batch, predictions)            # compute loss

gradients = tape.gradient(loss, model.trainable_variables)  # backprop

optimizer.apply_gradients(zip(gradients, model.trainable_variables))  # update