Recurrent Neural Networks (RNNs)

Recurrent neural networks are designed to handle data where order matters, such as sentences, time series, or audio signals. Unlike feedforward networks or CNNs, which treat each input independently, an RNN has a kind of memory that carries information from one step in a sequence to the next, allowing it to consider context from earlier in the sequence when processing later parts.

This memory comes from the way RNNs are structured internally. At each step, an RNN takes in the current input along with information passed forward from the previous step, combines them, and produces both an output and an updated memory state to carry into the next step. This makes RNNs naturally suited for tasks like language modeling, speech recognition, and stock price prediction, where understanding what came before is essential to making sense of what comes next.  

How RNNs Handle Sequences

Step 1: Process One Element at a Time

An RNN processes a sequence one element at a time, such as one word in a sentence or one time step in a series of stock prices.

Step 2: Use the Hidden State

At each step, the network considers:

• The current input.
• The hidden state, which acts as a running summary of everything the network has seen so far.

Step 3: Update the Hidden State

The hidden state is updated at every step by combining information from the current input with the information already stored from previous steps.

Step 4: Build a Sequence Summary

By the end of the sequence, the hidden state ideally contains a meaningful summary of the entire sequence.

Step 5: Make a Prediction

The final hidden state is used to make a prediction, such as classifying the sentiment of a sentence or predicting the next word.  

Backpropagation Through Time (BPTT)

Backpropagation through time is the technique used to train RNNs, and it works by extending the standard backpropagation process across every time step in a sequence. Since an RNN processes a sequence step by step, the error calculated at the final output needs to be traced backward not just through the network's layers, but through every single time step that contributed to producing that output.

This means that for a long sequence, backpropagation through time involves tracking gradients across many steps, which can quickly become computationally expensive. It also introduces a unique challenge where gradients can either grow extremely large or shrink toward nothing as they are passed backward through many time steps, depending on how the math works out. This second issue, where gradients shrink toward nothing, is significant enough that it has its own name and its own set of solutions, which is covered next.  

The Vanishing Gradient Problem

The vanishing gradient problem occurs when gradients become extremely small as they are passed backward through many layers or time steps during training. When this happens, the weights in the earlier layers or earlier time steps receive such a tiny update that they essentially stop learning altogether, even though the network as a whole is still technically training.

For RNNs specifically, this is a serious limitation, because it means the network effectively loses the ability to learn from information that appeared many steps earlier in a long sequence. In practical terms, a basic RNN trying to understand a long sentence might completely forget important context from the beginning of that sentence by the time it reaches the end. This limitation was a major motivation behind the development of more advanced recurrent architectures, particularly LSTM networks, which were specifically designed to solve this exact problem.