Menu

Deep Learning Architectures

Deep learning architectures are designed to process different types of information. Images contain spatial patterns, text contains contextual relationships, and time-series data contains ordered observations.

Different neural network architectures provide mathematical structures suited to these characteristics.

CNNs, RNNs, LSTMs, and Transformers are neural network architectures that process information using convolution, recurrence, gated memory, or attention mechanisms.

Understanding their differences helps data scientists select suitable approaches for computer vision, language processing, and sequential prediction.

What Is a Convolutional Neural Network?

A Convolutional Neural Network (CNN) uses convolutional operations to learn spatial patterns.

CNNs are widely used for image classification, object detection, and image segmentation.

For example, a manufacturing company may use a CNN to identify visible defects in product images.

How Does a CNN Work?

A basic CNN may contain:

  1. Convolutional layers
  2. Activation functions
  3. Pooling layers
  4. Dense or task-specific output layers

Convolutional Layer

A convolutional layer applies learned filters to local regions of an input.

These filters can detect patterns such as edges, textures, or more complex visual features.

A simplified two-dimensional convolution-like calculation can be written as:

\[ Y(i,j)=\sum_m\sum_n X(i+m,j+n)K(m,n) \]

Here:

  • \(X\) represents the input.
  • \(K\) represents the filter.
  • \(Y\) represents the resulting feature map.

In many deep learning implementations, the operation is technically cross-correlation because the kernel is not reversed.

Pooling Layer

Pooling reduces spatial dimensions.

Max pooling retains the largest value within each pooling region.

Average pooling calculates the mean.

Pooling can reduce computation and introduce some tolerance to small spatial changes.

CNN Example Using Keras

from tensorflow import kerasmodel = keras.Sequential([    keras.layers.Input(shape=(28, 28, 1)),    keras.layers.Conv2D(        32,        kernel_size=3,        activation="relu"    ),    keras.layers.MaxPooling2D(pool_size=2),    keras.layers.Flatten(),    keras.layers.Dense(64, activation="relu"),    keras.layers.Dense(10, activation="softmax")])model.summary()

The model accepts grayscale images of size 28 × 28.

The final layer produces probabilities for ten mutually exclusive classes.

What Is a Recurrent Neural Network?

A Recurrent Neural Network (RNN) processes sequential information by maintaining a hidden state.

The hidden state is updated as the model receives new elements in a sequence.

A simplified RNN update is:

\[ h_t=f(W_xx_t+W_hh_{t-1}+b) \]

Here:

  • \(x_t\) represents the input at time \(t\).
  • \(h_{t-1}\) represents the previous hidden state.
  • \(h_t\) represents the updated hidden state.

RNNs can process sequences such as text, sensor readings, and time-series observations.

However, basic RNNs can struggle to retain information across long sequences.

What Is an LSTM?

Long Short-Term Memory (LSTM) is a recurrent architecture designed to manage information across longer sequences.

It uses gating mechanisms to control information flow.

Its main components include:

  • Forget gate
  • Input gate
  • Cell state
  • Output gate

The gates determine which information should be retained, updated, or exposed.

LSTMs can help address some long-range dependency problems associated with basic RNNs.

LSTM Example

from tensorflow import kerasmodel = keras.Sequential([    keras.layers.Input(shape=(30, 5)),    keras.layers.LSTM(64),    keras.layers.Dense(1)])model.summary()

The input shape (30, 5) represents sequences containing 30 time steps and five features per step.

The model produces one numerical output.

For example, it could be adapted to forecast a future numerical measurement, provided the target and training setup are appropriate.

What Is a GRU?

Gated Recurrent Unit (GRU) is another gated recurrent architecture.

It uses update and reset gates to control information flow.

GRUs often contain fewer parameters than comparable LSTM layers.

They can be useful for sequential modelling, although performance depends on the task and model configuration.

What Is a Transformer?

A Transformer is a neural network architecture that uses attention mechanisms to model relationships between elements.

Transformers form the foundation of many modern language models and are also used in vision, audio, and multimodal applications.

Their defining mechanism is attention rather than traditional recurrence.

What Is Self-Attention?

Self-attention allows each element in a sequence to use information from other elements in the same sequence.

For example, the meaning of a word may depend on words appearing earlier or later in a sentence.

Scaled dot-product attention is expressed as:

\[ Attention(Q,K,V)= Softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V \]

Here:

  • \(Q\) represents queries.
  • \(K\) represents keys.
  • \(V\) represents values.
  • \(d_k\) represents the key dimension.

The attention scores determine how strongly different sequence elements contribute to the output.

Why Do Transformers Use Positional Information?

Self-attention alone does not inherently encode the ordering of sequence elements.

Transformers therefore incorporate positional information through mechanisms such as positional embeddings or positional encodings.

This allows the model to represent sequence order.

CNN vs RNN vs LSTM vs Transformer

Architecture Primary Mechanism Common Application
CNN Convolutional filters Image recognition
RNN Recurrent hidden state Sequential data
LSTM Gated recurrent memory Time-series and sequential modelling
GRU Simplified gated recurrence Sequential modelling
Transformer Attention mechanisms Language, vision, and multimodal models

Architecture selection depends on data structure, model requirements, and available resources.

Transformers are widely used, but CNNs and recurrent models remain useful in many practical systems.

Where Are These Architectures Used?

Computer Vision

CNNs and vision transformers support image classification, defect detection, and medical image analysis.

Natural Language Processing

Transformers support text classification, summarisation, translation, and language generation.

Time-Series Forecasting

LSTMs, GRUs, Transformers, and other specialised architectures can model sequential observations.

Speech Processing

Neural architectures transform audio signals into representations used for speech recognition and other audio tasks.

Recommendation Systems

Neural models can learn user and item representations from interaction data.

Common Deep Learning Architecture Mistakes

  1. Assuming one architecture is best for every data type.
  2. Using very complex models for small datasets without justification.
  3. Ignoring sequence length and input shape.
  4. Treating attention weights as definitive causal explanations.
  5. Confusing architectural capability with guaranteed predictive performance.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details