Autoencoders and VAEs

Autoencoders take a different approach compared to everything else here. Instead of learning to classify or predict something, their whole job is to compress data down and then rebuild it again, like learning to summarize something so well you can recreate the full thing from just your summary.

What is an Autoencoder?

An autoencoder takes high-dimensional data, learns an efficient way to encode it, and compresses it down into a smaller representation, often called a latent vector. This compressed version keeps the parts that actually matter and throws out the noise. 

It has two halves. The encoder squeezes the input down into that compact form, and the decoder takes that compact form and tries to rebuild the original as closely as possible. This squeezed middle layer is called the bottleneck, and it forces the encoder to actually be efficient about what it keeps, since there is no room to keep everything.

Variational Autoencoders Explained

A regular autoencoder always squeezes input down into one fixed point. A VAE, short for variational autoencoder, changes that in a clever way.

Instead of outputting one fixed value for each feature, it produces two values, a mean and a standard deviation, which define a range of possibilities instead of a single number. Rather than encoding the input as one fixed point, it then picks a random point from inside that range. 

It matters because it makes the latent space smooth and continuous instead of a bunch of scattered, disconnected points. A VAE connects an encoder network to its decoder through this probabilistic latent space, often shaped like a Gaussian distribution. That smoothness is exactly what lets you generate brand new data afterward, since you can sample from anywhere in that space and get something reasonable back, instead of just reconstructing exact copies of stuff you already saw.