The rise of generative AI has introduced a new class of models that are transforming how machines understand and produce content. At the heart of this revolution are foundation models, large, general-purpose AI models trained on vast amounts of data that serve as the base for a wide range of applications.<\/p>\n\n\n\n
From answering questions and writing code to generating photorealistic images, foundation models have become the engine powering the most capable AI systems in the world today.<\/p>\n\n\n\n
But what exactly are foundation models? How do they work? And why have they become so central to modern AI development?<\/p>\n\n\n\n
This guide answers all of those questions, covering what foundation models are, how they are built, what makes them powerful, and where they are headed.<\/p>\n\n\n\n
\n Foundation models are large-scale AI models trained on vast and diverse datasets using self-supervised learning techniques. They learn general patterns and representations across domains such as language, images, audio, and code, enabling them to perform a wide variety of tasks with minimal additional training. Once trained, foundation models can be fine-tuned or prompted for specific applications including text generation, summarization, translation, image creation, question answering, and code completion, making them the core building blocks of modern generative AI systems.\n <\/p>\n\n <\/div>\n\n<\/div>\n\n\n\n
The term foundation model was introduced by researchers at the Stanford Institute for Human-Centred Artificial Intelligence (HAI) in 2021. It describes a class of AI <\/a>models trained on broad data at scale, which can be adapted with relatively little additional training to a remarkably wide range of tasks.<\/p>\n\n\n\n Foundation models are defined by two core properties:<\/p>\n\n\n\n \u2022 Emergence: <\/strong>The ability to perform tasks they were not explicitly trained on, arising from scale and self-supervised learning.<\/p>\n\n\n\n \u2022 Homogenisation: <\/strong>A single model architecture that can be adapted to many domains, replacing the need for separate specialist models for each task.<\/p>\n\n\n\n Before foundation models, building an effective AI system typically required training a separate model for each specific task, one for translation, one for sentiment analysis, and another for image classification. Foundation models changed that paradigm entirely.<\/p>\n\n\n\n Building a foundation model involves three key stages: data collection, pre-training, and adaptation.<\/p>\n\n\n\n Foundation models require enormous volumes of training data. Large language models (LLM<\/a>s) like GPT are trained on hundreds of billions of tokens drawn from:<\/p>\n\n\n\n \u2022 Web pages and crawled internet data<\/p>\n\n\n\n \u2022 Books and academic publications<\/p>\n\n\n\n \u2022 Code repositories such as GitHub<\/p>\n\n\n\n \u2022 News articles and encyclopaedias<\/p>\n\n\n\n Image-based models like DALL-E are trained on hundreds of millions of image-text pairs. The breadth and diversity of this data is what enables generalisation, the ability to handle novel tasks not seen during training.<\/p>\n\n\n\n Self-supervised learning<\/a> is the training paradigm that makes foundation models possible at scale. Unlike supervised learning, it requires no manually labelled data. Instead, labels are derived automatically from the structure of the data itself.<\/p>\n\n\n\n For language models, common self-supervised objectives include:<\/p>\n\n\n\n \u2022 Next-token prediction (causal LM): <\/strong>Predict the next word given all previous words. This is how GPT-series models are trained.<\/p>\n\n\n\n \u2022 Masked language modelling (MLM): <\/strong>Randomly mask words in a sentence and predict the masked tokens. This is how BERT is trained.<\/p>\n\n\n\n For image models, objectives include predicting masked image patches or learning joint image-text embeddings (as in CLIP). Self-supervised pre-training on massive datasets gives the model a rich, general representation of the world.<\/p>\n\n\n\n Once pre-trained, a foundation model is a general-purpose base. Fine-tuning <\/a>adapts this base to a specific task using a smaller, labelled dataset. Because the model has already learned rich representations from pre-training, fine-tuning requires far less data and compute than training from scratch. This is the core principle of transfer learning.<\/p>\n\n\n\n Modern adaptation methods include:<\/p>\n\n\n\nHow Foundation Models Are Built<\/strong><\/h2>\n\n\n\n
Stage 1: Data Collection at Scale<\/strong><\/h3>\n\n\n\n
Stage 2: Self-Supervised Pre-Training<\/strong><\/h3>\n\n\n\n
Stage 3: Fine-Tuning and Adaptation<\/strong><\/h3>\n\n\n\n