Generative Pre-trained Transformer (GPT)

A Generative Pre-trained Transformer (GPT) is a decoder-only Transformer language model trained in two phases: large-scale unsupervised pre-training on text corpora (predicting the next token across trillions of tokens of training data), followed by alignment phases (supervised fine-tuning, RLHF, sometimes Constitutional AI) that shape the model's behavior to be useful and safe. The "GPT" name was coined by OpenAI in 2018; the architecture has since been adopted (with variations) by virtually every major language-model developer.

The OpenAI GPT lineage is instructive: GPT-1 (2018, 117M parameters) demonstrated the pre-train-then-fine-tune paradigm; GPT-2 (2019, 1.5B parameters) showed surprising zero-shot capabilities and was initially withheld over misuse concerns; GPT-3 (2020, 175B parameters) demonstrated in-context learning and made few-shot prompting practical; GPT-4 (2023) added multimodality, large context, and significantly better reasoning; GPT-4o (2024) integrated native audio/vision; GPT-5 (2025) brought another step-change in reasoning and tool use. Each generation roughly 10-100x the previous in capability per parameter.

Architecturally, GPT models are decoder-only Transformers — meaning they process input sequentially left-to-right and predict the next token at each position. They differ from encoder-decoder models (like the original Transformer, T5, BART) which are better suited for tasks like translation, and from encoder-only models (like BERT) which are better suited for classification. The decoder-only choice scaled best for generative tasks.

GPT models popularized three concepts that now dominate the field: (1) in-context learning — the model performs new tasks by seeing examples in the prompt, no fine-tuning required; (2) emergent capabilities — abilities that don't exist at smaller scales appear suddenly at sufficient scale; (3) the chat interface as a general-purpose AI front end. ChatGPT, built on GPT-3.5 and GPT-4, made these concepts accessible to non-developers.

Other major model families adopt the same core architecture with variations: Anthropic's Claude, Google's Gemini, Meta's Llama, Mistral's frontier models, and DeepSeek's R-series are all decoder-only Transformers descended from the GPT paradigm. The differences are in training data, alignment approach, and engineering details — but the architectural ancestor is the same.