Model Components
Autoregressive Model
Autoregressive models generate sequences by factoring the joint distribution as a product of conditional distributions: P(x) = ∏ P(xi | x1..xi-1). Each step's prediction uses all previous outputs as context, making generation sequential and order-sensitive.
All major LLMs are autoregressive: they generate text left-to-right, token by token. This architecture is simple, scales well, and produces high-quality outputs, but inference cannot be parallelized across the output sequence.
Authority Links
Related Terms
Techniques & Methods
Autoregression
Statistical modeling approach where future values are predicted from past observed values.
Model Components
Language Model
AI system that assigns probabilities to sequences of words and can generate coherent text.
Techniques & Methods
Sequence Generation
Process where models produce sequences—such as words or tokens—based on learned patterns.
Model Components
Transformer Decoder
Transformer component that generates output sequences by attending to encoded inputs and prior outputs.