Model Components
Parameter
Parameters are the numerical weights and biases within a neural network that encode the model's learned knowledge. During training, backpropagation adjusts parameters to minimize the loss function. A model with more parameters can represent more complex functions.
Frontier LLMs have tens to hundreds of billions of parameters (GPT-4 is estimated at ~1.8 trillion in a mixture-of-experts architecture). Parameter count is a proxy for model capacity, though efficiency improvements (LoRA, distillation) enable smaller models to approach larger models' performance.
Authority Links
Related Terms
Core Concepts
Hyperparameter
Parameter set before learning begins that controls the training process.
Techniques & Methods
Backpropagation
Training algorithm that adjusts neural network weights by propagating prediction errors backward through the network.
Techniques & Methods
Training
Teaching a model to make accurate predictions by exposing it to large datasets.
Model Components
Large Language Model (LLM)
A transformer-based neural network with billions to trillions of parameters, trained on broad text corpora to predict the next token and able to generate, summarize, classify, and reason over natural language.