The Essential Artificial Intelligence Glossary

A non-linear element-wise function σ(z) inserted between linear layers, enabling multi-layer networks to act as universal function approximators for non-linear manifolds.

A well-defined finite sequence of unambiguous mathematical or computational instructions that transforms an input state into an output state with deterministic time and space complexity O(f(n)).

The computational discipline focused on constructing software systems capable of performing cognitive tasks—such as perception, statistical inference, natural language processing, and automated reasoning—under empirical loss minimization.

A computational node executing an affine transformation followed by a non-linear activation: a = σ(∑_{i=1}^n w_i x_i + b), representing the atomic unit of neural networks.

An autonomous computational entity utilizing an LLM as its cognitive core, wrapped in an execution loop that inspects environment state, selects tool calls, and evaluates outcomes until a goal is achieved.

A sequence generation paradigm where output tokens are sampled iteratively from conditional probability distributions P(w_{t+1} | w_1, ..., w_t), appending each generated token to the input context.

An efficient algorithm for computing the gradient of an objective loss function with respect to all network weights by systematically applying the multivariable calculus chain rule backward from output to input.

An inference strategy prompting or conditioning language models to generate intermediate reasoning tokens, converting single-step generation into sequential problem decomposition.

The maximum sequence length L_seq of tokens that a Transformer architecture can attend across in a single forward pass without truncating earlier conversation history.

A specialized neural network architecture that enforces translation equivariance and local spatial inductive biases by applying sliding discrete 2D/3D convolution kernels across tensor feature maps.

A scale-invariant metric evaluating the cosine of the angle between two non-zero vectors: cos(θ) = (u · v) / (||u||_2 ||v||_2), normalized between -1.0 and +1.0.

The negative log-likelihood loss metric used in classification and language modeling, measuring the divergence between empirical target distribution y and predicted softmax probabilities ŷ.

Hierarchical representation learning utilizing computational graphs composed of multiple non-linear affine transformation layers (y = σ(W x + b)) capable of extracting feature representations without manual feature engineering.

An alignment algorithm that re-parameterizes the reward function directly in terms of the language model policy, optimizing preference loss implicitly without reinforcement learning stability issues.

The standard inner product operation between vectors u, v ∈ ℝ^n defined as u · v = ∑_{i=1}^n u_i v_i = ||u|| ||v|| cos(θ), serving as the foundational building block for matrix multiplication (GEMM).

One epoch denotes a complete training pass through the entire dataset. The batch size denotes the number of independent data vectors processed in parallel per forward/backward gradient update.

A binary container format designed by Georgi Gerganov and the llama.cpp team that packs quantized model weights, metadata, and tokenizers into a single file optimized for fast mmap execution on CPU and GPU.

A first-order iterative optimization algorithm that calculates the partial derivative gradient vector ∇_θ L and steps parameters in the opposite direction proportional to learning rate η.

A highly parallel SIMT (Single Instruction, Multiple Threads) processor specialized in throughput-oriented matrix multiplication (GEMM) rather than low-latency serial out-of-order execution.

A phenomenon where an autoregressive model produces syntactically and semantically fluent outputs that deviate from verified external ground truth or contradict user-provided context.

A 3D-stacked DRAM architecture connected via microscopic through-silicon vias (TSVs) to an ultra-wide memory bus (e.g. 5120-bit on NVIDIA H100), delivering over 3.35 TB/s of memory bandwidth.

Vector spaces ℝ^d where d is large (e.g. d = 4096). Characterized by high-dimensional geometry where random vectors are near-orthogonal and volume concentrates entirely on spherical boundaries.

A dedicated VRAM memory buffer that stores the computed Key and Value attention projection tensors for all prior tokens across layers: Memory_KV = 2 × 2 × n_layers × n_heads × d_head × L_seq × batch × bytes.

A critical training hyperparameter scalar η that scales the magnitude of parameter updates during gradient descent optimization (typically between 1e-5 and 1e-2).

A scalar function L(y, ŷ) measuring the discrepancy between model predictions ŷ and true target labels y (e.g. Mean Squared Error, Binary Cross-Entropy, KL Divergence).

A parameter-efficient fine-tuning (PEFT) technique that freezes pre-trained weight matrices W_0 ∈ ℝ^{d × k} and decomposes updates into two low-rank matrices: W = W_0 + B · A, where rank r << min(d, k).

The subset of AI where mathematical models infer latent function mappings f: X → Y by optimizing parameter weights θ over empirical training samples through objective loss minimization, rather than human rule specification.

An open JSON-RPC protocol specification that standardizes how AI agents discover tools, subscribe to real-time resources, and invoke executable capabilities across isolated servers.

Projecting queries, keys, and values into h distinct representation subspaces, executing self-attention in parallel, and concatenating outputs: MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W^O.

A feedforward deep neural network architecture consisting of fully connected linear layers interleaved with non-linear activation functions, mathematically represented as f(x) = W_3 σ(W_2 σ(W_1 x + b_1) + b_2) + b_3.

The ratio of mathematical floating-point operations executed to memory traffic transferred: I = Total FLOPs / Memory Bytes, determining whether an operation is compute-bound or memory-bound under the Roofline Model.

Overfitting occurs when a high-capacity model fits sample-specific stochastic noise, yielding low training error but high validation error (high variance). Underfitting occurs when the model cannot express the true data manifold (high bias).

The internal learnable numerical coefficients in neural layers. Weights (W) scale input features linearly (W · x), while biases (b) shift the activation threshold independently of inputs.

The compute-intensive self-supervised training phase where base model parameters θ are optimized over trillions of tokens using causal language modeling loss to learn general representations.

Inference consists of two phases: Prefill processes the prompt in parallel via dense compute-bound GEMM; Decode generates subsequent tokens autoregressively via memory-bandwidth-bound GEMV.

The numerical compression process of mapping continuous 16-bit floating-point weights into lower-precision discrete representations (e.g. 8-bit or 4-bit integers) using scaling factors and zero-point offsets.

A sequential neural architecture that maintains a recursive hidden state vector h_t = σ(W_h h_{t-1} + W_x x_t + b), subject to vanishing gradients across long time horizons.

A Markov Decision Process (MDP) framework where an autonomous agent learns an optimal action policy π(a|s) to maximize cumulative discounted rewards R_t = ∑ γ^k r_{t+k+1} through environment interaction.

A multi-stage post-training alignment protocol where a reward model is trained on human preference rankings and used to optimize the LLM policy via Proximal Policy Optimization (PPO) penalized by KL divergence.

Identity shortcut mappings that reformulate layer transformations as y = F(x, {W_i}) + x, ensuring unobstructed gradient flow directly to early layers and eliminating vanishing gradients.

A hybrid architecture that intercepts user queries, retrieves semantically relevant text chunks from a vector database via cosine similarity, and injects them as verified grounding context into the LLM prompt.

A relative positional encoding method that rotates query and key vectors in 2D coordinate pairs by multiplying them with orthogonal block-diagonal rotation matrices R_{Θ,m}^d.

A dynamic weighting operation computed across query (Q), key (K), and value (V) projections: Attention(Q, K, V) = softmax((Q K^T) / √d_k) V, mapping global token dependencies in O(n²) complexity.

An approximation of batch gradient descent where the parameter gradient ∇_θ L is computed over a randomly sampled mini-batch B ⊂ D rather than the entire dataset.

Second-stage training on curated prompt-response pairs D = {(prompt_i, response_i)} that shifts the base model from raw web text completion into an instruction-following assistant.

Training machine learning algorithms on labeled dataset pairs D = {(x_i, y_i)}_{i=1}^N, optimizing parameters to approximate the conditional mapping P(Y | X).

The privileged prefix conditioning string prepended to the conversational context before user turns, establishing role-based behavior, tone boundaries, and output format constraints.

Decoding hyperparameters: Temperature T scales logit variance prior to softmax (z_i / T); Top-p (nucleus sampling) truncates the probability mass to the smallest cumulative threshold p.

Hardwired execution units on modern GPU architectures (Hopper, Blackwell) designed to perform mixed-precision matrix multiply-accumulate operations (D = A × B + C) in a single hardware clock cycle.

Scaling inference-time compute via search algorithms (e.g. Monte Carlo Tree Search) and verification loops, improving model accuracy on complex reasoning tasks without increasing parameter size.

The architectural divergence where processor compute throughput (FLOPs/s) scales significantly faster than memory bus transfer speed (Bytes/s), causing compute units to stall while awaiting tensor weights.

The deterministic algorithmic mapping that converts continuous character sequences into discrete integer IDs from a fixed vocabulary V via algorithms like Byte-Pair Encoding (BPE).

Training is the computationally expensive optimization phase where parameter weights are iteratively updated via gradient backpropagation. Inference is the evaluation phase where the frozen model processes new inputs via forward propagation.

A foundational neural architecture published in 2017 that eliminated sequential recurrent recurrence in favor of multi-head self-attention mechanisms and feedforward networks with residual connections.

Discovering underlying structural patterns, probability densities P(X), or low-dimensional manifolds from unlabeled inputs X without external target supervision.

A dense continuous vector representation x ∈ ℝ^d mapping discrete semantic tokens or entities into a metric latent space where spatial distance reflects geometric similarity.

Dedicated high-bandwidth memory silicon located adjacent to or on the GPU package, responsible for holding model parameter weights, optimizer states, activations, and the KV cache.