Level 604 • Multi-Modal Systems

Multi-Modal Tokenization: Vision Patches, Audio Spectrograms & Unified Latents

How modern foundation transformers ingest high-resolution pixels, continuous speech spectrograms, and discrete text tokens into a shared semantic latent space via 2D patch projections, mel-filterbanks, and cross-attention projectors.

1. The Discrete vs. Continuous Ingestion Challenge

Text tokenization relies on discrete vocabulary lookups produced by algorithms like Byte-Pair Encoding (BPE). However, images and audio signals are continuous, high-dimensional arrays of spatial pixel intensities or temporal acoustic pressure waves. An uncompressed 1080p image contains over 2 million pixels across 3 color channels ($1920 \times 1080 \times 3 = 6.22 \times 10^6$ scalar values). Passing raw pixels directly into auto-regressive attention mechanisms with quadratic complexity $\mathcal{O}(N^2)$ would instantly exhaust high-bandwidth memory (HBM).

To bridge continuous physical signals with transformer sequence modeling, multi-modal architectures (such as GPT-4o, Gemini 1.5/2.0, and Claude 3.5/3.7) employ spatial and temporal patchification. The physical signal is decomposed into local tiles, mapped into discrete vector embeddings via learned linear projections or convolutional backbones, and concatenated with spatial position embeddings into the transformer sequence.

2. Vision Transformer (ViT) Patch Projection Mathematics

Given an input image $I \in \mathbb{R}^{H \times W \times C}$, where $H$ is height, $W$ is width, and $C$ is color channels (typically 3 for RGB), we divide the image into non-overlapping spatial patches of size $P \times P$ (typically $14 \times 14$ or $16 \times 16$). The number of resulting visual tokens $N$ is:

N = \frac{H \times W}{P^2}

Each 2D image patch $x_p^{(i)} \in \mathbb{R}^{P^2 \cdot C}$ is flattened into a 1D vector and mapped to the transformer hidden dimension $D$ via a learned linear projection matrix $E \in \mathbb{R}^{(P^2 \cdot C) \times D}$. Learnable 1D or 2D positional embeddings $E_{\text{pos}} \in \mathbb{R}^{(N + 1) \times D}$ are added to preserve spatial topology:

z_0 = \left[ x_{\text{class}}; \, x_p^{(1)} E; \, x_p^{(2)} E; \, \dots; \, x_p^{(N)} E \right] + E_{\text{pos}}

In high-resolution architectures (such as OpenAI's Tile System), an image is first scaled to fit within a $2048 \times 2048$ bounding box, divided into $512 \times 512$ tiles, and downsampled to a low-resolution thumbnail. Each $512 \times 512$ tile produces exactly 170 tokens, plus 85 base overview tokens, establishing the deterministic token formula:

\text{Total Image Tokens} = 85 + (170 \times \text{Tiles})

3. Audio Continuous Framing & Mel-Spectrogram Tokenization

Continuous audio streams are sampled at 16 kHz or 24 kHz. Applying a Short-Time Fourier Transform (STFT) with a 25ms Hanning window and a 10ms hop length transforms raw waveforms into a 2D time-frequency representation. This spectrogram is passed through an 80-channel or 128-channel Mel-filterbank, mimicking the non-linear frequency sensitivity of the human cochlea:

m = 2595 \log_{10}\left(1 + \frac{f}{700}\right)

1D convolutions with a stride of 2 or 4 downsample the temporal acoustic frames, yielding an empirical density of approximately 25 to 50 tokens per second of conversational speech. When ingested into native omni-models, audio tokens share the exact same transformer self-attention blocks as text tokens, enabling natural voice-to-voice turn-taking without intermediate speech-to-text transcription latency.

4. Production Python Implementation: Vision Patch Embedder

The following standalone Python module demonstrates the complete forward pass of a Vision Transformer patch projection layer using PyTorch:

import torch
import torch.nn as nn

class VisionPatchEmbedder(nn.Module):
    """
    Splits 2D images into patches and projects them into transformer embedding dimension D.
    Equivalence: Conv2d with kernel_size=patch_size and stride=patch_size.
    """
    def __init__(self, img_size: int = 224, patch_size: int = 14, in_chans: int = 3, embed_dim: int = 1024):
        super().__init__()
        self.img_size = img_size
        self.patch_size = patch_size
        self.num_patches = (img_size // patch_size) ** 2
        
        # 2D Convolution acts as efficient sliding patch extractor and linear projection
        self.proj = nn.Conv2d(
            in_channels=in_chans,
            out_channels=embed_dim,
            kernel_size=patch_size,
            stride=patch_size
        )
        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.pos_embed = nn.Parameter(torch.zeros(1, self.num_patches + 1, embed_dim))
        
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x shape: [Batch, Channels, Height, Width]
        B, C, H, W = x.shape
        assert H == self.img_size and W == self.img_size, f"Input image ({H}x{W}) must match img_size ({self.img_size})"
        
        # Conv projection -> [B, embed_dim, H/patch_size, W/patch_size]
        x = self.proj(x)
        # Flatten spatial dimensions -> [B, embed_dim, num_patches] -> Transpose -> [B, num_patches, embed_dim]
        x = x.flatten(2).transpose(1, 2)
        
        # Prepend learnable [CLS] classification token
        cls_tokens = self.cls_token.expand(B, -1, -1)
        x = torch.cat((cls_tokens, x), dim=1)
        
        # Add 1D learned position embeddings
        x = x + self.pos_embed
        return x

if __name__ == "__main__":
    embedder = VisionPatchEmbedder(img_size=224, patch_size=14, embed_dim=1024)
    dummy_img = torch.randn(2, 3, 224, 224)
    tokens = embedder(dummy_img)
    print(f"Output Token Tensor Shape: {tokens.shape}")
    # Expected: [2, 257, 1024] -> 256 visual patch tokens + 1 CLS token

Next in Curriculum

Level 605: Context Caching Economics & KV-Cache Eviction Algorithms

Explore Radix tree prefix caching mechanics, PagedAttention block allocation, and mathematical models of cache hit rate ROI in production multi-agent workflows.

Proceed to Level 605 →