7a:T4af,Consider the difference between a photograph and a video. The photograph captures a single instant—a frozen slice of reality with spatial structure but no temporal extent. The video, by contrast, is a sequence: a succession of frames whose meaning emerges not just from individual images but from their ordering, their transitions, and their evolution over time. This distinction lies at the heart of sequence modeling. While much of classical machine learning focuses on **independent and identically distributed (i.i.d.)** data—where each sample stands alone—an enormous class of real-world problems involves data where **order matters fundamentally**. The word 'dog' differs from 'god' not in its letters but in their arrangement. A stock price today depends on prices yesterday and last month. A medical diagnosis unfolds through a progression of symptoms over time. In this page, we develop a rigorous taxonomy of sequential data types, understanding their structural properties, the assumptions they encode, and the challenges they pose for machine learning systems. This foundation is essential before we can design neural architectures that capture temporal dependencies effectively.7b:T459,**Where I.I.D. Breaks Down:** However, vast domains of data violate the i.i.d. assumption fundamentally—not as a technical inconvenience but as a structural feature of the data itself: 1. **Text**: Words in a sentence are not independent. 'The cat sat on the ___' restricts the next word far more than knowing individual word frequencies. 2. **Speech**: Phonemes are influenced by surrounding sounds (coarticulation). The sound of 'k' in 'skip' differs from 'k' in 'ski' because of what follows. 3. **Time Series**: A stock price at time $t$ depends on prices at times $t-1, t-2, \ldots$. Temperature today correlates with temperature yesterday. 4. **Video**: Each frame depends on previous frames. Object trajectories, camera motion, and scene dynamics all create temporal structure. 5. **Genomic Sequences**: DNA bases follow statistical patterns. Promoter regions, codons, and regulatory elements all create non-random sequential structure. 6. **User Behavior Logs**: A user's next action depends on their previous actions. Session context, browsing history, and past purchases all inform predictions.7c:T452,**Natural Language Processing (NLP)** Text represents a paradigmatic example of sequential data with rich structure: **Tokenization levels:** - Character sequences: 'h', 'e', 'l', 'l', 'o' - Subword units: 'hel', 'lo' (used by modern models like BPE, WordPiece) - Word sequences: 'hello', 'world' - Sentence/document sequences: hierarchical structure **Key properties:** - **Discrete vocabulary**: Finite (though possibly large) set of symbols - **Variable length**: Sentences range from 1 to hundreds of words; documents can have thousands - **Compositional structure**: Words → phrases → clauses → sentences → paragraphs - **Long-range dependencies**: Coreference, discourse structure, narrative arcs - **Ambiguity**: Same word sequences can have different meanings (semantic ambiguity) **Sequential patterns in language:** - N-gram statistics: P('the' | 'in') differs from P('the' | 'eat') - Syntactic constraints: Verbs follow subjects, objects follow verbs (in English) - Semantic coherence: Topics persist across paragraphs - Pragmatic structure: Conversations follow turn-taking patterns7d:T407,**Speech and Audio Processing** Audio is a quintessentially continuous sequence, sampled at high rates: **Sampling and representation:** - Raw waveform: Amplitude values at 16,000-48,000 Hz - Spectral features: Mel spectrograms, MFCCs (typically 10-100ms frames) - Learned representations: End-to-end neural feature extraction **Key properties:** - **High sampling rate**: 1 second = 16,000-48,000 samples - **Continuous values**: Amplitude in a bounded range - **Short-term structure**: Phonemes last ~50-200ms - **Long-term structure**: Words, sentences, prosodic phrases - **Multi-scale patterns**: Pitch (fundamental frequency) varies at different rates than phoneme identity **Unique challenges:** - **Coarticulation**: Adjacent phonemes influence each other's acoustic realization - **Speaker variation**: Same words sound different from different speakers - **Noise and overlap**: Real-world audio contains background sounds, overlapping speakers - **Prosody**: Intonation, stress, and rhythm convey meaning beyond words7e:T445,**Time Series Analysis** Time series data arises whenever we record measurements over time: **Domains:** - Financial markets: Prices, volumes, volatility - Industrial processes: Sensor readings, equipment telemetry - Environmental monitoring: Temperature, pollution, seismic activity - Healthcare: Vital signs, lab values, disease progression **Key properties:** - **Regular or irregular sampling**: May or may not be evenly spaced - **Multiple simultaneous series**: Multivariate time series are common - **Stationarity assumptions**: Often violated (trends, regime changes) - **Noise characteristics**: Heteroscedasticity, outliers, missing values **Classical concepts:** - **Trend**: Long-term systematic change - **Seasonality**: Periodic fluctuations at known frequencies - **Cycles**: Fluctuations without fixed period - **Residuals**: Random variation after removing structure **Forecasting challenge:** The fundamental task is predicting $x_{T+1}, x_{T+2}, \ldots, x_{T+H}$ given observed $x_1, \ldots, x_T$. Horizon $H$ can range from one step to arbitrarily far into the future.7f:T45c,**Video Understanding** Video combines spatial (image) and temporal (sequence) structure: **Representation:** - Raw pixels: Each frame is a 3D tensor (height × width × channels) - Optical flow: Motion vectors between consecutive frames - Feature sequences: Frame-level CNN features extracted per frame **Key properties:** - **High dimensionality**: 1080p frame = 1920×1080×3 ≈ 6.2 million values - **Frame rates**: Typically 24-60 frames per second - **Strong temporal correlation**: Adjacent frames are nearly identical - **Hierarchical events**: Actions compose into activities, activities into events **Temporal patterns:** - Object trajectories: Position and appearance change smoothly - Actions: Temporal patterns of motion characterizing activities - Events: Higher-level occurrences spanning multiple actions - Narrative: Story-level structure in narrative video **Challenges:** - **Computational cost**: Processing millions of values per frame is expensive - **Long videos**: Movies span hours; surveillance runs continuously - **Multi-object tracking**: Multiple entities interact in complex ways80:T477,**Genomics and Biological Sequences** Biological sequences encode the instructions of life: **Sequence types:** - **DNA**: 4-letter alphabet {A, C, G, T} - **RNA**: 4-letter alphabet {A, C, G, U} - **Protein**: 20-letter amino acid alphabet **Key properties:** - **Extremely long sequences**: Human genome ≈ 3 billion base pairs - **Sparse functional elements**: Most sequence is non-coding; functional regions are distributed - **Evolutionary conservation**: Functionally important regions change slowly across species - **Hierarchical structure**: Bases → codons → genes → pathways → phenotypes **Sequential patterns:** - **Motifs**: Short patterns with functional significance (transcription factor binding sites) - **Secondary structure**: RNA and proteins fold based on sequence patterns - **Long-range interactions**: Distant regions interact when folded - **Positional encoding**: Position within a gene/chromosome matters **Applications:** - Variant effect prediction: Does this mutation alter function? - Gene finding: Where are the genes in a genome? - Regulatory element identification: What controls gene expression?81:T45c,**The Causality Constraint:** A particularly important constraint in sequence modeling is **causality**: at prediction time, we only have access to the past. When predicting $x_{t+1}$, we cannot use information from $x_{t+2}, x_{t+3}, \ldots$—those haven't occurred yet. This seems obvious, but it has profound implications: 1. **Training vs. inference asymmetry**: During training, we typically have access to the entire sequence. We must deliberately mask future information to avoid learning to 'cheat'. 2. **Autoregressive bottleneck**: At inference time, we must generate one element at a time, each conditioned on previously generated elements. This is inherently sequential and cannot be parallelized. 3. **Error accumulation**: Small errors in prediction compound. If we predict incorrectly at step $t$, all predictions from step $t+1$ onward may be influenced by this error. Note that in some applications (e.g., bidirectional text encoding for classification), we DO have access to the full sequence. But for generation tasks—language generation, forecasting, planning—causality is fundamental.72:{"json":{"id":"seed_704d969da2f365ff8d7d51e156a33e3f","slug":"sequence-modeling","category":"Machine Learning","topic":"Sequence Modeling","title":"Sequence Modeling","description":"Master the foundations of sequence modeling in deep learning. Understand the nature of sequential data, temporal dependencies, autoregressive formulations, and the computational paradigm of recurrent processing through unfolded computation graphs.","difficulty":"INTERMEDIATE","estimatedMinutes":90,"pages":[{"id":"seed_235183e88e4f00a358e8d13db4003a6c","title":"Sequential Data Types","order":0,"content":[{"id":"opening-hook","title":"The World Unfolds in Time","content":[{"data":{"content":"$7a"},"type":"text"},{"data":{"icon":"Target","title":"Learning Objectives","content":"By the end of this page, you will: (1) Distinguish between i.i.d. and sequential data paradigms, (2) Classify sequential data by structural properties, (3) Understand the unique challenges sequences pose, and (4) Recognize sequential patterns across diverse application domains.","variant":"info"},"type":"callout"}]},{"id":"iid-assumption-limits","title":"The I.I.D. Assumption and Its Limits","content":[{"data":{"content":"Most classical machine learning rests on a comfortable foundation: the assumption that training examples are **independent and identically distributed (i.i.d.)**. Under this assumption, each data point $(x_i, y_i)$ is drawn from the same underlying distribution $P(X, Y)$, and knowing one data point tells us nothing additional about any other.\n\nThis assumption enables powerful theoretical guarantees. It allows us to bound generalization error, prove convergence of learning algorithms, and treat the training set as an unbiased sample of the true distribution. Most supervised learning theory—from VC dimension to PAC learning—depends critically on the i.i.d. assumption."},"type":"text"},{"data":{"icon":"BookOpen","title":"The I.I.D. Definition","content":"A sequence of random variables $X_1, X_2, \\ldots, X_n$ is **independent and identically distributed** if: (1) Each $X_i$ follows the same probability distribution $P(X)$, and (2) For any subset of indices, $P(X_{i_1}, \\ldots, X_{i_k}) = P(X_{i_1}) \\cdot \\ldots \\cdot P(X_{i_k})$. The joint distribution factorizes completely.","variant":"note"},"type":"callout"},{"data":{"content":"**Where I.I.D. Works Well:**\n\nThe i.i.d. assumption is reasonable for many practical problems:\n\n- **Image classification**: Each photograph is typically captured independently\n- **Medical diagnosis from single measurements**: A blood test result, a single X-ray\n- **Click-through rate prediction**: Each ad impression, considered in isolation\n- **Document classification**: Each document as a standalone unit\n\nIn these settings, treating samples as independent introduces minimal modeling error, and the mathematical convenience is worth this approximation."},"type":"text"},{"data":{"content":"$7b"},"type":"text"},{"data":{"icon":"AlertCircle","title":"The Danger of Ignoring Sequence Structure","content":"Treating sequential data as i.i.d. doesn't just reduce accuracy—it can lead to fundamentally wrong conclusions. A language model trained on shuffled text would learn that 'the' is common, but not that it precedes nouns. A time series model ignoring temporal order would miss trends, periodicities, and dependencies that define the signal. The structure IS the information.","variant":"warning"},"type":"callout"}],"titleType":"number","titleNumber":1},{"id":"formal-definition-sequences","title":"Formal Definition of Sequences","content":[{"data":{"content":"A **sequence** is an ordered collection of elements, typically indexed by a discrete or continuous variable representing position or time. Formally:\n\n$$\\mathbf{x} = (x_1, x_2, \\ldots, x_T)$$\n\nwhere $T$ is the sequence length (which may vary across examples or even be infinite), and each $x_t$ belongs to some observation space $\\mathcal{X}$.\n\nThe critical property distinguishing sequences from sets or bags is that the **index $t$ carries semantic meaning**. Permuting a sequence generally produces a different object with different meaning:\n\n$$(x_1, x_2, x_3) \neq (x_3, x_1, x_2)$$\n\nThis contrasts with a set $\\{x_1, x_2, x_3\\}$ which equals $\\{x_3, x_1, x_2\\}$ by definition."},"type":"text"},{"data":{"rows":[["$$\\mathbf{x}$","Complete sequence","$$(x_1, x_2, \\ldots, x_T) \\in \\mathcal{X}^T$"],["$$x_t$","Element at position/time $t$","$$x_t \\in \\mathcal{X}$ (reals, vectors, symbols)"],["$$T$","Sequence length","$$T \\in \\mathbb{Z}^+$ (variable or fixed)"],["$$t$","Index variable","Time, position, step number"],["$$\\mathbf{x}_{

Autoregressive Models

Autoregressive Models