Transformer Position Embedding Generator

In modern sequence-to-sequence models like Transformers, understanding the order of tokens in a sequence is crucial for tasks like language understanding, translation, and generation. Unlike recurrent networks that inherently process sequences step-by-step, Transformers process all tokens simultaneously through self-attention mechanisms. This creates a fundamental challenge: how does the model know which position each token occupies in the sequence?

The solution is sinusoidal position embeddings—a deterministic encoding scheme that injects positional information into the model without requiring any learned parameters. This technique was introduced in the seminal "Attention Is All You Need" paper and remains foundational to understanding modern AI architectures.

The Core Idea

Position embeddings assign a unique, continuous vector to each position in a sequence. The sinusoidal approach uses sine and cosine functions at varying frequencies to create these vectors. The mathematical elegance lies in the fact that:

1. Each position gets a unique encoding that the model can distinguish
2. The model can extrapolate to longer sequences than seen during training
3. Relative positions can be computed through linear transformations of the embeddings

Mathematical Formulation

For a position pos in the sequence and dimension index i within the embedding vector of size d_model, the encoding is computed as:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

$$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

Where:

• pos ranges from 0 to sequence_length - 1
• i ranges from 0 to d_model/2 - 1
• Even indices (2i) use the sine function
• Odd indices (2i+1) use the cosine function
• The denominator creates wavelengths ranging from 2π to 10000·2π

Your Task

Implement a function that generates sinusoidal position embeddings for a given sequence length and embedding dimensionality. Your function should:

• Return a NumPy array of shape (1, sequence_length, embedding_dim) with dtype float16
• For each position, compute sine values at even indices and cosine values at odd indices
• Return -1 if sequence_length is 0 or if embedding_dim is less than or equal to 0

This implementation forms the backbone of positional understanding in transformer-based models and is essential knowledge for anyone working with modern NLP and AI systems.