Dataset Batch Generator for Training Pipelines

In machine learning, training models on large datasets requires efficient data handling strategies. One of the most fundamental techniques is batch processing, where the dataset is divided into smaller, equally-sized groups called batches. Instead of loading the entire dataset into memory at once, batches allow for incremental processing, enabling training on datasets that would otherwise exceed available memory.

A batch generator (or batch iterator) is a utility function that sequentially partitions a dataset into fixed-size chunks. This technique is essential for:

• Mini-batch gradient descent: Updates model weights using gradients computed from small subsets of data
• Memory efficiency: Processes large datasets that don't fit in RAM
• Regularization effect: Introduces beneficial noise through varying batch compositions

The Batching Process:

Given a feature matrix X with n samples and an optional target vector y of length n, the batch generator divides the data into ⌈n / batch_size⌉ sequential groups:

• Full batches: Most batches contain exactly batch_size samples
• Final batch: The last batch may contain fewer samples if n is not evenly divisible by batch_size

Your Task:

Implement a function that generates sequential batches from a numpy array X and an optional numpy array y. The function should:

1. Divide the input data into chunks of the specified batch_size
2. If y is provided, return batches as pairs [X_batch, y_batch]
3. If y is None, return batches containing only X_batch
4. Handle the final batch gracefully when the dataset size is not perfectly divisible by the batch size