Trajectory Cumulative Discounted Reward

In reinforcement learning, an agent interacts with an environment by taking actions and receiving rewards over time. A fundamental concept for evaluating the quality of a decision-making policy is the cumulative discounted reward, often denoted as Gₜ (the return starting from time step t).

The cumulative discounted reward captures the total value of rewards collected along a trajectory, with future rewards being progressively discounted by a factor γ (gamma). This discounting mechanism reflects the intuition that immediate rewards are typically more valuable than distant future rewards—a principle rooted in both economic theory (time value of money) and practical considerations (uncertainty about the future).

Mathematical Definition:

For a sequence of rewards [R₁, R₂, R₃, ..., Rₙ] received at consecutive timesteps, the cumulative discounted reward is computed as:

$$G_t = \sum_{k=0}^{n-1} \gamma^k \cdot R_{t+k+1} = R_1 + \gamma R_2 + \gamma^2 R_3 + \gamma^3 R_4 + \ldots + \gamma^{n-1} R_n$$

Where:

• Rₖ is the reward received at timestep k
• γ (gamma) is the discount factor, ranging from 0 to 1
• n is the total number of rewards in the trajectory

Understanding the Discount Factor γ:

• When γ = 0: Only the immediate reward matters (myopic agent)
• When γ = 1: All rewards are weighted equally (no discounting)
• When 0 < γ < 1: Future rewards are exponentially discounted, with rewards received further in the future contributing less to the total return

This quantity is central to defining the state-value function Vπ(s) and the action-value function Qπ(s, a) in reinforcement learning, which estimate the expected return from a given state or state-action pair under a policy π.

Your Task: Write a Python function that computes the cumulative discounted reward for a given sequence of rewards and a discount factor. Use NumPy for efficient computation.