Author(s) :

Rohan Madhok .

Volume/Issue :

Abstract :

Sequential motor learning, such as precise imitation of birdsong syllables, depends on the brain’s ability to link individual actions to delayed outcomes, a challenge known as the temporal credit assignment problem. This problem arises because feedback often arrives only after a sequence unfolds, obscuring which actions drive success or error. Inspired by songbird learning, this study investigates how reward-feedback frequency (granularity) within a sequential vocal imitation task affects learning efficiency in an actor-critic reinforcement learning agent. By systematically varying reward timing while keeping cumulative feedback constant, results show that finely grained, step-by-step feedback substantially accelerates learning and improves final imitation accuracy. In contrast, sparse feedback (delivered only at midpoint or endpoint) substantially impedes learning. Even when training was extended to 20,000 episodes, the end-only condition (K = 1) never reached the success criterion, and the half-only condition (K = 2) reached it in only a small fraction of seeds. This indicates a continuous relationship between feedback frequency and learning, rather than a sharp learnability cutoff. Notably, even under the densest feedback condition (reward after every action), performance plateaued at a non-zero error. Further analysis revealed that this plateau reflects not a single limit but two sources: the fixed exploration noise in the policy, and a residual imitation error that persists once the noise is removed. These findings characterize the trade-off between reward granularity and learning efficiency in a specific reinforcement learning model and task. They also suggest directions for future investigation of reward scheduling in biologically inspired learning systems.