Multi-armed Bandits

A bandit repeatedly selects an arm and observes its reward. In the basic setting, each choice is evaluated by its immediate reward without modeling how it changes a future state. Contextual bandits condition the action choice on information available for the current decision. Exploration strategies such as uncertainty-based selection or randomization allow the system to learn about less-used alternatives. Logged feedback is selective: the reward for an unchosen action is usually unknown. This distinguishes bandit learning from ordinary supervised learning, where each example can provide its target directly, and from broader reinforcement learning with delayed consequences through state transitions.

Define eligible actions, context and a reward that arrives on a usable timescale. Select an exploration policy compatible with operational constraints and record action probabilities, chosen actions and outcomes. Compare against a fixed policy and assess whether offline evaluation methods have sufficient overlap with the candidate policy. Monitor reward drift and performance across contexts. The practical result is an auditable selection policy with a deliberate exploration budget, rather than a greedy ranking system that never gathers evidence about alternatives it initially undervalued.

Rewards can be delayed, biased or misaligned with user benefit. Insufficient exploration makes alternatives hard to assess, while excessive exploration can reduce short-term quality. Offline estimates require assumptions about logging probabilities and action coverage and may have high variance. A basic bandit is inappropriate when choices materially change future states or long-term opportunities. Distinguish genuine reward changes from changes in exposure or measurement, and evaluate the policy rather than only the accuracy of its internal reward predictor.