Examples#

Runnable scripts under examples/ in the repository. Each writes a logfile and regret figures to results/:

python examples/plot_bandit_regret.py

Comparing bandit algorithms#

The shape of every experiment in statrl: an environment, the agents to compare, an oracle to define zero regret, and one call to the harness. Also shows how the choice of divergence changes IMED’s behaviour on the same instance.

 1"""
 2Comparing bandit algorithms on a Bernoulli instance
 3===================================================
 4
 5Benchmarks IMED against uniform exploration on a three-armed Bernoulli bandit
 6and plots the regret of each, averaged over replicates.
 7
 8This is the shape of every experiment in ``statrl``: build an environment, list
 9the agents to compare, add an oracle to define zero regret, and hand the four to
10:func:`~statrl.experiments.massiveruns.runLargeMulticoreExperiment`.
11
12Run with::
13
14    python examples/plot_bandit_regret.py
15
16Writes a logfile and regret figures under ``results/``.
17"""
18
19from statrl.experiments.massiveruns import runLargeMulticoreExperiment
20from statrl.settings.bandits.stochastic.anytime.agents._Oracle import Oracle
21from statrl.settings.bandits.stochastic.anytime.agents._Random import Random
22from statrl.settings.bandits.stochastic.anytime.agents.IMED import IMED
23from statrl.settings.bandits.stochastic.anytime.envs.parametric import BernoulliBandit
24from statrl.settings.bandits.stochastic.anytime.interaction import BanditInteraction
25from statrl.settings.utils import klBern, klGauss
26
27
28def main():
29    # Arm 1 is best. The gap to arm 2 is 0.4, wide enough that a good algorithm
30    # resolves it quickly and the regret curve flattens within the horizon.
31    env = BernoulliBandit([0.2, 0.9, 0.5])
32    nA = env.number_arms
33
34    agents = [
35        # Two divergences on the same rewards: klBern is the matched choice for
36        # Bernoulli arms, klGauss a valid but looser one. Distinct names, or the
37        # two collide in the dump filenames and the legend.
38        IMED(nA, kullback=klBern, name="IMED-Bernoulli"),
39        IMED(nA, kullback=klGauss, name="IMED-Gaussian"),
40        Random(env),
41    ]
42
43    runLargeMulticoreExperiment(
44        env,
45        agents=agents,
46        oracle=Oracle(env),
47        interact=BanditInteraction(),
48        timeHorizon=2000,
49        nbReplicates=50,
50        root_folder="results/",
51    )
52
53
54if __name__ == "__main__":
55    main()

Writing your own agent#

A greedy agent and an epsilon-greedy one, benchmarked against IMED. The interesting part is what greedy gets wrong, and how that failure shows up in the quantile bands of the figure rather than in the mean curve.

  1"""
  2Writing your own bandit agent
  3=============================
  4
  5Implements a greedy agent and an epsilon-greedy one, then benchmarks both
  6against IMED. The point of the comparison is what greedy gets wrong: with no
  7exploration it can lock onto an arm that looked good early and never revisit
  8that decision, so its regret grows linearly on some runs and stays near zero on
  9others. Averaged over replicates that shows up as a wide quantile band — which
 10is exactly what the shaded regions in the output figure display.
 11
 12Run with::
 13
 14    python examples/custom_agent.py
 15
 16Writes a logfile and regret figures under ``results/``.
 17"""
 18
 19import numpy as np
 20
 21from statrl.experiments.massiveruns import runLargeMulticoreExperiment
 22from statrl.settings.bandits.stochastic.anytime.agent import BanditAgent
 23from statrl.settings.bandits.stochastic.anytime.agents._Oracle import Oracle
 24from statrl.settings.bandits.stochastic.anytime.agents.IMED import IMED
 25from statrl.settings.bandits.stochastic.anytime.envs.parametric import BernoulliBandit
 26from statrl.settings.bandits.stochastic.anytime.interaction import BanditInteraction
 27from statrl.settings.utils import klBern, randmax
 28
 29
 30class Greedy(BanditAgent):
 31    """Always play the arm with the best empirical mean.
 32
 33    Parameters
 34    ----------
 35    nbArms : int
 36        Number of arms.
 37    name : str, default='Greedy'
 38        Label used in logfiles and plot legends.
 39    """
 40
 41    def __init__(self, nbArms, name="Greedy"):
 42        self.nbArms = nbArms
 43        super().__init__(name=name)
 44
 45    def reset(self):
 46        """Clear every statistic before a new run."""
 47        super().reset()
 48        self.counts = np.zeros(self.nbArms)
 49        self.means = np.zeros(self.nbArms)
 50
 51    def select_arm(self):
 52        """Play the arm of highest empirical mean, breaking ties at random.
 53
 54        Returns
 55        -------
 56        int
 57            Index of the selected arm.
 58        """
 59        # randmax, not np.argmax: at t=0 every mean is 0, and argmax would
 60        # always return arm 0, so the agent would never try anything else.
 61        return randmax(self.means)
 62
 63    def update(self, arm, reward):
 64        """Fold one reward into the running mean of its arm.
 65
 66        Parameters
 67        ----------
 68        arm : int
 69            Index of the arm that was pulled.
 70        reward : float
 71            Reward observed for it.
 72        """
 73        self.counts[arm] += 1
 74        self.means[arm] += (reward - self.means[arm]) / self.counts[arm]
 75
 76
 77class EpsilonGreedy(Greedy):
 78    """Greedy, but explore uniformly with probability ``epsilon``.
 79
 80    Parameters
 81    ----------
 82    nbArms : int
 83        Number of arms.
 84    epsilon : float, default=0.1
 85        Probability of ignoring the empirical means and drawing at random. A
 86        constant rate keeps exploring forever, so the regret stays linear —
 87        with a much smaller slope than greedy's worst case.
 88    name : str, default='eps-Greedy'
 89        Label used in logfiles and plot legends.
 90    """
 91
 92    def __init__(self, nbArms, epsilon=0.1, name="eps-Greedy"):
 93        self.epsilon = epsilon
 94        super().__init__(nbArms, name=name)
 95
 96    def select_arm(self):
 97        """Explore with probability ``epsilon``, otherwise play greedily.
 98
 99        Returns
100        -------
101        int
102            Index of the selected arm.
103        """
104        if self.np_random.random() < self.epsilon:
105            return int(self.np_random.integers(self.nbArms))
106        return randmax(self.means)
107
108
109def main():
110    env = BernoulliBandit([0.2, 0.9, 0.5])
111    nA = env.number_arms
112
113    runLargeMulticoreExperiment(
114        env,
115        agents=[
116            IMED(nA, kullback=klBern),
117            EpsilonGreedy(nA, epsilon=0.1),
118            Greedy(nA),
119        ],
120        oracle=Oracle(env),
121        interact=BanditInteraction(),
122        timeHorizon=2000,
123        nbReplicates=50,
124        root_folder="results/",
125    )
126
127
128if __name__ == "__main__":
129    main()

See also#

Short, self-contained snippets live in the docstrings themselves, under Examples; they run in CI, so they cannot drift from the code. The Quickstart is the shortest path from install to a regret number.