PSRL

class statrl.settings.markovdecisionprocess.discrete_nostructure.agents.PSRL.PSRL(nS, nA, delta=0.05)

Bases: MDPAgent

Posterior Sampling Reinforcement Learning.

PSRL is a Bayesian model-based reinforcement learning algorithm for finite Markov Decision Processes.

Rather than constructing optimistic confidence sets, PSRL maintains posterior distributions over the unknown reward function and transition probabilities.

At the beginning of each episode, a complete MDP is sampled from the posterior. The optimal policy of this sampled MDP is computed and executed until a stopping criterion triggers the beginning of a new episode.

This implementation assumes

• finite state and action spaces;

• Bernoulli rewards;

• Beta priors over rewards;

• Dirichlet priors over transitions.

Parameters:

• nS (int) – Number of states.

• nA (int) – Number of actions.

• delta (float, optional) – Confidence parameter. Present for compatibility with the common learner interface but not explicitly used by the PSRL algorithm.

t

Global interaction time.

Type:

int

policy

Current stochastic policy computed on the sampled MDP.

Type:

ndarray of shape (nS, nA)

u

Bias (relative value) function obtained by Value Iteration.

Type:

ndarray of shape (nS,)

Nk

Cumulative number of visits to every state-action pair over all completed episodes.

Type:

ndarray of shape (nS, nA)

vk

State-action visit counts during the current episode.

Type:

ndarray of shape (nS, nA)

r_successCounts

Beta posterior α parameters.

r_failureCounts

Beta posterior β parameters.

p_pseudoCounts

Dirichlet pseudo-counts defining the posterior transition model.

r_sampled

Reward function sampled from the posterior.

p_sampled

Transition kernel sampled from the posterior.

Methods

VI([epsilon, max_iter])	Solve the sampled MDP by average-reward (relative) value iteration.
__init__(nS, nA[, delta])	Construct a Posterior Sampling Reinforcement Learning learner.
new_episode()	Fold in the last episode's counts, sample a fresh MDP, and solve it.
play(state)	Sample an action from the current policy, starting an episode if due.
reset(inistate)	Clear every statistic, restore the priors, and open the first episode.
update(state, action, reward, observation)	Update the learner (the Beta and Dirichlet posteriors) with one transition (one step of the current policy).
updateN()	Auxiliary function to update N the current state-action count.

VI(epsilon=0.01, max_iter=1000)

Solve the sampled MDP by average-reward (relative) value iteration.

Iterates the Bellman operator on the bias function until the span of successive iterate differences contracts below epsilon (Puterman & Chan, “Markov Decision Processes and Reinforcement Learning”, Ch. 7, Algorithm 7.1), then computes the greedy policy for the converged bias function. Ties among greedy actions are broken uniformly among the least-visited ones, to keep exploring where the sampled model cannot distinguish actions.

Parameters:

• epsilon (float, default=0.01) – Stopping threshold on the span of successive bias differences.

• max_iter (int, default=1000) – Maximum number of sweeps. If reached without convergence, the current iterate is kept and a warning is printed.

new_episode()

Fold in the last episode’s counts, sample a fresh MDP, and solve it.

Draws a reward mean per state-action pair from its Beta posterior and a transition row from its Dirichlet posterior, then runs VI() on that sampled model. Sampling a whole MDP rather than perturbing each pair independently is what makes the exploration coherent across states — the policy commits to one plausible world for the episode.

The value-iteration precision tightens as 1 / t, so early episodes are solved coarsely and later ones exactly.

play(state)

Sample an action from the current policy, starting an episode if due.

Parameters:

state (int) – Current state.

Returns:

The chosen action.

Return type:

int

Notes

The doubling stopping criterion ends an episode as soon as a state-action pair has been visited as often within it as in all previous episodes combined. This bounds the number of episodes logarithmically, so the cost of resampling and re-solving the MDP stays negligible against the horizon.

reset(inistate)

Clear every statistic, restore the priors, and open the first episode.

Reward counts restart at Beta(1, 1) and transition pseudo-counts at a flat Dirichlet(1, ..., 1), i.e. uniform priors, before an initial model is sampled and solved.

Parameters:

inistate (int) – State the environment was reset to.

update(state, action, reward, observation)

Update the learner (the Beta and Dirichlet posteriors) with one transition (one step of the current policy).

Parameters:

• state (int) – State the action was taken in.

• action (int) – Action taken.

• reward (float) – Observed reward, treated as Bernoulli: it is added to the success count and its complement to the failure count. Rewards outside \([0, 1]\) therefore corrupt the posterior.

• observation (int) – State reached; increments that transition’s pseudo-count.

updateN()

Auxiliary function to update N the current state-action count.

Adds vk into Nk and zeroes vk for the next episode.