Dueling Bandits

Chapter II: Background

2.5 Dueling Bandits

In a number of real-world situations, absolute numerical feedback is unavailable. For instance, as discussed in Chapter 1, humans often cannot reliably specify numerical scores. Yet, in many cases, people can more accurately provide various forms of qualitative feedback, and in particular, much of this thesis focuses on online learning from pairwise preferences. In the bandit setting, preference-based learning is called thedueling bandits problemand was first formalized in Yue, Broder, et al.

(2012). The dueling bandit setting is a sequential optimization problem, in which the learning agent selects actions and receives relative pairwise preference feedback between them. In other words, the agent poses queries of the form, “Do you prefer A or B?” The dueling bandits paradigm formalizes the problem of online regret minimization through preference-based feedback.

The dueling bandits setting is defined by a tuple, (A, 𝜙), in whichA is the set of possible actions and 𝜙 is a function defining the feedback generation mechanism.

The setting proceeds in a sequence of iterations or rounds. In each iteration𝑖, the algorithm chooses a pair of actions 𝒙𝑖1,𝒙𝑖2 ∈ A to duel or compare, and observes the outcome of that comparison (𝒙𝑖1and𝒙𝑖2can be identical).3The outcome of each duel between𝒙,𝒙⁰ ∈ Arepresents a preference between them, and is an independent sample of a Bernoulli random variable. We denote a preference for action 𝒙 over action𝒙⁰as𝒙 𝒙⁰, and define the probability that𝒙is preferred to 𝒙⁰as:

𝑃(𝒙 𝒙⁰):=𝜙(𝒙,𝒙⁰) + 1 2,

where 𝜙(𝒙,𝒙⁰) ∈ [−1/2,1/2] denotes the stochastic preference between𝒙 and𝒙⁰, so that𝑃(𝒙 𝒙⁰) > ¹

2 if and only if𝜙(𝒙,𝒙⁰) > 0.

In each learning iteration 𝑖, the preference for the 𝑖^th query is denoted by 𝑦_𝑖 :=

I[𝒙𝑖2𝒙𝑖1]−¹₂ ∈

−¹₂,¹

2 , whereI[·]denotes the indicator function, so that𝑃

𝑦_𝑖= ¹₂

=

3The actions are bolded because they could be represented by 𝑑-dimensional vectors (i.e., A ⊂R^𝑑); however, such a representation is not required.

1− 𝑃

𝑦_𝑖 =−¹

2

= 𝑃(𝒙𝑖2 𝒙𝑖1) = 𝜙(𝒙𝑖2,𝒙𝑖1) + ¹

2. The outcome 𝑦_𝑖 is always a binary preference (not a tie), but preference feedback need not be received for every preference query.

Bounding the regret of a bandit algorithm requires some assumptions on the tuple (A, 𝜙). For instance, ifAis an infinite set and the outcomes associated with different actions are independent—that is, observing𝒙 𝒙⁰yields no information about any action𝒙⁰⁰ ∉{𝒙,𝒙⁰}—then, it would be impossible to achieve sublinear regret. Thus, eitherAmust be finite, or there must be structural dependencies between outcomes associated with different actions.4

In the dueling bandits setting, the agent’s decision-making quality can be quantified using a notion of cumulative regret in terms of the preference probabilities:

Reg^DB𝜙 (𝑇) =

𝑇

Õ

𝑖=1

[𝜙(𝒙^∗,𝒙𝑖1) +𝜙(𝒙^∗,𝒙𝑖2)], (2.15) where𝒙^∗is the optimal action. Importantly, this regret definition assumes the exis- tence of a Condorcet winner, that is, a unique action𝒙^∗superior to all other actions (Sui, Zoghi, et al., 2018); formally,𝒙^∗ ∈ Ais a Condorcet winner if𝑃(𝒙^∗ 𝒙) > ¹

2

for any𝒙^∗ ≠ 𝒙 ∈ A. When the learning algorithm has converged to the best action 𝒙^∗, then it can simply duel 𝒙^∗ against itself, incurring no additional regret. Intu- itively, one can interpret Eq. (2.15) as a measure of how much the user(s) would have preferred the best action over the ones presented by the algorithm.

The Multi-Dueling Bandits Setting

The multi-dueling bandits setting (Brost et al., 2016; Sui, Zhuang, et al., 2017) generalizes the dueling bandits setting such that in each iteration, the algorithm can select more than 2 actions. More specifically, in the𝑖^th iteration, the algorithm selects a set𝑆_𝑖 ⊂ Aof actions, and observes outcomes of duels between some pairs of the actions in𝑆_𝑖. These observed preferences could correspond to all action pairs in𝑆_𝑖or to any subset. The number of actions selected in each iteration is assumed to be a fixed constant,𝑚 := |𝑆_𝑖|. When𝑚 =2, the problem reduces to the original dueling bandits setting. For any 𝑚 ≥ 2, the dueling bandits regret formulation in Eq. (2.15) extends to:

Reg^MDB𝜙 (𝑇)=

𝑇

Õ

𝑖=1

Õ

𝒙∈𝑆_𝑖

𝜙(𝒙^∗,𝒙). (2.16)

4Both of these conditions can also be true simultaneously.

The learning algorithm seeks action subsets𝑆_𝑖that minimize the cumulative regret in Eq. (2.16). Intuitively, the algorithm must explore the action space while aiming to minimize the number of times that it selects suboptimal actions. As the algorithm converges to the best action 𝒙^∗, then it increasingly chooses 𝑆_𝑖 to only contain 𝒙^∗, thus incurring no additional regret.

The multi-dueling setting could correspond to a number of different feedback col- lection mechanisms. For example, in some applications it may be viable to collect pairwise feedback between all pairs of actions in𝑆_𝑖. In other applications, it is more realistic to observe a subset of these preferences, for example the “winner” of 𝑆_𝑖, that is, a preference set of the form,{𝒙 𝒙⁰ | 𝒙⁰∈ 𝑆_𝑖\𝒙}for some𝒙 ∈ 𝑆_𝑖.

Algorithms for the Dueling (and Multi-Dueling) Bandits Setting

A number of algorithms have been proposed for the preference-based bandit setting, many of which are equipped with regret bounds. Earlier work on dueling bandits considered an “explore-then-exploit” algorithmic structure; examples include the Interleaved Filter (Yue, Broder, et al., 2012), Beat-the-Mean (Yue and Joachims, 2011), and SAVAGE (Urvoy et al., 2013) algorithms. Confidence-based approaches include RUCB (Zoghi, Whiteson, Munos, et al., 2014), MergeRUCB (Zoghi, White- son, and Rijke, 2015), and CCB (Zoghi, Karnin, et al., 2015). The RMED algorithm (Komiyama et al., 2015), meanwhile, defines Bernoulli distributions whose param- eters are actions’ probabilities of being preferred to one another. The approach calculates Kullback-Leibler divergences between these distributions, and uses them to estimate each action’s probability of being the Condorcet winner and to decide whether actions have been sufficiently explored. RMED is optimal in the sense that when a Condorcet winner exists, the constant factor in its regret matches the regret lower bound for the dueling bandits problem.

Ailon, Karnin, and Joachims (2014) propose several strategies for reducing the dueling bandits problem to a standard multi-armed bandit problem with numerical feedback. In particular, this work introduces the Sparring method, in which two multi-armed bandit algorithms are trained in parallel. These two algorithms could be instances of any standard multi-armed bandit algorithm, for instance RUCB or RMED. In every iteration, each of the two algorithms selects an action; let 𝒙𝑖1and 𝒙𝑖2be the actions selected by algorithms 1 and 2 in iteration𝑖, respectively. These two actions are dueled against each other. If 𝒙₁ wins, then algorithm 1 receives a reward of 1 and algorithm 2 receives a reward of 0; conversely, if 𝒙₂ wins, then

Algorithm 3IndependentSelfSparring (Sui, Zhuang, et al., 2017)

1: Input:𝑚 =the number of actions taken in each iteration,𝜂=the learning rate

2: For each action𝑎 =1,2,· · · , 𝐴, set𝑊_𝑎 =0,𝐹_𝑎 =0.

3: for𝑖 =1,2, . . .do

4: for 𝑗 =1, . . . , 𝑚do

5: For each action𝑎 =1,2,· · · , 𝐴, sample𝜃_𝑎 ∼ Beta(𝑊_𝑎+1, 𝐹_𝑎+1)

6: Select𝑎_𝑗(𝑖) :=argmax𝑎𝜃_𝑎

7: end for

8: Execute𝑚actions:{𝑎₁(𝑖), . . . , 𝑎_𝑚(𝑖)}

9: Observe pairwise feedback matrix 𝑅∈ {0,1,∅}^𝑚×𝑚

10: for 𝑗 , 𝑙 =1, . . . , 𝑚do

11: if𝑟_{𝑗 𝑙} ≠ ∅then

12: 𝑊_𝑎

𝑗(𝑖) ←𝑊_𝑎

𝑗(𝑖)+𝜂·𝑟_{𝑗 𝑙},𝐹_𝑎

𝑗(𝑖) ← 𝐹_𝑎

𝑗(𝑖)+𝜂(1−𝑟_{𝑗 𝑙})

13: end if

14: end for

15: end for

algorithm 1 receives a reward of 0 and algorithm 2 receives a reward of 1.

There are also a number of posterior sampling-based strategies for the dueling bandit problem. The Relative Confidence Sampling algorithm, for instance, on each iteration selects one action via posterior sampling and another via an upper confidence bound strategy (Zoghi, Whiteson, De Rijke, et al., 2014). Meanwhile, the Double Thompson Sampling algorithm (Wu and Liu, 2016) selects both actions in each iteration via posterior sampling; however, each action is selected from a restricted set. The first action is selected among those thought to be relatively well performing, while the second is chosen among actions with less certain priors.

In particular, SelfSparring (Sui, Zhuang, et al., 2017) is a state-of-the-art posterior sampling framework for preference-based learning in the multi-dueling bandits setting, inspired by Sparring (Ailon, Karnin, and Joachims, 2014). SelfSparring maintains a distribution over the probability of selecting each action, and updates it during each iteration based on the observed preferences. Actions are selected by drawing independent samples from this probability distribution, and for each sample, selecting the action with the highest sampled reward. Intuitively, a flatter sampling distribution results in more exploration, while more peaked sampling distributions reflect certainty about the optimal action and yield more exploitation.

Algorithm 3 describes SelfSparring for the case of a finite action set (|A | = 𝐴) with independent actions, that is, observing a preference𝒙 𝒙⁰does not yield information about the performance of any action 𝒙⁰⁰ ∉ {𝒙,𝒙⁰}. SelfSparring maintains a Beta

posterior for each action, and updates it similarly to the Bernoulli bandit posterior sampling algorithm (Algorithm 1). For each action 𝑎, the variables 𝑊_𝑎 and 𝐹_𝑎, respectively, record the number of times that the action wins and loses against any other action (possibly scaled by a learning rate, 𝜂). Actions are then selected by drawing a sample for each action𝑎 ∈ A, 𝜃_𝑎 ∼ Beta(𝑊_𝑎+1, 𝐹_𝑎+1), and choosing the action for which𝜃_𝑎is maximized. Sui, Zhuang, et al. (2017) derive a no-regret guarantee for Algorithm 3 when the actions have a total ordering with respect to the preferences.

Additionally, Sui, Zhuang, et al. (2017) propose a kernel-based variant of Self- Sparring called KernelSelfSparring, which leverages Gaussian process regression to model dependencies among rewards corresponding to different actions. Other dueling bandit approaches which handle structured action spaces include the Duel- ing Bandit Gradient Descent algorithm for convex utility functions over continuous action spaces (Yue and Joachims, 2009); the Correlational Dueling Bandits al- gorithm, in which a large number of actions have a low-dimensional correlative structure based on a given similarity function (Sui, Yue, and Burdick, 2017); and the StageOpt algorithm for safe learning with Gaussian process dueling bandits (Sui, Zhuang, et al., 2018).

The multi-dueling bandits setting is also considered in Brost et al. (2016), which proposes a confidence interval strategy for estimating preference probabilities.

Related Problem Settings

Aside from pairwise preferences, a number of other forms of qualitative reward feedback have also been studied. For instance, in the coactive feedback problem (Shivaswamy and Joachims, 2015; Shivaswamy and Joachims, 2012; Raman et al., 2013), the agent selects an action 𝒂𝑖 in the 𝑖^th iteration, and the environment reveals an action 𝒂⁰_𝑖 which is preferred to 𝒂𝑖. With ordinal feedback, meanwhile, the user assigns each trial to one of a discrete set of ordered categories (Chu and Ghahramani, 2005a; Herbrich, Graepel, and Obermayer, 1999). In the “learning to rank” problem, an agent learns a ranking function for information retrieval using feedback such as users’ clickthrough data (Burges, Shaked, et al., 2005; Radlinski and Joachims, 2005; Burges, Ragno, and Le, 2007; Liu, 2009; Schuth, Sietsma, et al., 2014; Schuth, Oosterhuis, et al., 2016; Yue, Finley, et al., 2007).