Asymptotic Consistency of the Transition Dynamics in DPS in the

Chapter VI: Conclusions and Future Directions

B.2 Asymptotic Consistency of the Transition Dynamics in DPS in the

B.1 Facts about Convergence in Distribution

Before proceeding with the asymptotic consistency proofs, two facts about conver- gence in distribution are reviewed; these will be applied later.

Recall that for a random variable𝑋and a sequence of random variables(𝑋_𝑛),𝑛 ∈N, 𝑋_𝑛

−→𝐷 𝑋 denotes that𝑋_𝑛 converges to 𝑋 in distribution, while 𝑋_𝑛

−→𝑃 𝑋 denotes that𝑋_𝑛converges to 𝑋 in probability.

Fact 8(Billingsley, 1968). For random variables𝒙,𝒙𝑛,∈R^𝑑, where𝑛∈N, and any continuous function𝑔:R^𝑑 −→ R, if𝒙𝑛

−→𝐷 𝒙, then𝑔(𝒙𝑛)−→^𝐷 𝑔(𝒙).

Fact 9 (Billingsley, 1968). For random variables 𝒙𝑛 ∈ R^𝑑, 𝑛 ∈ N, and constant vector 𝒄 ∈ R^𝑑, 𝒙𝑛

−→𝐷 𝒄 is equivalent to 𝒙𝑛

−→𝑃 𝒄. Convergence in probability means that for any𝜀 >0, 𝑃(||𝒙𝑛−𝒄||₂ ≥ 𝜀) −→0as𝑛−→ ∞.

B.2 Asymptotic Consistency of the Transition Dynamics in DPS in the Preference-

dynamics parameters, respectively (hiding the dependency on the DPS episode𝑖1 or 𝑖2 for the latter three quantities); thus, [𝒑^(𝑗)]𝑘 denotes the true probability of transitioning from state-action pair ˜𝑠_𝑗 to the 𝑘^th state, and analogously for the 𝑘^th elements of 𝒑˜^(𝑗), 𝒑ˆ^(𝑗), and 𝒑ˆ^0(𝑗). Then, from the Dirichlet model,

[𝒑ˆ^(𝑗)]𝑘 =

𝑛_{𝑗 𝑘} +𝛼_{𝑗 𝑘 ,0} 𝑛_𝑗+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0} ,

where the prior for 𝒑^(𝑗)isÍ𝑆 ¹ 𝑚=1𝛼𝑗 𝑚,0

[𝛼_𝑗1,0, . . . , 𝛼_{𝑗 𝑆,0}]^𝑇 for user-defined hyperparam- eters𝛼_{𝑗 𝑘 ,0}> 0. Meanwhile, the maximum likelihood is given by[𝒑ˆ^0(𝑗)]𝑘 = _max(𝑛^{𝑛𝑗 𝑘}

𝑗,1)

(this is equivalent to[𝒑ˆ^(𝑗)]𝑘, except with the prior parameters set to zero). Consider the sampled dynamics at state-action pair ˜𝑠_𝑗. For any𝜀 >0,

𝑃 ||𝒑˜^(𝑗)− 𝒑^(𝑗)||₁ ≥ 𝜀

= 𝑃 ||𝒑˜^(𝑗)− 𝒑ˆ^(𝑗) + 𝒑ˆ^(𝑗) − 𝒑ˆ^0(𝑗)+ 𝒑ˆ^0(𝑗) − 𝒑^(𝑗)||₁≥ 𝜀

(𝑎)

≤ 𝑃 ||𝒑˜^(𝑗) − 𝒑ˆ^(𝑗)||₁+ ||𝒑ˆ^(𝑗) − 𝒑ˆ^0(𝑗)||₁+ ||𝒑ˆ^0(𝑗)− 𝒑^(𝑗)||₁ ≥ 𝜀

≤ 𝑃

||𝒑˜^(𝑗) − 𝒑ˆ^(𝑗)||₁ ≥ 𝜀 3

Ø ||𝒑ˆ^(𝑗)− 𝒑ˆ^0(𝑗)||₁ ≥ 𝜀 3

Ø ||𝒑ˆ^0(𝑗) − 𝒑^(𝑗)||₁≥ 𝜀 3

(𝑏)

≤ 𝑃

||𝒑˜^(𝑗) − 𝒑ˆ^(𝑗)||₁ ≥ 𝜀 3

+𝑃

||𝒑ˆ^(𝑗) − 𝒑ˆ^0(𝑗)||₁≥ 𝜀 3

+𝑃

||𝒑ˆ^0(𝑗) − 𝒑^(𝑗)||₁ ≥ 𝜀 3

, (B.2) where (a) holds due to the triangle inequality and (b) follows from the union bound.

This proof will upper-bound each term in Eq. (B.2) in terms of𝑛_𝑗 and show that it decays as𝑛_𝑗 −→ ∞, that is, as ˜𝑠_𝑗 is visited infinitely often. For the first term, this bound is achieved via Chebyshev’s inequality:

𝑃

||𝒑˜^(𝑗) − 𝒑ˆ^(𝑗)||₁ ≥ 𝜀 3

≤ 𝑃

𝑆

Ø

𝑘=1

n

[𝒑˜^(𝑗)]𝑘 − [𝒑ˆ^(𝑗)]𝑘

≥ 𝜀

3𝑆 o

!

(𝑎)

≤

𝑆

Õ

𝑘=1

𝑃

[𝒑˜^(𝑗)]𝑘 − [𝒑ˆ^(𝑗)]𝑘

≥ 𝜀

3𝑆 (𝑏)

≤

𝑆

Õ

𝑘=1

9𝑆² 𝜀²

Varh

[𝒑˜^(𝑗)]𝑘

i ,

where (a) follows from the union bound and (b) is an application of Chebyshev’s inequality. For a Dirichlet variable𝑋 with parameters(𝛼₁, . . . , 𝛼_𝑆),𝛼_𝑘 >0 for each 𝑘, the variance of the𝑘^thcomponent𝑋_𝑘 is given by:

Var[𝑋_𝑘] = 𝛼˜_𝑘(1−𝛼˜_𝑘) 1+Í^𝑆

𝑚=1𝛼_𝑚

≤ 1

2∗ 1

1+Í^𝑆

𝑚=1𝛼_𝑚 ,

where ˜𝛼_𝑘 := Í𝑆^𝛼𝑘 𝑚=1𝛼𝑚

. In the DPS algorithm, 𝒑˜^(𝑗) is drawn from a Dirichlet distribu-

tion with parameters (𝛼_𝑗1, . . . , 𝛼_{𝑗 𝑆}) =(𝛼_𝑗1,0+𝑛_𝑗1, . . . , 𝛼_{𝑗 𝑆,0}+𝑛_{𝑗 𝑆}), so that, Varh

[𝒑˜^(𝑗)]𝑘

i

≤ 1

2 ∗ 1

1+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚}

= 1

2 ∗ 1

1+Í^𝑆

𝑚=1(𝛼_{𝑗 𝑚,0}+𝑛_{𝑗 𝑚})

≤ 1

2 ∗ 1

1+Í𝑆 𝑚=1𝑛_{𝑗 𝑚}

= 1

2(1+𝑛_𝑗). Thus,

𝑃 ||𝒑˜^(𝑗)− 𝒑ˆ^(𝑗)||₁ ≥ 𝜀 3

!

≤

𝑆

Õ

𝑘=1

9𝑆² 𝜀²

1

2(1+𝑛_𝑗) = 9𝑆³ 2𝜀²(1+𝑛_𝑗). Considering the second term in Eq. (B.2),

𝑃 ||𝒑ˆ^(𝑗) − 𝒑ˆ^0(𝑗)||₁ ≥ 𝜀 3

!

≤ 𝑃

𝑆

Ø

𝑘=1

n

[𝒑ˆ^(𝑗)− 𝒑ˆ^0(𝑗)]𝑘

≥ 𝜀

3𝑆 o

!

(𝑎)

≤

𝑆

Õ

𝑘=1

𝑃

[𝒑ˆ^(𝑗)]𝑘 − [𝒑ˆ^0(𝑗)]𝑘

≥ 𝜀

3𝑆 (𝑏)

≤

𝑆

Õ

𝑘=1

𝑃

𝛼_{𝑗 𝑘 ,0}+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0} 𝑛_𝑗 +Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0}

≥ 𝜀 3𝑆

! ,

where (a) holds via the union bound and (b) follows for𝑛_𝑗 ≥1 because when𝑛_𝑗 ≥ 1:

[𝒑ˆ^(𝑗)]𝑘− [𝒑ˆ^0(𝑗)]𝑘

=

𝑛_{𝑗 𝑘} +𝛼_{𝑗 𝑘 ,0} 𝑛_𝑗+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0}

− 𝑛_{𝑗 𝑘} 𝑛_𝑗

=

𝛼_{𝑗 𝑘 ,0} 𝑛_𝑗+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0}

−

𝑛_{𝑗 𝑘}Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0} 𝑛_𝑗(𝑛_𝑗+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0})

≤

𝛼_{𝑗 𝑘 ,0} 𝑛_𝑗 +Í𝑆

𝑚=1𝛼_{𝑗 𝑚,0} +

𝑛_{𝑗 𝑘} 𝑛_𝑗

Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0} 𝑛_𝑗 +Í𝑆

𝑚=1𝛼_{𝑗 𝑚,0}

≤

𝛼_{𝑗 𝑘 ,0}+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0} 𝑛_𝑗+Í𝑆

𝑚=1𝛼_{𝑗 𝑚,0} .

For the third term in Eq. (B.2), one can apply the following concentration inequality for Dirichlet variables (see Appendix C.1 in Jaksch, Ortner, and Auer, 2010):

𝑃(||𝒑ˆ^0(𝑗)− 𝒑^(𝑗)||₁ ≥ 𝜀) ≤ (2^𝑆−2)exp −𝑛_𝑗𝜀² 2

! . Therefore:

𝑃

||𝒑ˆ^0(𝑗) − 𝒑^(𝑗)||₁≥ 𝜀 3

≤ (2^𝑆−2)exp −𝑛_𝑗𝜀² 18

! . Thus, to upper-bound the right-hand side of Eq. (B.2), for any𝜀 > 0:

𝑃 ||𝒑˜^(𝑗)−𝒑^(𝑗)||₁ ≥ 𝜀

≤ 9𝑆³ 2𝜀²(𝑛_𝑗 +1)+

𝑆

Õ

𝑘=1

𝑃

𝛼_{𝑗 𝑘 ,0}+Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0} 𝑛_𝑗 +Í^𝑆

𝑚=1𝛼_{𝑗 𝑚,0}

≥ 𝜀 3𝑆

!

+(2^𝑆−2)exp −𝑛_𝑗𝜀² 18

! .

On the right hand side, the first and third terms clearly decay as 𝑛_𝑗 −→ ∞. The middle term is identically zero for𝑛_𝑗 large enough, since the𝛼_{𝑗 𝑘 ,0}values are user- defined constants. Given this inequality, it is clear that for any𝜀 > 0, as𝑛_𝑗 −→ ∞,

𝑃 ||𝒑˜^(𝑗) − 𝒑^(𝑗)||₁ ≥ 𝜀

−→ 0. If every state-action pair is visited infinitely often, then 𝑛_𝑗 −→ ∞ for each 𝑗, and therefore, 𝒑˜^(𝑗) converges in probability to 𝒑^(𝑗): 𝒑˜^(𝑗) −→^𝑃 𝒑^(𝑗). Convergence in probability implies convergence in distribution, the desired result.

To continue proving that DPS’s model of the transition dynamics converges, this analysis uses that the magnitude of the utility estimator||𝒓ˆ𝑛||₂, the mean of the utility posterior sampling distribution, is uniformly upper-bounded; in other words, there exists𝑏 < ∞such that ||𝒓ˆ𝑛||₂ ≤ 𝑏.

Lemma 4. When preferences are given by a linear or logistic link function, across all 𝑛 ≥ 1, there exists some 𝑏 < ∞ such that estimated reward at DPS trial 𝑛 is bounded by𝑏:||𝒓ˆ𝑛||₂ ≤ 𝑏.

Proof. Firstly, if the link function is logistic, the desired result holds automatically by the definition of𝒓ˆ𝑛given in Eq. (4.10): the quantity is projected onto the compact setΘ ⊂ R^𝑑 of all possible values of𝒓, and a compact set onR^𝑑must be bounded.

Secondly, the result is proven in the case of a linear link function. In this case, recall that the MAP reward estimate𝒓ˆ𝑛is the solution to a ridge regression problem:

𝒓ˆ𝑛=arg inf_𝒓 (_𝑛−1

Õ

𝑖=1

(𝒙^𝑇_𝑖 𝒓−𝑦_𝑖)²+𝜆||𝒓||²₂ )

=arg inf_𝒓 (_𝑛−1

Õ

𝑖=1

(𝒙^𝑇_𝑖 𝒓−𝑦_𝑖)²+ 1

𝑛−1𝜆||𝒓||²₂ )

. (B.3) The desired result is proven by contradiction. Assuming that there exists no upper bound𝑏, the proof will identify a subsequence(𝒓ˆ𝑛𝑖)of MAP estimates whose lengths increase unboundedly, but whose directions converge. Then, it will show that such vectors fail to minimize the objective in Eq. (B.3), achieving a contradiction.

Firstly, the vectors 𝒙𝑖 = 𝒙𝑖2 − 𝒙𝑖1 have bounded magnitude: in the bandit case, 𝒙𝑖1,𝒙𝑖2∈ A, and the action spaceAis compact, while in the RL setting,||𝒙𝑖 𝑗||₁= ℎ for 𝑗 ∈ {1,2}. The binary labels𝑦_𝑖are also bounded, as they take values in

−¹

2,¹

2 . Note that for 𝒓 = 0, (𝒙^𝑇_𝑖 𝒓− 𝑦_𝑖)²+ _𝑛−1¹ 𝜆||𝒓||²₂ = ¹₄. The desired statement is proven by contradiction: assume that there is no 𝑏 < ∞ such that ||𝒓ˆ𝑛||₂ ≤ 𝑏 for all 𝑛. Then, the sequence 𝒓ˆ₁,𝒓ˆ₂, . . . must have a subsequence indexed by (𝑛_𝑖) such that lim𝑖−→∞||𝒓ˆ𝑛𝑖||₂ = ∞. Consider the sequence of unit vectors _||𝒓ˆ^{𝒓ˆ𝑛𝑖}

𝑛𝑖||₂. This sequence lies within the compact set of unit vectors in R^𝑑, so it must have a convergent subsequence; we index this subsequence of the sequence (𝑛_𝑖) by (𝑛_𝑖

𝑗). Then, the

sequence(𝒓ˆ𝑖𝑗) is such that lim𝑗−→∞||𝒓ˆ𝑖𝑗||₂ =∞and lim𝑗−→∞

𝒓ˆ𝑖 𝑗

||𝒓ˆ𝑖

𝑗||₂ = 𝒓ˆ𝑢𝑛𝑖𝑡, where 𝒓ˆ𝑢𝑛𝑖𝑡 ∈R^𝑑 is a fixed unit vector.

For any 𝒙𝑖 such that |𝒙_𝑖^𝑇𝒓ˆ𝑢𝑛𝑖𝑡| ≠ 0, lim𝑛𝑖

𝑗−→∞(𝒙^𝑇_𝑖 𝒓ˆ𝑛𝑖 𝑗

− 𝑦_𝑖)² = ∞, and thus, the corresponding terms in Eq. (B.3) approach infinity. However, a lower value of the optimization objective in Eq. (B.3) can be realized by replacing 𝒓ˆ𝑛_𝑖

𝑗 with the assignment 𝒓 = 0. Meanwhile, for any 𝒙𝑖 such that |𝒙^𝑇_𝑖 𝒓ˆ| = 0, replacing 𝒓ˆ𝑛_𝑖

𝑗 with

𝒓 = 0 would also decrease the value of the optimization objective in Eq. (B.3).

Therefore, for large 𝑗, 𝒓 =0results in a smaller objective function value than𝒓ˆ𝑛𝑖 𝑗. This is a contradiction, proving that the elements of the sequence 𝒓ˆ𝑛𝑖

𝑗 cannot have arbitrarily large magnitudes. Thus, the elements of the original sequence 𝒓ˆ𝑖 also cannot become arbitrarily large, and||𝒓ˆ𝑖|| ≤ 𝑏 for some𝑏 < ∞.

The next intermediate result relates the matrix ˜𝑀_𝑛, defined in Eq. (B.1), and the matrix 𝑀_𝑛 =𝜆 𝐼+Í^𝑛−1

𝑖=1 𝒙𝑖𝒙_𝑖^𝑇.

Lemma 5. On iteration𝑛of DPS, the posterior covariance matrix for the rewards is Σ^(𝑛) = 𝛽_𝑛(𝛿)²𝑀˜_𝑛; if the link function𝑔is linear, then𝑀˜_𝑛 =𝑀_𝑛, while if𝑔is logistic, then 𝑀˜_𝑛 = 𝜆 𝐼 + Í𝑛−1

𝑖=1 𝑔˜(2𝑦_𝑖𝒙_𝑖^𝑇𝒓ˆ𝑛)𝒙𝑖𝒙^𝑇_𝑖. In both cases, there exist two constants 𝑚_min, 𝑚_maxsuch that0 < 𝑚_min ≤ 𝑚_max< ∞and𝑚_min𝑀_𝑛 𝑀˜_𝑛 𝑚_max𝑀_𝑛.

Proof. Firstly, if𝑔 is linear, then ˜𝑀_𝑛 = 𝑀_𝑛, so the desired result clearly holds with 𝑚_min =𝑚_max =1.

If𝑔is logistic, the desired statement is equivalent to:

𝑚_min 𝜆 𝐼+

𝑛−1

Õ

𝑖=1

𝒙𝑖𝒙^𝑇𝑖

!

𝜆 𝐼+

𝑛−1

Õ

𝑖=1

˜

𝑔(2𝑦_𝑖𝒙^𝑇𝑖 𝒓ˆ𝑛)𝒙𝑖𝒙^𝑇𝑖 𝑚_max 𝜆 𝐼+

𝑛−1

Õ

𝑖=1

𝒙𝑖𝒙^𝑇𝑖

! .

By definition of ˜𝑔, ˜𝑔(𝑥) ∈ (0,∞) for all 𝑥 ∈ R. Moreover, the domain of ˜𝑔 has bounded magnitude, since all possible inputs to ˜𝑔are of the form 2𝑦_𝑖𝒙^𝑇_𝑖 𝒓ˆ𝑛, in which

|𝑦_𝑖| = ¹₂, 𝒙𝑖 belongs to a compact set, and ||𝒓ˆ𝑛|| ≤ 𝑏 by Lemma 4. Therefore, all possible inputs to ˜𝑔belong to a compact set. A continuous function over a compact set always attains its maximum and minimum values; therefore, there exist values

˜

𝑔_min,𝑔˜_max such that 0 < 𝑔˜_min ≤ 𝑔˜(𝑥) ≤ 𝑔˜_max < ∞for all possible inputs𝑥 to ˜𝑔(𝑥).

Therefore, 𝜆 𝐼+

𝑛−1

Õ

𝑖=1

˜

𝑔(2𝑦_𝑖𝒙^𝑇_𝑖 𝒓ˆ𝑛)𝒙𝑖𝒙^𝑇_𝑖 𝜆 𝐼+

𝑛−1

Õ

𝑖=1

˜

𝑔_min𝒙𝑖𝒙^𝑇_𝑖 min{𝑔˜_min,1}

"

𝜆 𝐼+

𝑛−1

Õ

𝑖=1

𝒙𝑖𝒙^𝑇_𝑖

# , and

𝜆 𝐼+

𝑛−1

Õ

𝑖=1

˜

𝑔(2𝑦_𝑖𝒙^𝑇_𝑖 𝒓ˆ𝑛)𝒙𝑖𝒙^𝑇_𝑖 𝜆 𝐼+

𝑛−1

Õ

𝑖=1

˜

𝑔_max𝒙𝑖𝒙^𝑇_𝑖 max{𝑔˜_max,1}

"

𝜆 𝐼+

𝑛−1

Õ

𝑖=1

𝒙𝑖𝒙^𝑇_𝑖

# ,

which proves the desired result for𝑚_min =min{𝑔˜_min,1}and𝑚_max =max{𝑔˜_max,1}.

To finish proving convergence of the transition dynamics Bayesian model, Lemma 6 demonstrates that every state-action pair is visited infinitely often.

Lemma 6. Under DPS with preference-based RL, assume that the dynamics are modeled via a Dirichlet model and that the utilities are modeled either via the linear or logistic link functions, with posterior sampling distributions given in Eq.(4.13).

Then, every state-action pair is visited infinitely often.

This consistency result also holds when removing the𝛽_𝑛(𝛿)and𝛽⁰

𝑛(𝛿)factors from the distributions in Eq.(4.13).

Proof. The proof proceeds by assuming that there exists a state-action pair that is visited only finitely-many times. This assumption will lead to a contradiction1: once this state-action pair is no longer visited, the posterior sampling distribution for the utilities 𝒓 is no longer updated with respect to it. Then, DPS is guaranteed to eventually sample a high enough reward for this state-action that the resultant policy will prioritize visiting it.

First, note that DPS is guaranteed to reach at least one state-action pair infinitely often: given the problem’s finite state and action spaces, at least one state-action pair must be visited infinitely often during DPS execution. If all state-actions arenot visited infinitely often, there must exist a state-action pair(𝑠, 𝑎)such that𝑠is visited infinitely often, while (𝑠, 𝑎) is not. Otherwise, if all actions are selected infinitely often in all infinitely-visited states, the finitely-visited states are unreachable (in which case these states are irrelevant to the learning process and regret minimization,

1Note that in finite-horizon MDPs, the concept of visiting a state finitely-many times is not the same as that of a transient state in an infinite Markov chain, because: 1) due to a finite horizon, the state is resampled from the initial state distribution 𝑝₀(𝑠) everyℎ timesteps, and 2) the policy—

which determines which state-action pairs can be reached in an episode—is also resampled everyℎ timesteps.

and can be ignored). Without loss of generality, this state-action pair(𝑠, 𝑎)is labeled as ˜𝑠₁. To reach a contradiction, it suffices to show that ˜𝑠₁is visited infinitely often.

Let 𝒓₁ be the utility vector with a reward of 1 in state-action pair ˜𝑠₁ and rewards of zero elsewhere. From Definition 6, 𝜋_𝑣𝑖(𝒑˜,𝒓₁) is the policy that maximizes the expected number of visits to ˜𝑠₁under dynamics 𝒑˜and utility vector𝒓₁:

𝜋_𝑣𝑖(𝒑˜,𝒓₁) =argmax𝜋𝑉(𝒑˜,𝒓₁, 𝜋),

where𝑉(𝒑˜,𝒓₁, 𝜋) is the expected total reward of a length-ℎtrajectory under 𝒑˜,𝒓₁, and𝜋, or equivalently (by definition of 𝒓₁), the expected number of visits to state- action ˜𝑠₁.

Next, it will be shown that there exists a 𝜌 > 0 such that 𝑃(𝜋 = 𝜋_𝑣𝑖(𝒑˜,𝒓₁)) > 𝜌 for all possible values of 𝒑. That is, for any sampled parameters˜ 𝒑, the probability˜ of selecting policy𝜋_𝑣𝑖(𝒑˜,𝒓₁) is uniformly lower-bounded, implying that DPS must eventually select𝜋_𝑣𝑖(𝒑˜,𝒓₁).

Let ˜𝑟_𝑗 be the sampled utility (also referred to as reward) associated with state- action pair ˜𝑠_𝑗 in a particular DPS episode, for each state-action 𝑗 ∈ {1, . . . , 𝑑}, with 𝑑 = 𝑆 𝐴. The proof will show that conditioned on 𝒑, there exists˜ 𝑣 > 0 such that if ˜𝑟₁ exceeds max{𝑣𝑟˜₂, 𝑣𝑟˜₃, . . . , 𝑣𝑟˜_𝑑}, then value iteration returns the policy 𝜋_𝑣𝑖(𝒑˜,𝒓₁), which is the policy maximizing the expected amount of time spent in

˜

𝑠₁. This can be seen by setting 𝑣 := _𝜌^ℎ

1, where ℎis the time horizon and 𝜌₁ is the expected number of visits to ˜𝑠₁under𝜋_𝑣𝑖(𝒑˜,𝒓₁). Under this definition of𝑣, the event {𝑟˜₁ ≥max{𝑣𝑟˜₂, 𝑣𝑟˜₃, . . . , 𝑣𝑟˜_𝑑}}is equivalent to{𝑟˜₁𝜌₁ ≥ ℎmax{𝑟˜₂,𝑟˜₃, . . . ,𝑟˜_𝑑}}; the latter inequality implies that given 𝒑˜and𝒓, the expected reward accumulated solely˜ in state-action ˜𝑠₁exceeds the reward gained by repeatedly (during all ℎtime-steps) visiting the state-action pair in the set {𝑠˜₂, . . . ,𝑠˜_𝑑} having the highest sampled reward. Clearly, in this situation, value iteration results in the policy𝜋_𝑣𝑖(𝒑˜,𝒓₁).

Next, it is shown that𝑣 = ^ℎ

𝜌₁ is continuous in the sampled dynamics 𝒑˜ by showing that𝜌₁is continuous in 𝒑. Recall that˜ 𝜌₁is defined as expected number of visits to ˜𝑠₁ under𝜋_𝑣𝑖(𝒑˜,𝒓₁). This is equivalent to the expected reward for following 𝜋_𝑣𝑖(𝒑˜,𝒓₁) under dynamics 𝒑˜and rewards 𝒓₁:

𝜌₁ =𝑉(𝒑˜,𝒓₁, 𝜋_𝑣𝑖(𝒑˜,𝒓₁)) =max

𝜋

𝑉(𝒑˜,𝒓₁, 𝜋). (B.4) The value of any policy 𝜋 is continuous in the transition dynamics parameters, so 𝑉(𝒑˜,𝒓₁, 𝜋)is continuous in 𝒑. The maximum in Eq. (B.4) is taken over the finite set˜

of deterministic policies; because a maximum over a finite number of continuous functions is also continuous, 𝜌₁is continuous in 𝒑.˜

Next, recall that a continuous function on a compact set achieves its maximum and minimum values on that set. The set of all possible dynamics parameters 𝒑˜ is such that for each state-action pair 𝑗,Í^𝑆

𝑘=1𝑝_{𝑗 𝑘} =1 and𝑝_{𝑗 𝑘} ≥ 0∀𝑘; the set of all possible vectors 𝒑˜ is clearly closed and bounded, and hence compact. Therefore,𝑣 achieves its maximum and minimum values on this set, and for any 𝒑,˜ 𝑣 ∈ [𝑣_min, 𝑣_max], where 𝑣_min >0 (𝑣is nonnegative by definition, and𝑣 =0 is impossible, as it would imply that ˜𝑠₁is unreachable).

Then, 𝑃(𝜋 = 𝜋_𝑣𝑖(𝒑˜,𝒓₁)) can be expressed in terms of 𝑣 and the parameters of the reward posterior. Firstly,

𝑃(𝜋 =𝜋_𝑣𝑖(𝒑˜,𝒓₁)) ≥ 𝑃(𝑟˜₁ >max{𝑣𝑟˜₂, 𝑣𝑟˜₃, . . . , 𝑣𝑟˜_𝑑}) ≥

𝑑

Ö

𝑗=2

𝑃(𝑟˜₁> 𝑣𝑟˜_𝑗).

In the 𝑛^{𝑡 ℎ} DPS iteration, the sampled rewards are drawn from a jointly Gaussian posterior: 𝒓˜ ∼ N (𝝁^(𝑛),Σ^(𝑛)) for some 𝝁^(𝑛) and Σ^(𝑛), where [𝝁^(𝑛)]𝑗 = 𝜇^(𝑛)

𝑗 and

[Σ^(𝑛)]𝑗 𝑘 = Σ^(𝑛)

𝑗 𝑘 . Then,(𝑟˜₁−𝑣𝑟˜_𝑗) ∼ N (𝜇⁽

𝑛) 1 −𝑣 𝜇⁽

𝑛)

𝑗 , Σ₁₁^(𝑛)+𝑣²Σ⁽_{𝑗 𝑗}^𝑛)−2𝑣Σ⁽₁^𝑛)

𝑗 ), so that:

𝑃(𝜋_𝑛1 =𝜋_𝑣𝑖(𝒑˜,𝒓₁)) ≥

𝑑

Ö

𝑗=2















1−Φ©

­

­

«

−𝜇^(𝑛)

1 +𝑣 𝜇^(𝑛)

𝑗

q Σ^(𝑛)

11 +𝑣²Σ^(𝑛)_{𝑗 𝑗} −2𝑣Σ^(𝑛)

1𝑗

ª

®

®

¬















=

𝑑

Ö

𝑗=2

Φ©

­

­

«

𝜇⁽

𝑛) 1 −𝑣 𝜇⁽

𝑛) 𝑗

q Σ^(𝑛)

11 +𝑣²Σ^(𝑛)

𝑗 𝑗 −2𝑣Σ^(𝑛)

1𝑗

ª

®

®

¬

, (B.5)

where Φis the standard Gaussian cumulative distribution function. For the right- hand expression in Eq. (B.5) to have a lower bound greater than zero, the argument of Φ(·)must be lower-bounded. It suffices to upper-bound the numerator’s magnitude and to lower-bound the denominator above zero for each product factor 𝑗 and over all iterations𝑛.

The numerator can be upper-bounded using Lemma 4. Since 𝝁^(𝑛) equals 𝒓ˆ𝑛 at iteration 𝑛, ||𝝁^(𝑛)||₂ ≤ 𝑏; therefore, |𝜇^(𝑛)

1 |,|𝜇^(𝑛)

𝑗 | ≤ 𝑏. Because 0 < 𝑣 ≤ 𝑣_max,

|𝜇₁−𝑣 𝜇_𝑗| ≤ |𝜇⁽

𝑛)

1 | +𝑣|𝜇⁽

𝑛)

𝑗 | ≤ (1+𝑣_max)𝑏.

To lower-bound the denominator, first note that it is equal toq

𝒘^𝑇_𝑗Σ^(𝑛)𝒘𝑗, in which 𝒘𝑗 ∈R^𝑑is defined as a vector with 1 in the first position,−𝑣in the 𝑗^thposition for

some 𝑗 ∈ {2, . . . , 𝑑}, and zero elsewhere:

𝒘𝑗 :=[1,0, . . . ,0,−𝑣 ,0, . . . ,0]^𝑇. (B.6) Equivalently, it must be shown that 𝒘^𝑇_𝑗Σ^(𝑛)𝒘𝑗 is lower-bounded above zero. By Lemma 5, it holds thatΣ^{(𝑛) 𝛽𝑛(𝛿)2}

𝑚_max

𝑀⁻¹

𝑛 , implying that𝒘^𝑇_𝑗Σ^(𝑛)𝒘𝑗 ≥ ^{𝛽𝑛(𝛿)2}

𝑚_max 𝒘^𝑇_𝑗𝑀⁻¹

𝑛 𝒘𝑗. Because𝑚_maxis a constant and 𝛽_𝑛(𝛿), defined in Eq. (4.4), is non-decreasing in𝑛, it suffices to prove that 𝒘^𝑇_𝑗𝑀⁻¹

𝑛 𝒘𝑗 is lower-bounded above zero. (Thus, the result holds regardless of the presence of 𝛽_𝑛(𝛿)in the utility sampling distribution.) Recall from Definition 7 that the eigenvectors of𝑀⁻¹

𝑛 are𝒖⁽₁^𝑛), . . . ,𝒖^(𝑛)

𝑑 , with corre- sponding eigenvalues

𝜈^(𝑛)

1

−1

, . . . ,

𝜈^(𝑛)

𝑑

−1

. The vector𝒘𝑗can be written in terms of the orthonormal basis formed by the eigenvectors{𝒖^(𝑛)

𝑘 }: 𝒘𝑗 =

𝑑

Õ

𝑘=1

𝛼⁽

𝑛) 𝑘 𝒖^(𝑛)

𝑘

, (B.7)

for some coefficients 𝛼^(𝑛)

𝑘 ∈ R. Using Eq. (B.7), the quantity to be lower-bounded can now be written as:

𝒘^𝑇_𝑗𝑀⁻¹

𝑛 𝒘𝑗 =

𝑑

Õ

𝑘=1

𝛼⁽

𝑛) 𝑘 𝒖^(𝑛)𝑇

𝑘

! _𝑑 Õ

𝑙=1

1 𝜈⁽

𝑛) 𝑙

𝒖^(𝑛)

𝑙 𝒖^(𝑛)𝑇

𝑙

! _𝑑 Õ

𝑚=1

𝛼⁽

𝑛) 𝑚 𝒖⁽𝑚^𝑛)

!

(𝑎)=

𝑑

Õ

𝑘=1

𝛼^(𝑛)

𝑘

2 1 𝜈^(𝑛)

𝑘 (𝑏)

≥

𝛼^(𝑛)

𝑘₀

2 1 𝜈^(𝑛)

𝑘₀

, (B.8)

where equality (a) follows by orthonormality of the eigenvector basis, and (b) holds for any 𝑘₀ ∈ {1, . . . , 𝑑} due to positivity of the eigenvalues (𝜈_𝑘)⁻¹. Therefore, to show that the denominator is bounded away from zero, it suffices to show that for every𝑛, there exists some𝑘₀such that

𝛼^(𝑛)

𝑘₀

2 𝜈^(𝑛)

𝑘₀

−1

is bounded away from zero.

To prove the previous statement, note that by definition of 𝑀_𝑛, the eigenvalues (𝜈^(𝑛)

𝑘 )⁻¹are non-increasing in𝑛. Below, the proof will show that for any eigenvalue (𝜈^(𝑛)

𝑘 )⁻¹ such that lim𝑛−→∞(𝜈^(𝑛)

𝑘 )⁻¹ = 0, the first element of its corresponding eigenvector, h

𝒖^(𝑛)_𝑘 i

1, also converges to zero. Since the first element of𝒘𝑗 equals 1, Eq. (B.6) implies that there must exist some𝑘₀such that

h 𝒖^(𝑛)

𝑘₀

i

1

6

−→0 and𝛼⁽

𝑛) 𝑘₀ is bounded away from 0. If these implications did not hold, then𝒘𝑗 would not have a value of 1 in its first element, contradicting its definition. These observations imply that for every 𝑛, there must be some 𝑘₀ such that as 𝑛 −→ ∞, (𝜈^(𝑛)

𝑘₀ )⁻¹ −→6 0 and 𝛼⁽

𝑛)

𝑘₀ is bounded away from zero.

Let𝑋_𝑛denote the observation matrix after𝑛−1 observations:𝑋_𝑛 := h

𝒙₁ . . . 𝒙𝑛−1

i^𝑇 . Then, 𝑀⁻¹

𝑛 = (𝑋^𝑇

𝑛𝑋_𝑛+𝜆 𝐼)⁻¹. The matrices 𝑀⁻¹

𝑛 and 𝑋^𝑇

𝑛𝑋_𝑛 have the same eigen- vectors. Meanwhile, for each eigenvalue (𝜈⁽

𝑛)

𝑖 )⁻¹of 𝑀⁻¹

𝑛 , 𝑋^𝑇

𝑛𝑋_𝑛 has an eigenvalue 𝜉^(𝑛)

𝑖 :=𝜈^(𝑛)

𝑖 −𝜆≥ 0 corresponding to the same eigenvector. We aim to characterize the eigenvectors of𝑀⁻¹

𝑛 whose eigenvalues approach zero. Since these eigenvectors are identical to those of𝑋^𝑇

𝑛𝑋_𝑛whose eigenvalues approach infinity, the latter can be considered instead.

Without loss of generality, assume that all finitely-visited state-action pairs (in- cluding ˜𝑠₁) occur in the first 𝑚 < 𝑛−1 iterations, and index these finitely-visited state-action pairs from 1 to 𝑟 ≥ 1, so that the finitely-visited state-actions are:

{𝑠˜₁,𝑠˜₂,· · · ,𝑠˜_𝑟}. Let𝑋_1:𝑚 ∈R^𝑚×𝑑 denote the matrix containing the first𝑚 rows of 𝑋_𝑛, while𝑋_𝑚+1:𝑛∈R^{𝑛−𝑚×𝑑} denotes the remaining rows of𝑋_𝑛. With this notation,

𝑋^𝑇

𝑛𝑋_𝑛=

𝑛−1

Õ

𝑖=1

𝒙𝑖𝒙^𝑇_𝑖 = 𝑋^𝑇

1:𝑚𝑋_1:𝑚 +𝑋^𝑇

𝑚+1:𝑛𝑋_𝑚+1:𝑛.

Because the first 𝑟 state-action pairs, {𝑠˜₁,𝑠˜₂,· · · ,𝑠˜_𝑟}, are unvisited after iteration 𝑚, the first𝑟 elements of𝒙𝑖are zero for all𝑖 > 𝑚. Therefore, 𝑋^𝑇

𝑚+1:𝑛𝑋_𝑚+1:𝑛can be written in the following block matrix form:

𝑋^𝑇

𝑚+1:𝑛𝑋_𝑚+1:𝑛=

"

𝑂_𝑟×𝑟 𝑂_{𝑟×(𝑑−𝑟)}

𝑂_{(𝑑−𝑟)×𝑟} 𝐴_𝑛

# ,

where 𝑂_𝑎×𝑏 denotes the all-zero matrix with dimensions 𝑎 × 𝑏. The matrix 𝐴_𝑛 includes elements that are unbounded as 𝑛 −→ ∞. In particular, the diagonal elements of 𝐴_𝑛 approach infinity as𝑛 −→ ∞. The matrix 𝑋^𝑇

𝑛𝑋_𝑛 can be written in the following block matrix form:

𝑋^𝑇

𝑛𝑋_𝑛 =𝑋^𝑇

1:𝑚𝑋_1:𝑚+ 𝑋^𝑇

𝑚+1:𝑛𝑋_𝑚+1:𝑛

=

"

[𝑋^𝑇

1:𝑚

𝑋_1:𝑚]_(1:𝑟 ,1:𝑟) [𝑋^𝑇

1:𝑚

𝑋_1:𝑚]_(1:𝑟 ,𝑟+1:𝑑)

[𝑋^𝑇

1:𝑚𝑋_1:𝑚]₍𝑟+1:𝑑 ,1:𝑟) [𝑋^𝑇

1:𝑚𝑋_1:𝑚]₍𝑟+1:𝑑 ,𝑟+1:𝑑)+ 𝐴_𝑛

# :=

"

𝐵 𝐶

𝐶^𝑇 𝐷_𝑛

# , where𝑀_{(𝑎:𝑏,𝑐:𝑑)} denotes the submatrix of𝑀 obtained by extracting rows𝑎through 𝑏and columns𝑐through𝑑. Because matrices𝐵and𝐶only depend upon𝑋_1:𝑚, they are fixed as𝑛increases, while matrix𝐷_𝑛contains values that grow towards infinity with increasing 𝑛. In particular, all elements along 𝐷_𝑛’s diagonal are unbounded.

Intuitively, in the limit,𝐵and𝐶are close to zero compared to𝐷_𝑛, and𝑋^𝑇

𝑛𝑋_𝑛(when normalized) increasingly resembles a matrix in which only the bottom-right block is nonzero. This intuitive notion is formalized next.

Consider an eigenpair (𝒖_𝑖^(𝑛), 𝜉^(𝑛)

𝑖 ) of 𝑋^𝑇

𝑛𝑋_𝑛 such that lim𝑛−→∞𝜉^(𝑛)

𝑖 = ∞. The fol- lowing argument shows that the first element of 𝒖_𝑖^(𝑛) must approach 0. Letting 𝒖_𝑖^(𝑛) =

h

𝒛_𝑖^(𝑛)𝑇 𝒒_𝑖^(𝑛)𝑇 i^𝑇

, where𝒛_𝑖^(𝑛) ∈R^𝑚 and𝒒_𝑖^(𝑛) ∈R^{𝑛−1−𝑚}: (𝑋^𝑇

𝑛𝑋_𝑛)𝒖_𝑖^(𝑛) = 𝑋^𝑇

𝑛𝑋_𝑛

"

𝒛_𝑖^(𝑛) 𝒒_𝑖^(𝑛)

#

=

"

𝐵 𝐶

𝐶^𝑇 𝐷_𝑛

# "

𝒛_𝑖^(𝑛) 𝒒_𝑖^(𝑛)

#

=

"

𝐵𝒛_𝑖^(𝑛) +𝐶𝒒_𝑖^(𝑛) 𝐶^𝑇𝒛_𝑖^(𝑛) +𝐷_𝑛𝒒_𝑖^(𝑛)

#

=𝜉⁽

𝑛) 𝑖

"

𝒛_𝑖^(𝑛) 𝒒_𝑖^(𝑛)

# .

Dividing both sides by𝜉⁽

𝑛) 𝑖 , 1

𝜉^(𝑛)

𝑖

𝑋^𝑇

𝑛𝑋_𝑛

"

𝒛_𝑖^(𝑛) 𝒒_𝑖^(𝑛)

#

=















1 𝜉^(𝑛)

𝑖

𝐵𝒛_𝑖^(𝑛) +𝐶𝒒^(𝑛)_𝑖

1 𝜉^(𝑛)

𝑖

𝐶^𝑇𝒛_𝑖^(𝑛) +𝐷_𝑛𝒒_𝑖^(𝑛)















=

"

𝒛_𝑖^(𝑛) 𝒒_𝑖^(𝑛)

# .

In the upper matrix block: lim𝑛−→∞𝜉^(𝑛)

𝑖 = ∞, 𝐵 and 𝐶 are fixed as 𝑛 increases, and𝒛_𝑖^(𝑛) and𝒒_𝑖^(𝑛) have upper-bounded elements because𝒖_𝑖^(𝑛) is a unit vector. Thus, lim𝑛−→∞𝒛_𝑖^(𝑛) =lim𝑛−→∞ 1

𝜉^(𝑛)

𝑖

𝐵𝒛_𝑖^(𝑛)+𝐶𝒒_𝑖^(𝑛)

=0. In particular, the first element of

𝒛_𝑖^(𝑛) converges to zero, implying that the same is true of𝒖_𝑖^(𝑛).

As justified above, this result implies that for each iteration𝑛, there exists an index 𝑘₀ ∈ {1, . . . , 𝑑}such that the right-hand side of Eq. (B.8) has a lower bound above zero. This completes the proof that the denominator in Eq. (B.5) does not decay to zero. As a result, there exists some 𝜌 >0 such that 𝑃(𝜋 =𝜋_𝑣𝑖(𝒑˜,𝒓₁)) ≥ 𝜌 >0.

In consequence, DPS is guaranteed to infinitely often sample pairs(𝒑˜, 𝜋)such that 𝜋 = 𝜋_𝑣𝑖(𝒑˜,𝒓₁). As a result, DPS infinitely often samples policies that prioritize reaching ˜𝑠₁ as quickly as possible. Such a policy always takes action 𝑎 in state 𝑠. Furthermore, because 𝑠 is visited infinitely often, either a) 𝑝₀(𝑠) > 0 or b) the infinitely-visited state-action pairs include a path with a nonzero probability of reaching 𝑠. In case a), since the initial state distribution is fixed, the MDP will infinitely often begin in state𝑠under the policy𝜋 =𝜋_𝑣𝑖(𝒑˜,𝒓₁), so ˜𝑠₁will be visited infinitely often. In case b), due to Lemma 3, the transition dynamics parameters for state-actions along the path to 𝑠 converge to their true values (intuitively, the algorithm knows how to reach𝑠). In episodes with the policy𝜋 =𝜋_𝑣𝑖(𝒑˜,𝒓₁), DPS is thus guaranteed to reach ˜𝑠₁infinitely often. Since DPS selects 𝜋_𝑣𝑖(𝒑˜,𝒓₁) infinitely often, it must reach ˜𝑠₁ infinitely often. This presents a contradiction, proving that every state-action pair must be visited infinitely often.

The direct combination of Lemmas 3 and 6 prove asymptotic consistency of the transition dynamics model:

Proposition 3.Assume that DPS is executed in the preference-based RL setting, with transition dynamics modeled via a Dirichlet model, utilities modeled via either the linear or logistic link function, and utility posterior sampling distributions given in Eq.(4.13). Then, the sampled transition dynamics 𝒑˜𝑖1, 𝒑˜𝑖2converge in distribution to the true dynamics, 𝒑˜𝑖1, 𝒑˜𝑖2

−→𝐷 𝒑. This consistency result also holds when removing the𝛽_𝑛(𝛿)factors from the distributions in Eq.(4.13).