Introduction
Motivation
In practice, human-specified outcomes can vary over time (Payne et al., 1993; Brochu, Brochu, & de Freitas, 2010), while incorrectly specified rewards can exhibit unintended side effects—for example, a robot can knock one down. In another example (Popov et al., 2017), an agent fails to complete a simulated robotic block-picking task because it is rewarded for raising the vertical coordinate of the side of the block facing the ground and learns to do so by turn the block upside down. down (instead of picking it up).
The Bandit and Reinforcement Learning Problems
Meanwhile, in the RL configuration, the agent takes actions and observes rewards, but in addition to generating rewards, these actions change the underlying state of the environment. After each step, the agent not only receives the reward signal, but also observes the state of the environment.
Human-in-the-Loop Learning
While this work deals with online regret minimization, several existing algorithms instead minimize priority queries for the user (Christiano et al., 2017; Wirth, Fürnkranz, & Neumann, 2016). Coactive feedback has been used in online search and movie recommendation tasks in simulation (Shivaswamy and Joachims, 2015), as well as to optimize trajectory planning in robotic manipulation tasks such as grocery checkout (Jain, Sharma, et al., 2015).
Lower-Body Exoskeletons for Mobility Assistance
In fact, a survey of wheelchair users and healthcare professionals working with mobility impaired individuals (Wolff et al., 2014) identified “health benefits” as the most highly recommended reason for using an exoskeleton. The Atalante exoskeleton, first introduced in Harib et al. 2018) describes the device's mechanical components and control architecture in detail.
Contributions
This result leads to a Bayesian no-regret rate for DPS under credit allocation via Bayesian linear regression. In summary, concurrent with the development of the DPS algorithm, this work mathematically integrates Bayesian credit allocation and preference generation within the conventional posterior sampling framework, evaluates different credit allocation models, and presents a framework for regret analysis.
Organization
Gaussian process optimization in bandit environment: Regretless and experimental design. In: International Conference on Machine Learning (ICML). A survey of preference-based reinforcement learning methods.” In: Journal of Machine Learning Research pp.
Background
Bayesian Inference for Parameter Estimation
One can use Bayesian inference to estimate probabilistically, given the combination of data and any prior knowledge about it. In Bayes' theorem, given by Eq. 2.2), replacing 𝐴 with the model parameters𝜽 and𝐵 with the observed datasetD yields:. The maximum a posteriori estimate𝜽, or MAP estimate, is defined as the mode of the posterior distribution.
Gaussian Processes
Note that 𝐻 must be positive semidefinite for the approximate posterior covariance matrix 𝐻−1 to be well defined. Each positive semidefinite kernel function K (·,·) corresponds to a unique reproducing kernel Hilbert space (RKHS) HK of real-valued functions associated with that kernel (Wainwright, 2019). RKHS satisfies two properties: 1) for any fixed 𝒙 ∈ A the function K (·, 𝒙) belongs to HK and 2) for any fixed 𝒙 ∈ A the function K (·, 𝒙) satisfies the reproduction property, namely that < 𝑓 ,K (·,𝒙) >H.
Entropy, Mutual Information, and Kullback-Leibler Divergence
Then the Kullback-Leibler divergence is an (asymmetric) measure of the distance between two probability distributions. The mutual information of two random variables 𝑋 and 𝑌 quantifies the amount of information gained about one of the variables by observing the value of the other.
Bandit Learning
Russo and Van Roy (2016) demonstrate that in the K-armed bandit problem with 𝐴 actions, posterior sampling reaches Γ ≤ 𝐴2. Second, Russo and Van Roy (2014a) propose the information-driven sampling framework, which uses the information ratio Γ𝑖, defined in Eq. 2.12), to guide the selection of actions in the Bayesian setting.
Dueling Bandits
The multi-dueling bandits setting (Brost et al., 2016; Sui, Zhuang, et al., 2017) generalizes the dueling bandits setting so that in each iteration the algorithm can choose more than 2 actions. In particular, SelfSparring (Sui, Zhuang, et al., 2017) is a state-of-the-art posterior sampling framework for preference-based learning in multi-dueling bandits, inspired by Sparring (Ailon, Karnin, & Joachims, 2014).
Episodic Reinforcement Learning
Expectation Propagation for the Generative Aspect Model." In: Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence. Clinical Online Recommendation with Feedback from Subgroup Ranking." In: Proceedings of the 8th ACM Conference on Recommender Systems.
The Preference-Based Generalized Linear Bandit and Reinforce-
The Generalized Linear Dueling Bandit Problem Setting
Each action𝒙 ∈ Is a tool for the user that reflects the value that the user attaches to 𝒙. The first approach, which is typically used in dueling bandits (Yue, Broder, et al., 2012; Sui, Zhuang, et al., 2017), was previously presented in Eq.
The Preference-Based Reinforcement Learning Problem Setting
Each trajectory is represented as a feature vector, where each feature records the number of times a particular state-action pair is visited. Analogous to the generalized setting of linear dueling bandits discussed in section 3.1, preferences are assumed to be generated from underlying trajectory utilities that are linear in the characteristics of state action attendance, as formalized in the next two assumptions.
Comparing the Preference-Based Generalized Linear Bandit and RL
Portfolio Allocation for Bayesian Optimization.” In: Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence. High-Dimensional Bayesian Optimization Using Dropout.” In: Proceedings of the 26th International Joint Conference on Artificial Intelligence.
Dueling Posterior Sampling for Preference-Based Bandits and
The Dueling Posterior Sampling Algorithm
Advance (Algorithm 6) selects a policy to execute by drawing samples from the Bayesian posteriors of both the dynamics and utilities. Feedback (Algorithm 7) updates the Bayesian posteriors of the dynamics and utility models based on new data.
Additional Notation
Third, each action is associated with a 𝑑dimensional vector and not just an index in {1,. While in the case of RL the utility vector 𝒓 indicates the utility of each state-action pair, in the case of a generalized linear thug it specifies a linear weight for each dimension of the action space.
Posterior Modeling for Utility Inference and Credit Assignment
This conjugate Gaussian prior and probability pair leads to the following closed form Gaussian posterior:. 𝐿𝑔𝛽𝑖(𝛿), where𝐿𝑔is the Lipschitz constant of𝑔(denoted by𝑘𝜇 in Filippi et al., 2010) and where each reward lies in [0, 𝑅max] in the neighborhood described in Filippi et al. be described.
Theoretical Analysis
This section extends the concept of the information ratio introduced in Russo and Van Roy (2016) to analyze regret in the preference-based setting. Furthermore, as seen in the figure, all the information ratio values are less than 𝑑 =𝑆 𝐴by a significant margin.
Empirical Performance of DPS
This may be because unlike DPS, Sparring involves two competing learning algorithms, each of which only receives information about whether their actions "beat" the competing algorithms' actions, but does not account for the competing algorithm's action choices. DPS is competitive against both baselines and is robust to the utility inference method and against noise in the preference feedback.
Discussion
Analysis of Thompson sampling for the multi-armed bandit problem. In: Conference on Learning Theory (COLT). Why is posterior sampling better than optimism for reinforcement learning?” In: International Conference on Machine Learning.
Mixed-Initiative Learning for Exoskeleton Gait Optimization
Introduction
Indeed, previous studies have shown that preferences are more reliable than numerical scores in various domains, including information retrieval (Chapelle, Joachims et al., 2012) and autonomous driving (Basu, Yang et al., 2017). This work also presents the LineCoSpar algorithm, which integrates CoSpar with techniques from high-dimensional Bayesian optimization (Kirschner, Mutny et al., 2019) to create a unified framework for implementing high-dimensional preference-based learning.
Background on the Atalante Exoskeleton and Gait Generation for
This section briefly explains this method to illustrate how exoskeleton gaits can be customized in response to user preferences; for more details on the gait generation process, refer to Harib et al. To translate the gait generation process to lower body exoskeletons, one must choose an optimization cost function and physical constraints to achieve user-preferred gaits.
The CoSpar Algorithm for Preference-Based Learning
𝑁} consisting of preferences 𝑁, where 𝒙𝑘1 𝒙𝑘2 indicates that the user prefers action 𝒙𝑘1 ∈ A to action 𝒙𝑘2 ∈ A in preference 𝑘. For 𝑛 = 𝑏 = 1, the user can report preferences between any pair of two consecutive trials, i.e., the user is asked,.
Simulation Results for CoSpar
We use CoSpar to minimize the one-dimensional objective function in Figure 5.3 using pairwise preferences obtained by comparing cost of transport values for the different step lengths. We then test CoSpar on a set of 100 synthetic two-dimensional utility functions, such as the one shown in Figure 5.4a.
Deployment of CoSpar in Human Subject Exoskeleton Experiments . 117
The user walked each of these step lengths in a random order in the exoskeleton and blindly ranked the three, as shown in Figure 5.6. The posterior 𝑃(D | 𝒇) is calculated over V𝑖 := L𝑖 ∪ W, where is the set of actions appearing in the preference feedback data setD.
Performance of LineCoSpar in Simulation
Coactive feedback was simulated for each sampled action𝒙𝑖 by finding the action𝒙0𝑖 with a higher objective value that differs from𝒙𝑖 along only one dimension. Patterned actions converge to high objective values in relatively few iterations, and coactive feedback accelerates this process.
Deployment of LineCoSpar in Human Subject Exoskeleton Experi-
Note that the algorithm explores different parts of the action space for each subject. For each subject, Table 5.1 lists the predicted optimal walking parameters, max, identified by LineCoSpar.
Discussion
Exemplary Complexity of Episodic Fixed-Horizon Reinforcement Learning.” In: Conference on Neural Information Processing Systems. Learning unknown Markov decision processes: A Thompson sampling approach. In: Conference on Neural Information Processing Systems.
Conclusions and Future Directions
Conclusion
With this goal in mind, this thesis begins by presenting the Dueling Posterior Sampling (DPS) framework for learning from human preferences in the generalized linear dueling bandit and preference-based reinforcement learning (RL) settings. The DPS framework integrates preference-based posterior sampling with Bayesian inference of: 1) the utilities underlying the user's preferences and 2) the environment's transition dynamics in the RL environment. This thesis also presents the CoSpar and LineCoSpar algorithms for mixed initiative learning from pairwise preferences and coactive feedback, in which the user suggests improved actions to the algorithm.
Future Work
Reward learns from human preferences and demos in Atari.” In: Conference on Neural Information Processing Systems. Interactive optimization of information retrieval systems as a dueling bandit problem. In: International Conference on Machine Learning (ICML).
Bayesian Linear Regression
This appendix provides the mathematical details of the credit allocation models evaluated in the Dueling Posterior Sampling (DPS) experiments presented in Section 4.5.
Gaussian Process Regression
A Gaussian process prior is placed on the underlying utilities𝒓; for example, in the RL setting, this prior is placed on the utilities of the individual state-action pairs. A state action utility𝑟(𝑠˜) corresponds to the utility weight of an action space dimension in the bandit environment, which is also𝒓𝑇𝒆𝑗(for some 𝑗).
Gaussian Process Preference Model
In order for the Laplace approximation to be valid, 𝑆(𝒓) must be convex in 𝒓: this guarantees that the optimization problem in Eq. A.8) is convex and that the covariance matrix defined by Eq. A.9) is positive definite, and therefore a valid Gaussian covariance matrix. In particular, this condition is satisfied for the Gaussian link function,𝑔Gaussian(·)= Φ(·), where Φ is the standard Gaussian CDF, as well as for the sigmoidal link function, 𝑔log(𝑥) := 𝜎(𝑥) = 1+exp(−1 𝑥).
Asymptotic Consistency of the Transition Dynamics in DPS in the
If each state-action pair is visited infinitely often, then 𝑛𝑗 −→ ∞ for each 𝑗 and therefore 𝒑˜(𝑗) converges in probability to 𝒑(𝑗): 𝒑˜(𝑗) −→𝑃 𝒑(𝑗). To conclude the proof of the convergence of the Bayesian transition dynamics model, Lemma 6 proves that every state-action pair is visited infinitely often.
Asymptotic Consistency of the Utilities in DPS
One can take the expectation and covariance of Eq. B.14) with respect to the variables{𝑧𝑖 𝑗}to show that they match the expressions in Eq. Right-hand sum in Eq. B.21) can then be split by the indices (𝑛𝑖) and the remaining indices:.
Asymptotic Consistency of the Selected Policies in DPS
Baselines: Sparring with Upper Confidence Bound (UCB) Algorithms192
