The final inclusion for an efficient and numerically stable particle filter is working with logarithms. As mentioned in Section 2.3, we work with logarithms as the probabilities we are working with are very small. Working with log-weights inside of a particle filter still requires exponentiation when the simulation terminates. This is prone to numerical insta- bilities and a common approach used to avoid exponentiation is theLog-Sum-Exponential (LSE) “trick” [10]. This can easily be applied alongside particle filters to reduce numerical instability and is essentially a way of evaluating the summation in Eq. 2.22. The LSE,ϕ, can be calculated as,
ϕ=c+ log
Np
X
i=1
exp (w(i)−c), (2.23)
where w(i) now denotes the log-weights and c = max{w(1), w(2), . . . , w(Np)} ensures that the largest normalised weight is 1 [10]. Using the LSE, the log-likelihood is then calculated as,
log ˆp(y) =−logNp+ϕ, (2.24)
which is simply the log of Eq. (2.22). The weights normalised in this way can then be used in the resampling strategies avoiding issues of direct renormalisation.
27 2.5. The SPSA algorithm There are various approaches for minimising a function when it has a closed form [16, 69]. Many of these rely on calculation of the gradient of f(x) [16]. In instances where the exact derivatives are not easily attainable, approximations are usually made.
A particular method we build on in this thesis, the SPSA algorithm, relies on using a first order finite difference approximation to the gradient [69]. There are two first order finite difference approximations used in stochastic optimisation, a two-sided approximation;
∇f(x)≈ f(x+h)−f(x−h)
2 h−1, (2.26)
and, a one-sided approximation;
∇f(x)≈(f(x+h)−f(x))h−1, (2.27) whereh is some step vector and h−1 is a vector of inverses,
h−1 = 1
h1, . . . , 1 hm
.
When we estimate the loss function there is some noise in the result, and the loss function is more appropriately expressed as,
f(x) = ˜f(x) +(x), (2.28)
where (x) is some noise term which is often state dependent, ˜f(x) is the actual loss measurement and x is some point in the set Ω ∈ Rm which may be constrained. The function in Eq. (2.28) is referred to as a loss function. In the context here—as our loss function is stochastic in nature—this form of optimisation is referred to as stochastic optimisation [16, 69].
Spall [69] provides the technical link between the two sided approximation and the true gradient for particular choices of the step vector. This derivation can be easily adapted to the one-sided finite difference and we provide the key result here. At a given iteration k, let hk = ck∆k where {ck} is a specially defined sequence and ∆k has independent and identically distributed elements ∆k,i which satisfy an inverse moments property. Let gk,j(·) denote element j of the estimate of the gradient as per Eq. (2.27). Under the assumptions on ck and ∆k,
Eh
gj(ˆθk)|θˆki
≈fj0(θk) +X
l6=m
fl0(θk)E ∆k,l
∆k,m
where f0 denotes the true gradient and ∆k,m denotes element m of ∆k. The ‘≈’ arises as there are some neglected higher order terms. Furthermore, the error terms on the loss function may introduce an additional source of error, however this often proves negligible
as the function estimate is often distributed about the true function value and hence the error has mean 0. The error terms introduced by the Taylor expansion are of order O(c2k) and so can be neglected for suitably chosen sequences ck. Under the assumption on the elements of ∆k satisfying an inverse moment property and being a mean 0 distribution, E[∆k,m/∆k,j] = 0. As a result the summation term is equal to zero and hence,
Eh
gk,j(ˆθk)|θˆki
≈fj0(ˆθk).
We provide details regarding the choices of ck and ∆k in Chapter 3. The details in [69]
provide the detailed walkthrough for the derivation of this result when assuming a central finite difference. The same derivation can easily be adapted to produce the result for the single sided finite difference used here.
In a stochastic optimisation framework there are a range of commonly used techniques such as simulated annealing and genetic algorithms [16]. The issue with these types of algorithms is that they are computationally expensive, and (often) slow approaches which can prove difficult to tune [69]. These methods are most suited in instances where the problem is solely to solve the optimisation problem to a certain level of precision.
The key purpose of obtaining the MAP in this thesis is to tune particle filters and as a result of this we prefer lower runtimes over highly precise estimates. An alternative to these methodologies is the simultaneous perturbation stochastic approximation (SPSA) algorithm which relies on the idea of a finite difference approximation to the gradient.
The SPSA algorithm searches in an approximate steepest descent direction and uses gain sequences which increase the step sizes during the earlier iterations, while reducing it later on [69]. This accounts for noise in the estimation of the loss function.
The SPSA algorithm was developed in Spall [68] and while this provided most of the analysis of the properties of the algorithm, later work provided a detailed explanation and proof of the convergence properties of the SPSA algorithm [69]. We refer the interested reader to [69] for the detailed analysis and proof of the performance of the general algo- rithms. We begin by presenting the basic version of the SPSA algorithm (Algorithm 5) with many generalised steps to demonstrate the main ideas.
In Algorithm 5,∆is referred to as the perturbation vector and is used to determine the direction of the update to the current estimate of the parameters. The gain sequences,c(k) anda(k)are sequences of positive decreasing numbers approaching 0. These gain sequences allow the search to improve precision during later iterations. The parameters of the SPSA algorithm are crucial to its implementation and while there are no certain choices for each of these, Spall [69] provides many reasonable starting choices. We leave discussion on these sequences until Chapter 3 as we provide indications of some of these suggestions as well as some adaptations that we have made in order to improve the performance in a Bayesian framework where SMC methods are used to obtain estimates of the likelihood.
29 2.5. The SPSA algorithm
Algorithm 5 Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm Inputs Initial parameters x0, number of iterations M, loss function f
1: for k= 1 : M do
2: for i= 1 :m, wherem is the number of parameters do
3: Sample ∆(k)i from a mean 0 distribution satisfying an inverse moments property (Eq. 3.5).
4: end for
5: Set ∆(k)=
∆(k)1 ,∆(k)2 , . . . ,∆(k)m
6: Set h(k) =c(k)∆(k)
7: Estimate the derivative of the loss function,
∇f x(k)
≈gˆ(k)
8: Update the current estimate of the minimum using the standard stochastic ap- proximation form,
x(k+1) =x(k)−a(k)gˆ(k),
9: end for
Return Search path
x(k) Mk=1 and estimate of the minimum x(M)
Chapter 3
MAP estimation using noisy likelihood estimates
This chapter details the development of a stochastic optimisation algorithm to deter- mine an estimate of the maximum a posteriori (MAP) of a posterior distribution when only noisy estimates are available. This chapter explores the simultaneous perturbation stochastic approximation (SPSA) algorithm and the alterations needed in order to es- timate the MAP when the likelihood is estimated using a particle filter. The modified SPSA algorithm is tested on several simulated datasets in order to assess its practical performance.
3.1 The MAP estimation problem
The MAP is the vector of parametersθMAP ∈Θ, where Θ is the support of the posterior, such that,
θMAP= arg max
θ∈Θ
p(θ|D), (3.1)
whereD is the observed data.
From a Bayesian modelling perspective, the MAP is important for two main reasons.
The first is that an estimate of the MAP provides information about the most likely parameters to have resulted in our data while accounting for our prior beliefs. This can be used to then make quick decisions in different contexts without having to sample the whole posterior. The second reason is to tune the particle filters used to obtain estimates of the log-likelihood and hence lead to improved efficiency of pmMH methods [8, 57, 65].
This latter point is key to improved use of pmMH methods to learn posterior distributions 31
of parameters. Within this thesis we place emphasis on determining the MAP estimate in order to improve the computational efficiency of the pmMH methods by determining the optimal number of particles to use in a given particle filter. This optimal number of particles is problem dependent and a suitable choice of particles is one which keeps the variance of the log-likelihood within tight bounds [3, 22, 65].