Routine 4.3.3: Density matrix exponentiation (in superposition)
11.5 Analytical investigation of the accuracy-weighted en- semblesemble
In order to explore the accuracy-weighted ensemble classifier further, I want to conduct some ana- lytical investigations. It is convenient to assume that we know the probability distributionp(x, y) from which the data is picked (that is either the ‘true’ probability distribution with which data is generated, or the approximate distribution inferred by some data mining technique). Furthermore, we consider the continuous limitP
→R
. Each parameterθdefines decision regions in the input space,Rθ−1i for class−1 andRθ1for class 1 (i.e. regions of inputs that are mapped to the respective
3If one can analytically prove that 12E of all possible models will be accurate, one can artificially extend the superposition to twice the size, prevent half of the subspace from being flagged and thereby achieve the optimal amplitude amplification scheme.
x x class−1 class 1
((w0)1, o= 1)
class−1 class 1
((w0)2, o=−1)
Figure 11.6: Very simple model of a 1-dimensional classifier analysed in the text: The parameter w0 denotes the position of the decision boundary while parameterodetermines its orientation.
classes). The accuracy can then be expressed as a(θ) = 1
2 Z
Rθ-1
p(x, y= -1)dx+1 2
Z
Rθ1
p(x, y= 1)dx. (11.9)
In words, this expression measures how much of the density of a class falls into the decision region proposed by the model for that class. Good parameters will propose decision regions that contain the high-density areas of a class distrubution. The factor of 1/2 is necessary to ensure that the accuracy is always in the interval [0,1] since the two probability densities are each normalised to 1.
The probability distributions we consider here will be of the formp(x, y=±1) =g(x;µ±, σ±) = g±(x). They are normalised,R∞
−∞g±(x)dx= 1, and vanish asymptotically,g±(x)→0 forx→ −∞
and x → +∞. The hyperparameters µ−, σ− and µ+, σ+ define the mean or ‘position’ and the variance or width of the distribution for the two classes −1 and 1. Prominent examples are Gaussians or step functions. LetG(x;µ±, σ±) =G±(x) be the integral ofg(x;µ±, σ±), which will fulfilsG±(x)→0 for x→ −∞andG±(x)→1 for x→+∞. Two expressions following from this property which will become important later are
a
Z
−∞
g±(x)dx=G±(a)−G±(−∞) =G±(a),
and ∞
Z
a
g±(x)dx=G±(∞)−G±(a) = 1−G±(a).
I consider a minimal toy example for a classifier, namely a perceptron model on a one-dimensional input space,f(x;w, w0) = sgn(wx+w0) withx, w, w0∈R. While one parameter would be sufficient to mark the position of the point-like ‘decision boundary’, a second one is required to define its orientation. One can simplify the model even further by letting the biasw0define the position of the decision boundary and introducing a binary ‘orientation’ parametero∈ {−1,1}(as illustrated in Figure 11.6),
f(x;o, w0) = sgn(o(x−w0)). (11.10) For this simplified perceptron model the decision regions are given by
R−1= [−∞, w0],R1= [w0,∞],
Figure 11.7: The black line plots the expectation value for the blue and green class densities for different new inputs ˜x. A positive expectation value yields the prediction 1 while a negative expectation value predicts class −1. At point zero lies the decision boundary. The plot shows that the model predicts the decision boundary where we would expect it, namely between the two distributions
2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 x1
2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0
x2
class -1 class 1
Figure 11.8: Perceptron classifier with 2-dimensional inputs and a bias. The ensemble consist of 8000 models, each of the three parameters taking 20 equally distributed values from [−1,1]. The resolution to compute the decision regions was ∆x1= ∆x2= 0.05. The dataset was generated with python scikit-learn’s blob function and both classes have the same variance parameter. One can see that in two dimensions, the decision boundary still lies in between the two means ofµ−1= [−1,1]
andµ1= [1,−1].
for the orientationo= 1 and
R−1= [w0,∞],R1= [−∞, w0], foro=−1. Our goal is to compute the expectation value
E[f(˜x;w0, o)]∝ Z
dθ a(θ)f(˜x;o, w0), of which the sign function evaluates the desired prediction ˜y.
Inserting the definitions from above, as well as some calculus using the properties ofp(x) brings this expression to
Z∞
−∞
dw0(G−(w0)−G+(w0)) sgn(˜x−w0), and evaluating the sign function for the two cases ˜x > w0,x < w˜ 0leads to
˜ x
Z
−∞
dw0
nG+(w0)−G−(w0)o
− Z∞
˜ x
dw0
nG+(w0)−G−(w0)o
. (11.11)
To analyse this expression further, consider the two class densities to be Gaussian probability distributions
p(x, y=±1) = 1 σ±√
2πexp−
(x−µ±)2 2σ2
± .
The indefinite integral over the Gaussian distributions is given by G±(x) = 1
2[1 + erfx−µ±
√2σ± ].
Inserting this into Eq (11.11) we get
˜ x
Z
−∞
dw0
nerfw0−µ+
√2σ+ − erfw0−µ−
√2σ− o −
Z∞
˜ x
dw0
nerfw0−µ+
√2σ+ − erfw0−µ−
√2σ−
o. (11.12)
Figure 11.7 plots the expectation value for different inputs ˜xfor the case of two Gaussians with σ−=σ+= 0.5, andµ− =−1, µ+= 1. The decision boundary is at the point where the expectation value changes from a negative to a positive number,E[f(ˆx;w0, o)] = 0. One can see that for this simple case, the decision boundary will be in between the two means, which we would naturally expect. This is an important finding, since it implies that the accuracy-weighted ensemble classifier works - arguebaly only for a very simple model and dataset. A quick calculation shows that we can expect to find the decision boundaries midway between the two means for all cases withσ−=σ+. In this case the integrals in Equation (11.12) evaluate to
˜ x
Z
−∞
dw0erfw0−µ±
√2σ± =γ±(˜x)− lim
R→−∞γ±(R),
and ∞
Z
˜ x
dw0erfw0−µ±
√2σ± = lim
R→∞γ±(R)−γ±(˜x), with the integral over the error function
γ±(x) = (x−µ±) erf
x−µ±
√2σ±
+ r2
πσ±e−(
x−µ±
√ 2σ±)2
.
Assuming that the mean and variance of the two class distributions are of reasonable (i.e. finite) value, the error function evaluates to 0 or 1 before the limit process becomes important, and one can therefore write
R→−∞lim γ±(R) = lim
R→−∞
r2 πσ±e−(
R−µ√ ± 2σ±)2
, (11.13)
R→∞lim γ±(R) = lim
R→∞R + lim
R→∞
r2 πσ±e−(
R−µ√ ± 2σ±)2
−µ. (11.14)
The expectation value for the case of equal variances therefore becomes
E[f(˜x;w0, o)] = 2γ−(˜x)−2γ+(˜x). (11.15) Setting ˜x = ˆµ = µ−+ 0.5(µ++µ−) turns the expectation value to zero; the point ˆµ between
−6 −4 −2 0 2 4 6
˜ x
−1.0
−0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0
integrand
µ= 6 µ= 2 µ= 0.4
Figure 11.9: Plot of the integrand in Equation (11.12), I = erfx−µ√2σ+
+ −erfx−mu√2σ−
+ , for σ− = 0.5, σ+= 1.5 and different distancesµ=µ−−µ+ of the means whose average value is chosen to be zero. A higher variance accounts for a ‘flatter’ flank of the function. When the means of the data distributions have a sufficient distance, the area under the curve right of 0 is equal to the area under the curve left of 0, and the resulting decision boundary lies exactly between the means.
If the two distributions are too close together, this symmetry vanishes for σ− 6=σ+. In the case σ−=σ+ the symmetry is always given.
the two variances is shown to be the decision boundary. Simulations confirm that this is also true for other distributions, such as a square, exponential or Lorentz distribution, as well as for two-dimensional data (see Figure 11.8).
To understand what happens in the case of different variances, the integrand of Equation (11.12) is plotted in Figure 11.9 for different distances µ = µ+ −µ−. The value of σ± changes the steepness of the flanks of the distribution. As long as the means of the distributions are far enough apart from one another, this does not effect the decision boundary lying in their middle (since the steepness of the flank does not affect the area under the curve for positive or negative x-values).
However, for certain small values ofµthe distribution does not carry the same area on the right and left side ofx= 0.
The simplicity of the core model allows us to have a deeper look into the structure of the expectation value. Figure 11.10 shows the different component functions of the ensemble classifier over the weight space, namely the accuracy, the core model function as well as their product, for the examplesσ± = 1, ˜x= 0.7, the two cases o=−1,1 and different means. One can see that the final function over which we integrate in order to obtain the expectation value (marked in grey) is point symmetric if ˜x lies in the middle of the class distributions’ means. At this point, the expectation value vanishes which defines the location of the decision boundary. One can see that for two class distributions close to each other, the accuracy function is only symmetric if their variance is the same. One can conclude that the function of the accuracy over the parameter plays an important role for the location of the decision boundary. As a summary, the analytical considerations give evidence that the weighted average ensemble classifier works well with linear models and for simple data distributions.
Further investigations would have to show whether the accuracy-weighted ensemble classifier is able to realise nonlinear decision boundaries if the base classifiers are more flexible. This is a huge computational effort, since for example the next more complex neural network model requires two hidden layers to be point symmetric, and with one bias neuron and for two-dimensional inputs
the architecture has already seven or more weights. If each weight is encoded in three qubits only (including one qubit for the sign), we get an ensemble of 221 members whose collective decision needs to be computed for an entire input space in order to determine the decision boundaries.
However, low resolution simulations with one-dimensional inputs confirm that nonlinear decision boundaries can be learnt. Sampling methods could help to obtain approximations to this result.
Another important question concerns overfitting. In AdaBoost, regularisation is equivalent to early stopping [241]. Bayesian learning does not encounter problems of overfitting. Simple models like the perceptron considered here may have an ‘inbuilt’ regularisation mechanism. However, more flexible models considered for the accuracy-weighted ensemble may very well suffer from overfitting.
A third avenue of further investigation is to look for speedups in the quantum ensemble models, for example by constructing weighting schemes that are difficult to implement with classical sampling methods, or by applying pure quantum classifiers.
In summary, this chapter introduced the quantum ensemble framework as a template for collective decisions of quantum models. Going beyond the quest for speedups from earlier chapters, this work demonstrates that thinking about quantum algorithms can lead to new insights into models that can also be interesting from the perspective of classical computation.
Example 1
−6 −4 −2 0 2 4 6 x
0.0 0.2 0.4 0.6 0.8 1.0 1.2
class distributions p(x, y=−1) p(x, y= 1)
−6 −4 −2 0 2 4 6 w0
−1.0
−0.5 0.0 0.5 1.0
predictionf(x=1;w0, o)
o= 1 o=−1
−6 −4 −2 0 2 4 6 w0
0.0 0.2 0.4 0.6 0.8 1.0
accuracya(w0, o) o= 1 o=−1
−6 −4 −2 0 2 4 6 w0
−1.0
−0.5 0.0 0.5 1.0
1.5 resulting function
Example 2
−6 −4 −2 0 2 4 6 x
0.0 0.2 0.4 0.6 0.8 1.0 1.2
class distributions p(x, y=−1) p(x, y= 1)
−6 −4 −2 0 2 4 6 w0
−1.0
−0.5 0.0 0.5 1.0
predictionf(x=1;w0, o)
o= 1 o=−1
−6 −4 −2 0 2 4 6 w0
0.0 0.2 0.4 0.6 0.8 1.0
accuracya(w0, o) o= 1
o=−1
−6 −4 −2 0 2 4 6 w0
−1.0
−0.5 0.0 0.5 1.0
1.5 resulting function
Figure 11.10: Detailed analysis of the classification of ˜x= 0.7 for two examples of data distribu- tions. The upper Example 1 showsp(x, y=−1) =N(−1,0.5) andp(x, y= 1) = N(1,0.5) while the lower Example 2 showsp(x, y=−1) =N(−1,0.5) andp(x, y= 1) =N(1,2) (plotted each in the top left figure of the four). The top right figure in each block shows the classification of a given new input ˜x for varying parameters b, plotted for o = 1 and o = −1 respectively. The bottom left shows the accuracy or classification performancea(w0, o= 1) and a(w0, o=−1) on the data distribution. The bottom right plots the product of the previous two foro= 1 ando=−1, as well as the resulting function under the integral. The prediction outcome is the integral over the black curve, or the total of the gray shaded areas. One can see that for models with different variances, the accuracies loose their symmetry and the decision boundary will therefore not lie in the middle between the two means.
Conclusion
This thesis investigated how the problem of supervised pattern recognition - finding the label to a feature vector given a dataset of labelled feature vectors - can be solved using a universal quantum computer. As a conclusion I want to highlight the central points made in each chapter, summarise the general findings and suggest avenues for further research.