Consider that one wants to build an approximation to a function of a vector-valued input x, based only onm observations of the inputs and outputs: Y(x1), . . . , Y(xm). Appropriate ap- proaches to choosing those inputsxfor which the simulator should be run, known as the design of computer experiments, are discussed by Sacks et al. (1989b); Simpson et al. (1997); Sacks et al. (1989a); McKay et al. (1979); Sacks and Schiller (1988); Welch (1983); Morris et al.
(1993); Currin et al. (1991). The basic idea of the GP interpolation model is that the outputs, Y, are modeled as a Gaussian process that is indexed by the inputs,x. A Gaussian process is simply a set of random variables such that any finite subset has a multivariate Gaussian distri- bution1. A Gaussian process is defined by its mean function and covariance function, which in this case are functions ofx. Once the Gaussian process is observed atmlocationsx1, . . . ,xm, the conditional distribution of the process can be computed at any new location, x∗, which provides both an expected value and variance (uncertainty) of the underlying function.
The key here is that the function describing the covariance among the outputs, Y, is a function of the inputs, x. The covariance function is constructed such that the covariance between two outputs is large when the corresponding inputs are close together, and the covari- ance between two outputs is small when the corresponding inputs are far apart. As shown be- low, the conditional expected value ofY(x∗)is a linear combination of the observed outputs,
1The Gaussian model assumption is needed to derive the fundamental equations within the stochastic process framework. Interestingly, the same equations can be arrived at using different arguments (within a framework known as kriging), and it turns out the Gaussian assumption is in fact not needed to derive the equation for the predictor and its variance (mean-squared error). Nevertheless, the Gaussian assumption does come into play when one wants to construct confidence intervals for the true value of the function. O’Hagan (2006) has this to say about the Gaussian assumption: “Although the assumption of normality, implicit in the use of a GP, may seem to represent rather a strong limitation, in practice it has no impact if we can make enough runs of the computer code to produce an accurate emulation. In more complex problems, where it is not practical to make enough code runs to emulate with negligible code uncertainty, normality can matter.” The use of transformations of the code output (such as a logarithmic transformation) in those cases when the uncertainty is significant and the assumption of normality does not seem appropriate is a subject of ongoing research.
Y(x1), . . . , Y(xm), in which the weights depend on how closex∗ is to each of x1, . . . ,xm (one is reminded of radial basis functions). In addition, the conditional variance (uncertainty) ofY(x∗)is small ifx∗is close to the training points and large if it is not.
Further, the GP model may incorporate a systematic, parametric trend function whose pur- pose is to capture large-scale variations. This trend function can be, for example, a linear or quadratic regression of the training points. It turns out that this trend function is actually the (unconditional) mean function of the Gaussian process. The effect of the mean function on predictions that interpolate the training data tends to be small, but when the model is used for extrapolation, the predictions will follow the mean function very closely as soon as the correlations with the training data become negligible.
To develop the theory, letY(x)denote a Gaussian process with mean and covariance given by
E Y(x)
=fT(x)β (3.1)
and
Cov
Y(x), Y(x∗)
=λc(x,x∗ |ξ), (3.2)
where fT(x) defines q basis functions for the trend, and is given by 1 for a constant trend and[1 xT]T for a linear trend;β gives the coefficients of the regression trend;c(x,x∗ | ξ)is the correlation betweenxandx∗; andξis the vector of parameters governing the correlation function. While the process variance is typically denoted by σ2, λ is used instead throughout this dissertation in order to avoid confusion with the error variance in calibration analysis (Chapter V).
Consider that the process has been observed atmlocations (the training or design points) x1, . . . ,xm of a d-dimensional input variable, yielding the resulting observed random vector
Y = Y(x1), . . . , Y(xm)T
. By definition, the joint distribution ofY satisfies
Y ∼Nm fT(x)β, λR
, (3.3)
whereRis them×mmatrix of correlations among the training points. Under the assumption that the parameters governing both the trend function and the covariance function are known, the conditional expected value and variance (uncertainty) of the process at an untested location x∗ are calculated as
E
Y(x∗)|Y
=fT(x∗)β+rT(x∗)R−1(Y −F β) (3.4)
and
Var
Y(x∗)|Y
=λ 1−rTR−1r
, (3.5)
whereF is anm×qmatrix with rowsfT(xi)(the trend basis functions at each of the training points), andr is the vector of correlations betweenx∗ and each of the training points. Further, the full covariance matrix associated with a vector of predictions can be constructed using the following equation for the pairwise covariance elements:
Cov
Y(x), Y(x∗)|Y
=λ
c(x,x∗)−rTR−1r∗
, (3.6)
wherer is the vector of correlations betweenxand each of the training points, and r∗ is the vector of correlations betweenx∗and each of the training points.
When the coefficients of the trend function are not known, but are estimated using a gener- alized least squares procedure, or equivalently, maximum likelihood, the variance estimate of
Eq. (3.5) can be expanded as (Ripley, 1981): 2
Var
Y(x∗)|Y
=λ
1−rTR−1r
+h
f(x∗)−FTR−1riTh
FTR−1Fi−1h
f(x∗)−FTR−1ri
, (3.7)
which can also be written in matrix form as
Var
Y(x∗)|Y
=λ−h
fT(x∗), rT i
0 FT F R
−1
f(x∗)
r
. (3.8)
Note that when using Eq. (3.5), the variance ofY takes values between 0 and λ. Ifx∗ is equal to or very close to one of the training points, then the termrTR−1rwill go to 1, and the variance ofY(x∗)will be 0 (because there is no uncertainty at the tested locations). The effect of a known trend function on this prediction variance manifests itself through the value of the parameter λ. If the trend function captures much of the variation ofY, then the “maximum likelihood” estimate ofλwill be smaller (for more information on this, refer to Section 3.3).
When the trend function coefficients are not assumed known, the prediction variance of Eq. (3.7) still has a lower bound of 0, but there is no upper bound. In this case, the term f(x∗)−FTR−1r
also goes to zero at a location equal to one of the training points. When x∗ interpolates the training points, this author has not found a significant difference in the prediction uncertainties given by Eqs. (3.5) and (3.7), although Eq. (3.7) is significantly more expensive to evaluate.
2There are some notational peculiarities that are specific to the models developed from the standpoint of
“kriging,” as opposed to Gaussian processes. In kriging, Eq. (3.4) is termed the “universal kriging predictor”
when β is replaced by its generalized least squares estimate and the parametersλandξare assumed known (although in practice the covariance parameters are rarely known). Whenβ, λ, andξ are all estimated using the maximum likelihood procedure, the estimate given by (3.4) is sometimes referred to as the “unified universal kriging predictor” (Mardia and Marshall, 1984).
There are several different methods of parametrizing the correlation function. The form implemented by this author is the squared exponential form, given by
c(x,x∗) = exp
"
−
d
X
i=1
ξi(xi−x∗i)2
#
, (3.9)
whered is the dimension ofx, and the d parametersξi must be non-negative. The exponent must lie in the range [0,2] in order for the covariance matrix to be positive definite, but the value 2 is usually chosen because it produces a function that is infinitely differentiable. This form of the correlation function dictates that the degree of correlation of the outputs depends on the closeness of the inputs.
Also, the relative magnitudes of the parametersξ are related to the amount of importance each dimension ofxhas in predicting the outputY: a large value forξi(i.e., a small correlation length) indicates a high amount of “activity” (and likewise a low amount of correlation) in that direction. For example, if the response is independent of one of the inputs, then that input will have an infinite correlation length (because the response does not change in that direction) and aξof 0.
If the Gaussian process predictor is to be used to estimate gradients of the response, it can be tempting to estimate the gradients using finite differencing because the surrogate is so cheap to evaluate. This can be dangerous, though, because for even modest finite difference step sizes, numerical round-off error can render such estimates useless. Fortunately, it is not difficult to derive the exact expressions for the gradients. Only the case of a constant trend function is considered here, but more general expressions for the gradients are available (see, for example, Vazquez and Walter, 2005).
For the constant trend case, the predictor is given by
E
Y(x)|Y
=β+rT(x)R−1(Y −1β). (3.10)
Since onlyris a function ofx, it is easy to show that by using the chain rule, the derivative of Y with respect toxkis
∂E
Y(x)|Y
∂xk = ∂E
Y(x)|Y
∂r
∂r
∂xk = ˙rTkR−1(Y −1β), (3.11)
wherer˙kis the derivative ofrwith respect toxk.
Since the values of x are typically standardized before computing the correlations (see Section 3.3.3), this transformation must be accounted for in the computation ofr˙k. Consider a linear transformation for each component of xto a standardized space: x0i = ai +bixi. Then by the chain rule, and for the correlation function of Eq. (3.9), the derivative of the correlation vector with respect toxkis given by
˙
rk = ∂r
∂xk = ∂r
∂x0k
∂x0k
∂xk =
∂c(x,x(i))
∂x0k
i
∂x0k
∂xk =h
−2ξk
x0k−x0(i)k
c(x,x(i))bki
i, (3.12)
wherex(i)is theithtraining point, andc(x,x∗)is computed based on the standardized values ofx, as usual.
Finally, some remarks on the dimensionality of the input are probably warranted. Many users would like to know how many training points are typically required to model a function having a certain number of inputs, and for how many inputs it may be feasible to use the Gaussian process interpolation approach. The number of training points needed may actually depend more on the complexity of the function than on the dimensionality of the input, making
it difficult to provide any general rules of thumb. However, as the number of inputs increases, there is certainly more “space” between the training points, suggesting that the number of points needed will generally increase rapidly with the dimensionality of the input. However, this effect is diminished somewhat by the fact that in practice, most models are not strongly sensitive to all of their inputs (O’Hagan, 2006).
In fact, through the process of estimating the correlation parameters (discussed below), those inputs to which the model is most sensitive are identified (with the parametrization of Eq. (3.9), larger values ofξcorrespond to more important inputs). As pointed out by O’Hagan (2006), the Gaussian process approach effectively “projects points down through those smooth [unimportant] dimensions into the lower dimensional space of inputs that matter.” While it is true that there has been limited experience with very high dimensions in practice, and most re- alistic simulations will not respond strongly to a very large number of inputs, O’Hagan (2006) asserts that “GP emulation can be implemented effectively with up to 50 inputs on modern computing platforms.”