Gaussian process
In probability theory and statistics, a Gaussian process is a stochastic process (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a multivariate normal distribution, i.e. every finite linear combination of them is normally distributed. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space.
A machinelearning algorithm that involves a Gaussian process uses lazy learning and a measure of the similarity between points (the kernel function) to predict the value for an unseen point from training data. The prediction is not just an estimate for that point, but also has uncertainty information—it is a onedimensional Gaussian distribution (which is the marginal distribution at that point).^{[1]}
For some kernel functions, matrix algebra can be used to calculate the predictions using the technique of kriging. When a parameterised kernel is used, optimisation software is typically used to fit a Gaussian process model.
The concept of Gaussian processes is named after Carl Friedrich Gauss because it is based on the notion of the Gaussian distribution (normal distribution). Gaussian processes can be seen as an infinitedimensional generalization of multivariate normal distributions.
Gaussian processes are useful in statistical modelling, benefiting from properties inherited from the normal distribution. For example, if a random process is modelled as a Gaussian process, the distributions of various derived quantities can be obtained explicitly. Such quantities include the average value of the process over a range of times and the error in estimating the average using sample values at a small set of times.
Contents
Definition[edit]
A time continuous stochastic process is Gaussian if and only if for every finite set of indices in the index set
is a multivariate Gaussian random variable.^{[2]} That is the same as saying every linear combination of has a univariate normal (or Gaussian) distribution.
Using characteristic functions of random variables, the Gaussian property can be formulated as follows: is Gaussian if and only if, for every finite set of indices , there are realvalued , with such that the following equality holds for all
 .
where denotes the imaginary unit such that .
The numbers and can be shown to be the covariances and means of the variables in the process.^{[3]}
Variance[edit]
The variance of a Gaussian process is finite at any time , formally^{[4]}^{:p. 515}
 .
Stationarity[edit]
For general stochastic processes strictsense stationarity implies widesense stationarity but not every widesense stationary stochastic process is strictsense stationary. However, for a Gaussian stochastic process the two concepts are equivalent.^{[4]}^{:p. 518}
A Gaussian stochastic process is strictsense stationary if, and only if, it is widesense stationary.
Covariance functions[edit]
A key fact of Gaussian processes is that they can be completely defined by their secondorder statistics.^{[5]} Thus, if a Gaussian process is assumed to have mean zero, defining the covariance function completely defines the process' behaviour. Importantly the nonnegative definiteness of this function enables its spectral decomposition using the Karhunen–Loève expansion. Basic aspects that can be defined through the covariance function are the process' stationarity, isotropy, smoothness and periodicity.^{[6]}^{[7]}
Stationarity refers to the process' behaviour regarding the separation of any two points and . If the process is stationary, it depends on their separation, , while if nonstationary it depends on the actual position of the points and . For example, the special case of an Ornstein–Uhlenbeck process, a Brownian motion process, is stationary.
If the process depends only on , the Euclidean distance (not the direction) between and , then the process is considered isotropic. A process that is concurrently stationary and isotropic is considered to be homogeneous;^{[8]} in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.
Ultimately Gaussian processes translate as taking priors on functions and the smoothness of these priors can be induced by the covariance function.^{[6]} If we expect that for "nearby" input points and their corresponding output points and to be "nearby" also, then the assumption of continuity is present. If we wish to allow for significant displacement then we might choose a rougher covariance function. Extreme examples of the behaviour is the Ornstein–Uhlenbeck covariance function and the squared exponential where the former is never differentiable and the latter infinitely differentiable.
Periodicity refers to inducing periodic patterns within the behaviour of the process. Formally, this is achieved by mapping the input to a two dimensional vector .
Usual covariance functions[edit]
There are a number of common covariance functions:^{[7]}
 Constant :
 Linear:
 white Gaussian noise:
 Squared exponential:
 Ornstein–Uhlenbeck:
 Matérn:
 Periodic:
 Rational quadratic:
Here . The parameter ℓ is the characteristic lengthscale of the process (practically, "how close" two points and have to be to influence each other significantly), δ is the Kronecker delta and σ the standard deviation of the noise fluctuations. Moreover, is the modified Bessel function of order and is the gamma function evaluated at . Importantly, a complicated covariance function can be defined as a linear combination of other simpler covariance functions in order to incorporate different insights about the dataset at hand.
Clearly, the inferential results are dependent on the values of the hyperparameters θ (e.g. ℓ and σ) defining the model's behaviour. A popular choice for θ is to provide maximum a posteriori (MAP) estimates of it with some chosen prior. If the prior is very near uniform, this is the same as maximizing the marginal likelihood of the process; the marginalization being done over the observed process values .^{[7]} This approach is also known as maximum likelihood II, evidence maximization, or empirical Bayes.^{[9]}
Continuity[edit]
For a Gaussian process, continuity in probability is equivalent to meansquare continuity, ^{[10]}^{:145} and continuity with probability one is equivalent to sample continuity.^{[11]}^{:91 "Gaussian processes are discontinuous at fixed points."} The latter implies, but is not implied by, continuity in probability. Continuity in probability holds if and only if the mean and autocovariance are continuous functions. In contrast, sample continuity was challenging even for stationary Gaussian processes (as probably noted first by Andrey Kolmogorov), and more challenging for more general processes.^{[12]}^{:Sect. 2.8} ^{[13]}^{:69,81} ^{[14]}^{:80} ^{[15]} As usual, by a sample continuous process one means a process that admits a sample continuous modification. ^{[16]}^{:292} ^{[17]}^{:424}
Stationary case[edit]
For a stationary Gaussian process some conditions on its spectrum are sufficient for sample continuity, but fail to be necessary. A necessary and sufficient condition, sometimes called DudleyFernique theorem, involves the function defined by
(the righthand side does not depend on due to stationarity). Continuity of in probability is equivalent to continuity of at When convergence of to (as ) is too slow, sample continuity of may fail. Convergence of the following integrals matters:
these two integrals being equal according to integration by substitution The first integrand need not be bounded as thus the integral may converge () or diverge (). Taking for example for large that is, for small one obtains when and when In these two cases the function is increasing on but generally it is not. Moreover, the condition
 there exists such that is monotone on
does not follow from continuity of and the evident relations (for all ) and
Theorem 1. Let be continuous and satisfy Then the condition is necessary and sufficient for sample continuity of
Some history.^{[17]}^{:424} Sufficiency was announced by Xavier Fernique in 1964, but the first proof was published by Richard M. Dudley in 1967.^{[16]}^{:Theorem 7.1} Necessity was proved by Michael B. Marcus and Lawrence Shepp in 1970.^{[18]}^{:380}
There exist sample continuous processes such that they violate condition An example found by Marcus and Shepp ^{[18]}^{:387} is a random lacunary Fourier series
where are independent random variables with standard normal distribution; frequencies are a fast growing sequence; and coefficients satisfy The latter relation implies whence almost surely, which ensures uniform convergence of the Fourier series almost surely, and sample continuity of
Its autocovariation function
is nowhere monotone (see the picture), as well as the corresponding function
Brownian motion as the integral of Gaussian processes[edit]
A Wiener process (aka Brownian motion) is the integral of a white noise generalized Gaussian process. It is not stationary, but it has stationary increments.
The Ornstein–Uhlenbeck process is a stationary Gaussian process.
The Brownian bridge is (like the Ornstein–Uhlenbeck process) an example of a Gaussian process whose increments are not independent.
The fractional Brownian motion is a Gaussian process whose covariance function is a generalisation of that of the Wiener process.
Applications[edit]
A Gaussian process can be used as a prior probability distribution over functions in Bayesian inference.^{[7]}^{[20]} Given any set of N points in the desired domain of your functions, take a multivariate Gaussian whose covariance matrix parameter is the Gram matrix of your N points with some desired kernel, and sample from that Gaussian.
Inference of continuous values with a Gaussian process prior is known as Gaussian process regression, or kriging; extending Gaussian process regression to multiple target variables is known as cokriging.^{[21]} Gaussian processes are thus useful as a powerful nonlinear multivariate interpolation tool. Gaussian process regression can be further extended to address learning tasks in both supervised (e.g. probabilistic classification^{[7]}) and unsupervised (e.g. manifold learning^{[5]}) learning frameworks.
Gaussian processes can also be used in the context of mixture of experts models, for example.^{[22]}^{[23]} The underlying rationale of such a learning framework consists in the assumption that a given mapping cannot be well captured by a single Gaussian process model. Instead, the observation space is divided into subsets, each of which is characterized by a different mapping function; each of these is learned via a different Gaussian process component in the postulated mixture.
Gaussian process prediction, or Kriging[edit]
When concerned with a general Gaussian process regression problem (Kriging), it is assumed that for a Gaussian process f observed at coordinates x, the vector of values is just one sample from a multivariate Gaussian distribution of dimension equal to number of observed coordinates . Therefore, under the assumption of a zeromean distribution, , where is the covariance matrix between all possible pairs for a given set of hyperparameters θ.^{[7]} As such the log marginal likelihood is:
and maximizing this marginal likelihood towards θ provides the complete specification of the Gaussian process f. One can briefly note at this point that the first term corresponds to a penalty term for a model's failure to fit observed values and the second term to a penalty term that increases proportionally to a model's complexity. Having specified θ making predictions about unobserved values at coordinates x* is then only a matter of drawing samples from the predictive distribution where the posterior mean estimate A is defined as
and the posterior variance estimate B is defined as:
where is the covariance between the new coordinate of estimation x* and all other observed coordinates x for a given hyperparameter vector θ, and are defined as before and is the variance at point x* as dictated by θ. It is important to note that practically the posterior mean estimate (the "point estimate") is just a linear combination of the observations ; in a similar manner the variance of is actually independent of the observations . A known bottleneck in Gaussian process prediction is that the computational complexity of prediction is cubic in the number of points x and as such can become unfeasible for larger data sets.^{[6]} Works on sparse Gaussian processes, that usually are based on the idea of building a representative set for the given process f, try to circumvent this issue.^{[24]}^{[25]}
See also[edit]
 Bayes linear statistics
 Bayesian interpretation of regularization
 Kriging
 Gaussian free field
 Gradientenhanced kriging (GEK)
References[edit]
 ^ "Platypus Innovation: A Simple Intro to Gaussian Processes (a great data modelling tool)". 20160510.

^ MacKay, David, J.C. (2003). Information Theory, Inference, and Learning Algorithms (PDF). Cambridge University Press. p. 540. ISBN 9780521642989.
The probability distribution of a function is a Gaussian processes if for any finite selection of points , the density is a Gaussian
 ^ Dudley, R.M. (1989). Real Analysis and Probability. Wadsworth and Brooks/Cole.
 ^ ^{a} ^{b} Amos Lapidoth (8 February 2017). A Foundation in Digital Communication. Cambridge University Press. ISBN 9781107177321.
 ^ ^{a} ^{b} Bishop, C.M. (2006). Pattern Recognition and Machine Learning. Springer. ISBN 9780387310732.
 ^ ^{a} ^{b} ^{c} Barber, David (2012). Bayesian Reasoning and Machine Learning. Cambridge University Press. ISBN 9780521518147.
 ^ ^{a} ^{b} ^{c} ^{d} ^{e} ^{f} Rasmussen, C.E.; Williams, C.K.I (2006). Gaussian Processes for Machine Learning. MIT Press. ISBN 9780262182539.
 ^ Grimmett, Geoffrey; David Stirzaker (2001). Probability and Random Processes. Oxford University Press. ISBN 9780198572220.
 ^ Seeger, Matthias (2004). "Gaussian Processes for Machine Learning". International Journal of Neural Systems. 14 (2): 69–104. CiteSeerX 10.1.1.71.1079. doi:10.1142/s0129065704001899. PMID 15112367.
 ^ Dudley, R. M. (1975). "The Gaussian process and how to approach it" (PDF). Proceedings of the International Congress of Mathematicians. 2. pp. 143–146.
 ^ Dudley, R. M. (1973). "Sample functions of the Gaussian process". Annals of Probability. 1 (1): 66–103. doi:10.1007/9781441958211_13. ISBN 9781441958204.
 ^ Talagrand, Michel (2014). Upper and lower bounds for stochastic processes: modern methods and classical problems. Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge / A Series of Modern Surveys in Mathematics. Springer, Heidelberg. ISBN 9783642540745.
 ^ Ledoux, Michel (1994). "Isoperimetry and Gaussian analysis". Lecture Notes in Mathematics. 1648. Springer, Berlin. pp. 165–294. doi:10.1007/BFb0095676. ISBN 9783540620556.
 ^ Adler, Robert J. (1990). "An introduction to continuity, extrema, and related topics for general Gaussian processes". Lecture NotesMonograph Series. Institute of Mathematical Statistics. 12: i–155. JSTOR 4355563.
 ^ Berman, Simeon M. (1992). "Review of: Adler 1990 'An introduction to continuity...'". Mathematical Reviews. MR 1088478.
 ^ ^{a} ^{b} Dudley, R. M. (1967). "The sizes of compact subsets of Hilbert space and continuity of Gaussian processes". Journal of Functional Analysis. 1 (3): 290–330. doi:10.1016/00221236(67)900171.
 ^ ^{a} ^{b} Marcus, M.B.; Shepp, Lawrence A. (1972). "Sample behavior of Gaussian processes". Proceedings of the sixth Berkeley symposium on mathematical statistics and probability, vol. II: probability theory. Univ. California, Berkeley. pp. 423–441.
 ^ ^{a} ^{b} Marcus, Michael B.; Shepp, Lawrence A. (1970). "Continuity of Gaussian processes". Transactions of the American Mathematical Society. 151 (2): 377–391. doi:10.2307/1995502. JSTOR 1995502.
 ^ The documentation for scikitlearn also has similar examples.
 ^ Liu, W.; Principe, J.C.; Haykin, S. (2010). Kernel Adaptive Filtering: A Comprehensive Introduction. John Wiley. ISBN 9780470447536.
 ^ Stein, M.L. (1999). Interpolation of Spatial Data: Some Theory for Kriging. Springer.
 ^ Platanios, Emmanouil A.; Chatzis, Sotirios P. (2014). "Gaussian ProcessMixture Conditional Heteroscedasticity". IEEE Transactions on Pattern Analysis and Machine Intelligence. 36 (5): 888–900. doi:10.1109/TPAMI.2013.183. PMID 26353224.
 ^ Chatzis, Sotirios P. (2013). "A latent variable Gaussian process model with Pitman–Yor process priors for multiclass classification". Neurocomputing. 120: 482–489. doi:10.1016/j.neucom.2013.04.029.
 ^ Smola, A.J.; Schoellkopf, B. (2000). "Sparse greedy matrix approximation for machine learning". Proceedings of the Seventeenth International Conference on Machine Learning: 911–918.
 ^ Csato, L.; Opper, M. (2002). "Sparse online Gaussian processes". Neural Computation. 14 (3): 641–668. CiteSeerX 10.1.1.335.9713. doi:10.1162/089976602317250933. PMID 11860686.
External links[edit]
 The Gaussian Processes Web Site, including the text of Rasmussen and Williams' Gaussian Processes for Machine Learning
 A gentle introduction to Gaussian processes
 A Review of Gaussian Random Fields and Correlation Functions
 Efficient Reinforcement Learning using Gaussian Processes
Software[edit]
 STK: a Small (Matlab/Octave) Toolbox for Kriging and GP modeling
 Kriging module in UQLab framework (Matlab)
 Matlab/Octave function for stationary Gaussian fields
 Yelp MOE – A black box optimization engine using Gaussian process learning
 ooDACE – A flexible objectoriented Kriging matlab toolbox.
 GPstuff – Gaussian process toolbox for Matlab and Octave
 GPy – A Gaussian processes framework in Python
 Interactive Gaussian process regression demo
 Basic Gaussian process library written in C++11
 scikitlearn – A machine learning library for Python which includes Gaussian process regression and classification
 [1]  The Kriging toolKit (KriKit) is developed at the Institute of Bio and Geosciences 1 (IBG1) of Forschungszentrum Jülich (FZJ)