Bayesian Inference in Neural Networks

Approximate marginal Bayesian computation and inference are developed
for neural network models. The marginal considerations include:
(i) determination of approximate Bayes factors for model choice about
the number of nonlinear sigmoidal terms; (ii) approximate predictive
density computation for a future observable; (iii) determination of
approximate Bayes estimates for the nonlinear regression function; and
(iv) marginal density computation. Important use is made of the inherent
partial linearity of the model which leads to an explicit marginal posterior
on the nonlinear parameter when used with appropriate conjugate priors. Such
exact marginalization simplifies marginal Bayes calculations so that further
marginalization in the nonlinear parameter can be performed using a Laplace
approximation. The choice of prior and the use of an alternative sigmoidal
lead to posterior invariance in the nonlinear parameter which is discussed
in connection with the lack of sigmoidal identifiability. The proposed
methods are illustrated in the context of two nonlinear data sets: a
nonlinear regression model and a nonlinear autoregressive time
series.