text: In many fields of social and industrial sciences, simulation is crucial in comprehending a target system. A major task in simulation is the estimation of optimal parameters to express the observed data need to directly elucidate the properties of the target system as a modeling based on the expert's domain knowledge. However, skilled human experts struggle to obtain the desired parameters. Data assimilation therefore becomes an unavoidable task to reduce the cost of simulator optimization. Another necessary task is extrapolation; in many practical cases, predictions based on simulation results will be often outside of the dominant range of a given data area, and this is referred to as the covariate shift. This paper focuses on a regression problem with covariate shift. While the parameter estimation for the covariate shift has been studied thoroughly in parametric and nonparametric settings, conventional statistical methods of parameter searching are not applicable in the data assimilation of the simulation owing to the properties of the likelihood function: intractable or nondifferentiable. Hence, we propose a novel framework of Bayesian inference based on kernel mean embedding. This framework allows for predictions in covariate shift situations, and its effectiveness is evaluated in both synthetic numerical experiments and a widely used production simulator reproducing real-world manufacturing factories.
  @cite_1: As the power of computing has grown over the past few decades, the field of machine learning has advanced rapidly in both theory and practice. Machine learning methods are usually based on the assumption that the data generation mechanism does not change over time. Yet real-world applications of machine learning, including image recognition, natural language processing, speech recognition, robot control, and bioinformatics, often violate this common assumption. Dealing with non-stationarity is one of modern machine learning's greatest challenges. This book focuses on a specific non-stationary environment known as covariate shift, in which the distributions of inputs (queries) change but the conditional distribution of outputs (answers) is unchanged, and presents machine learning theory, algorithms, and applications to overcome this variety of non-stationarity. After reviewing the state-of-the-art research in the field, the authors discuss topics that include learning under covariate shift, model selection, importance estimation, and active learning. They describe such real world applications of covariate shift adaption as brain-computer interface, speaker identification, and age prediction from facial images. With this book, they aim to encourage future research in machine learning, statistics, and engineering that strives to create truly autonomous learning machines able to learn under non-stationarity.
  @cite_0: 
  @cite_4: Abstract A class of predictive densities is derived by weighting the observed samples in maximizing the log-likelihood function. This approach is effective in cases such as sample surveys or design of experiments, where the observed covariate follows a different distribution than that in the whole population. Under misspecification of the parametric model, the optimal choice of the weight function is asymptotically shown to be the ratio of the density function of the covariate in the population to that in the observations. This is the pseudo-maximum likelihood estimation of sample surveys. The optimality is defined by the expected Kullback–Leibler loss, and the optimal weight is obtained by considering the importance sampling identity. Under correct specification of the model, however, the ordinary maximum likelihood estimate (i.e. the uniform weight) is shown to be optimal asymptotically. For moderate sample size, the situation is in between the two extreme cases, and the weight function is selected by minimizing a variant of the information criterion derived as an estimate of the expected loss. The method is also applied to a weighted version of the Bayesian predictive density. Numerical examples as well as Monte-Carlo simulations are shown for polynomial regression. A connection with the robust parametric estimation is discussed.
  @cite_2: In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian generalization error and establish its upper bound under a very general setting. Our important finding is that lower order terms---which can be ignored in the absence of the distribution change---play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.
summary: A series of studies exist for covariate shift assuming that a regression function is an analytical function @cite_4 @cite_2 @cite_1 , such as kernel ridge regression @cite_0 . In our problem, however, we assume that the functional relation of the regression model @math is only given as a nonanalytical function: a simulation. The difference between kernel ridge regression and the proposed method with formulation is presented in .