# A Model Of Inductive Bias Learning

Jonathan Baxter JONATHAN.BAXTER@ANU.EDU.AU
Research School of Information Sciences and Engineering Australian National University, Canberra 0200, Australia

## Abstract

A major problem in machine learning is that of inductive bias: how to choose a learner's hypothesis space so that it is large enough to contain a solution to the problem being learnt, yet small enough to ensure reliable generalization from reasonably-sized training sets. Typically such bias is supplied by hand through the skill and insights of experts. In this paper a model for automatically learning bias is investigated. The central assumption of the model is that the learner is embedded within an *environment* of related learning tasks. Within such an environment the learner can sample from multiple tasks, and hence it can search for a hypothesis space that contains good solutions to many of the problems in the environment. Under certain restrictions on the set of all hypothesis spaces available to the learner, we show that a hypothesis space that performs well on a sufficiently large number of training tasks will also perform well when learning novel tasks in the same environment. Explicit bounds are also derived demonstrating that learning multiple tasks within an environment of related tasks can potentially give much better generalization than learning a single task.

## 1. Introduction

Often the hardest problem in any machine learning task is the initial choice of hypothesis space; it has to be large enough to contain a solution to the problem at hand, yet small enough to ensure good generalization from a small number of examples (Mitchell, 1991). Once a suitable bias has been found, the actual learning task is often straightforward. Existing methods of bias generally require the input of a human expert in the form of heuristics and domain knowledge (for example, through the selection of an appropriate set of features). Despite their successes, such methods are clearly limited by the accuracy and reliability of the expert's knowledge and also by the extent to which that knowledge can be transferred to the learner. Thus it is natural to search for methods for automatically learning the bias.

In this paper we introduce and analyze a formal model of *bias learning* that builds upon the PAC model of machine learning and its variants (Vapnik, 1982; Valiant, 1984; Blumer, Ehrenfeucht, Haussler, & Warmuth, 1989; Haussler, 1992). These models typically take the following general form: the learner is supplied with a hypothesis space H and training data z = f(x1; y1); : : : ; (xm; ym)g drawn independently according to some underlying distribution P on X - Y . Based on the information contained in z, the learner's goal is to select a hypothesis h : X ! Y from H minimizing some measure erP (h) of expected loss with respect to P (for example, in the case of squared loss erP (h) := E (x;y)P (h(x)  y)
2
). In such models the learner's bias is represented by the choice of H; if H does not contain a good solution to the problem, then, regardless of how much data the learner receives, it cannot learn.

Of course, the best way to bias the learner is to supply it with an H containing just a single optimal hypothesis. But finding such a hypothesis is precisely the original learning problem, so in the