FactoredBellmanErr (P, R, γ, H, O, wb )
//P *is the factored transition model.*
//R is the set of factored reward functions. //γ is the discount factor.

//H *is the set of basis functions* H = {h1**, . . . , h**k}.

//O stores the elimination order.

//wb *are the weights for the linear value function.*
//Return the Bellman error for the value function Hwb.

//Cache the backprojections of the basis functions.

For each basis function hi ∈ H; for each action a:
Let g a i = **Backproj**a (hi).

//Compute decision list policy for value function Hwb.

Let ∆ = b DecisionListPolicy(Ra + γPi wbig a i
).

//Initialize indicators.

Let I = {}.

//Initialize Bellman error.

Let ε = 0.

//For every branch of the decision list policy, generate the relevant cost networks, solve it with variable elimination, and update the indicators to constraint the state space for future branches.

For each branch htj , aj i in the decision list policy ∆b :
//Instantiate the variables in Tj *to the assignment given in* tj .

Instantiate the set of functions {wb1(h1−γg aj 1
)**, . . . ,** wbk(hk−γg aj k
)} with the partial state assignment tj and store in C.

Instantiate the target functions Raj with the partial state assignment tj and store in b.

Instantiate the indicator functions I with the partial state assignment tj and store in I
′.

//Use variable elimination to solve first cost network, and update Bellman error, if error for this branch is larger.

Let ε **= max (**ε, VariableElimination(C − b + I
′, O)).

//Use variable elimination to solve second cost network, and update Bellman error, if error for this branch is larger.

Let ε **= max (**ε, VariableElimination(−C + b + I
′, O)).

//Update the indicator functions.

Let Ij (x) = −∞1(x = tj ) and update the indicators I = **I ∪ I**j .

Return ε.

Figure 10: Algorithm for computing Bellman error for factored value function Hwb .