Bartlett, P. L., & Baxter, J. (1999). Hebbian synaptic modifications in spiking neurons that learn.

Tech. rep., Research School of Information Sciences and Engineering, Australian National University. http://csl.anu.edu.au/∼bartlett/papers/BartlettBaxter-Nov99.ps.gz.

Bartlett, P. L., & Baxter, J. (2001). Estimation and approximation bounds for gradient-based reinforcement learning. *Journal of Computer and Systems Sciences,* 62. Invited Paper: Special Issue on COLT 2000.

Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. *IEEE Transactions on Systems, Man, and Cybernetics*, SMC-13, 834–846.

Baxter, J., Bartlett, P. L., & Weaver, L. (2001). Gradient-ascent algorithms and experiments with infinite-horizon, policy-gradient estimation. *Journal of Artificial Intelligence Research,* 14.

Baxter, J., Tridgell, A., & Weaver, L. (2000). Learning to play chess using temporal-differences.

Machine Learning, 40(3), 243–263.

Bertsekas, D. P., & Tsitsiklis, J. N. (1996). *Neuro-Dynamic Programming*. Athena Scientific.

Bertsekas, D. P. (1995). *Dynamic Programming and Optimal Control, Vol II*. Athena Scientific. Breiman, L. (1966). *Probability*. Addison-Wesley.

Cao, X.-R., & Wan, Y.-W. (1998). Algorithms for Sensitivity Analysis of Markov Chains Through Potentials and Perturbation Realization. *IEEE Transactions on Control Systems Technology*, 6, 482–492.

Dudley, R. M. (1989). *Real Analysis and Probability*. Wadsworth & Brooks/Cole, Belmont, California.

Glynn, P. W. (1986). Stochastic approximation for monte-carlo optimization. In *Proceedings of the* 1986 Winter Simulation Conference, pp. 356–365.

Glynn, P. W. (1990). Likelihood ratio gradient estimation for stochastic systems. *Communications* of the ACM, 33, 75–84.

Glynn, P. W., & L'Ecuyer, P. (1995). Likelihood ratio gradient estimation for regenerative stochastic recursions. *Advances in Applied Probability, 27, 4 (1995),* 27, 1019–1053.

Ho, Y.-C., & Cao, X.-R. (1991). *Perturbation Analysis of Discrete Event Dynamic Systems*. Kluwer Academic, Boston.

Jaakkola, T., Singh, S. P., & Jordan, M. I. (1995). Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems. In Tesauro, G., Touretzky, D., & Leen, T. (Eds.),
Advances in Neural Information Processing Systems, Vol. 7. MIT Press, Cambridge, MA.

Kimura, H., & Kobayashi, S. (1998a). An analysis of actor/critic algorithms using eligibility traces:
Reinforcement learning with imperfect value functions. In *Fifteenth International Conference* on Machine Learning, pp. 278–286.