span class="math notranslate nohighlight">\(U^\pi (s) = \mathbb E_{s_1,...,s_t}\left[\sum_t \gamma^t R(s_t)\right]\)</span></p>
<ul>
<li><p>when we use discounted utilities, <span class="math notranslate nohighlight">\(\pi\)</span> is independent of starting state</p></li>
<li><p><span class="math notranslate nohighlight">\(\pi^*(s) = \underset{\pi}{\text{argmax}} \: U^\pi (s) = \underset{a}{\text{argmax}} \sum_{s'} P(s' \vert s,a) U(s')\)</span></p></li>
</ul>
</li>
<li><p><em>experience replay</em> - inst