\theta}(\tau)}\left[\sum_{t} r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right]\)</span></p>
<ul>
<li><p><em>Value-based</em>: estimate value function or <span class="math notranslate nohighlight">\(Q\)</span>-function of the optimal policy (no explicit policy)</p></li>
<li><p><em>Policy gradients</em>: directly differentiate the above objective</p>
<ul>
<li><p>policy network <span class="math notranslate nohighlight">\(\pi_\theta(a|s)\)</span></p></li>
<li><p>gradients are extremely noisy compared to supe