height:0.954008em;"><span style="top:-3.2029em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight" style=""><span class="mord mtight" style="">2</span></span></span></span></span></span></span></span></span></span></span></span></span></span></p>
<p>Squared ReLU is used as the activation function in the  <a href="../feed_forward.html">position wise feedforward module</a>.</p>

        </div>
        <div class='code'>
            <div class="