ins a lot faster than the vanilla transformer.</p>
<h3>Squared ReLU</h3>
<p>The most effective modification found by the search is using a square ReLU instead of ReLU in the <a href="../feed_forward.html">position-wise feedforward module</a>.</p>
<p><span ><span class="katex-display"><span class="katex"><span aria-hidden="true" class="katex-html"><span class="base"><span class="strut" style="height:1.204008em;vertical-align:-0.25em;"></span><span class="mord coloredeq eqa" style=""><span class="mord mathnor