 this change, the JobManager doesn&rsquo;t need to keep all the copies of ShuffleDescriptors in heap memory until they are sent. Moreover, the frequency of garbage collections for large-scale jobs is significantly reduced. Also, task deployment will be faster since there will be no bottleneck during task deployment anymore, because the DFS provides multiple distributed nodes for TaskManagers to download the ShuffleDescriptors from.</p>
<center>
<br/>
<img src="/img/blog/2022-01-05-scheduler-performance/3-ho