  Continuous integration at scale is costly but essential to software
development. Various test optimization techniques including test selection and
prioritization aim to reduce the cost. Test batching is an effective
alternative, but overlooked technique. This study evaluates parallelization's
effect by adjusting machine count for test batching and introduces two novel
approaches.
  We establish TestAll as a baseline to study the impact of parallelism and
machine count on feedback time. We re-evaluate ConstantBatching and introduce
DynamicBatching, which adapts batch size based on the remaining changes in the
queue. We also propose TestCaseBatching, enabling new builds to join a batch
before full test execution, thus speeding up continuous integration. Our
evaluations utilize Ericsson's results and 276 million test outcomes from
open-source Chrome, assessing feedback time, execution reduction, and providing
access to Chrome project scripts and data.
  The results reveal a non-linear impact of test parallelization on feedback
time, as each test delay compounds across the entire test queue.
ConstantBatching, with a batch size of 4, utilizes up to 72% fewer machines to
maintain the actual average feedback time and provides a constant execution
reduction of up to 75%. Similarly, DynamicBatching maintains the actual average
feedback time with up to 91% fewer machines and exhibits variable execution
reduction of up to 99%. TestCaseBatching holds the line of the actual average
feedback time with up to 81% fewer machines and demonstrates variable execution
reduction of up to 67%. We recommend practitioners use DynamicBatching and
TestCaseBatching to reduce the required testing machines efficiently. Analyzing
historical data to find the threshold where adding more machines has minimal
impact on feedback time is also crucial for resource-effective testing.
