Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Towards Robust and Reproducible Active Learning Using Neural Networks   Active learning (AL) is a promising ML paradigm that has the potential to
parse through large unlabeled data and help reduce annotation cost in domains
where labelling data can be prohibitive. Recently proposed neural network based
AL methods use different heuristics to accomplish this goal. In this study, we
demonstrate that under identical experimental settings, different types of AL
algorithms (uncertainty based, diversity-based, and committee based) produce an
inconsistent gain over random sampling baseline. Through a variety of
experiments, controlling for sources of stochasticity, we show that variance in
performance metrics achieved by AL algorithms can lead to results that are not
consistent with the previously reported results. We also found that under
strong regularization, AL methods show marginal or no advantage over the random
sampling baseline under a variety of experimental conditions. Finally, we
conclude with a set of recommendations on how to assess the results using a new
AL algorithm to ensure results are reproducible and robust under changes in
experimental conditions. We share our codes to facilitate AL evaluations. We
believe our findings and recommendations will help advance reproducible
research in AL using neural networks.

20
The categories of this paper are: acc-phys
15
The categories of this paper are: nucl-ex
27
The categories of this paper are: comp-gas
30
The categories of this paper are: q-alg
28
The categories of this paper are: alg-geom
7
The categories of this paper are: hep-th
9
The categories of this paper are: nlin
23
The categories of this paper are: solv-int
18
The categories of this paper are: econ
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
35
The categories of this paper are: mtrl-th
21
The categories of this paper are: adap-org
22
The categories of this paper are: chao-dyn
29
The categories of this paper are: funct-an