Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Exploiting an Oracle that Reports AUC Scores in Machine Learning
  Contests   In machine learning contests such as the ImageNet Large Scale Visual
Recognition Challenge and the KDD Cup, contestants can submit candidate
solutions and receive from an oracle (typically the organizers of the
competition) the accuracy of their guesses compared to the ground-truth labels.
One of the most commonly used accuracy metrics for binary classification tasks
is the Area Under the Receiver Operating Characteristics Curve (AUC). In this
paper we provide proofs-of-concept of how knowledge of the AUC of a set of
guesses can be used, in two different kinds of attacks, to improve the accuracy
of those guesses. On the other hand, we also demonstrate the intractability of
one kind of AUC exploit by proving that the number of possible binary labelings
of $n$ examples for which a candidate solution obtains a AUC score of $c$ grows
exponentially in $n$, for every $c\in (0,1)$.

7
The categories of this paper are: hep-th
3
The categories of this paper are: physics
6
The categories of this paper are: astro-ph
14
The categories of this paper are: math-ph
35
The categories of this paper are: mtrl-th
25
The categories of this paper are: patt-sol
21
The categories of this paper are: adap-org
24
The categories of this paper are: dg-ga
23
The categories of this paper are: solv-int
2
The categories of this paper are: math
0
The categories of this paper are: hep-ph
30
The categories of this paper are: q-alg
22
The categories of this paper are: chao-dyn
33
The categories of this paper are: plasm-ph
16
The categories of this paper are: stat
31
The categories of this paper are: ao-sci