  This paper proposes a Heaviside composite optimization approach and presents
a progressive (mixed) integer programming (PIP) method for solving multi-class
classification and multi-action treatment problems with constraints. A
Heaviside composite function is a composite of a Heaviside function (i.e., the
indicator function of either the open $( \, 0,\infty )$ or closed $[ \,
0,\infty \, )$ interval) with a possibly nondifferentiable function.
Modeling-wise, we show how Heaviside composite optimization provides a unified
formulation for learning the optimal multi-class classification and
multi-action treatment rules, subject to rule-dependent constraints stipulating
a variety of domain restrictions. A Heaviside composite function has an
equivalent discrete formulation, and the resulting optimization problem can in
principle be solved by integer programming (IP) methods. Nevertheless, for
constrained learning problems with large data sets, a straightforward
application of off-the-shelf IP solvers is usually ineffective in achieving
global optimality. To alleviate such a computational burden, our major
contribution is the proposal of the PIP method by leveraging the effectiveness
of state-of-the-art IP solvers for problems of modest sizes. We provide the
theoretical advantage of the PIP method with the connection to continuous
optimization and show that the computed solution is locally optimal for a broad
class of Heaviside composite optimization problems. The numerical performance
of the PIP method is demonstrated by extensive computational experimentation.
