  By drawing on ideas from optimisation theory, artificial neural networks
(ANN), graph embeddings and sparse representations, I develop a novel
technique, termed SENNS (Sparse Extraction Neural NetworkS), aimed at
addressing the feature extraction problem. The proposed method uses (preferably
deep) ANNs for projecting input attribute vectors to an output space wherein
pairwise distances are maximized for vectors belonging to different classes,
but minimized for those belonging to the same class, while simultaneously
enforcing sparsity on the ANN outputs. The vectors that result from the
projection can then be used as features in any classifier of choice.
Mathematically, I formulate the proposed method as the minimisation of an
objective function which can be interpreted, in the ANN output space, as a
negative factor of the sum of the squares of the pair-wise distances between
output vectors belonging to different classes, added to a positive factor of
the sum of squares of the pair-wise distances between output vectors belonging
to the same classes, plus sparsity and weight decay terms. To derive an
algorithm for minimizing the objective function via gradient descent, I use the
multi-variate version of the chain rule to obtain the partial derivatives of
the function with respect to ANN weights and biases, and find that each of the
required partial derivatives can be expressed as a sum of six terms. As it
turns out, four of those six terms can be computed using the standard back
propagation algorithm; the fifth can be computed via a slight modification of
the standard backpropagation algorithm; while the sixth one can be computed via
simple arithmetic. Finally, I propose experiments on the ARABASE Arabic corpora
of digits and letters, the CMU PIE database of faces, the MNIST digits
database, and other standard machine learning databases.
