  Many problems that arise in machine learning domain deal with nonlinearity
and quite often demand users to obtain global optimal solutions rather than
local optimal ones. Optimization problems are inherent in machine learning
algorithms and hence many methods in machine learning were inherited from the
optimization literature. Popularly known as the initialization problem, the
ideal set of parameters required will significantly depend on the given
initialization values. The recently developed TRUST-TECH (TRansformation Under
STability-reTaining Equilibria CHaracterization) methodology systematically
explores the subspace of the parameters to obtain a complete set of local
optimal solutions. In this thesis work, we propose TRUST-TECH based methods for
solving several optimization and machine learning problems. Two stages namely,
the local stage and the neighborhood-search stage, are repeated alternatively
in the solution space to achieve improvements in the quality of the solutions.
Our methods were tested on both synthetic and real datasets and the advantages
of using this novel framework are clearly manifested. This framework not only
reduces the sensitivity to initialization, but also allows the flexibility for
the practitioners to use various global and local methods that work well for a
particular problem of interest. Other hierarchical stochastic algorithms like
evolutionary algorithms and smoothing algorithms are also studied and
frameworks for combining these methods with TRUST-TECH have been proposed and
evaluated on several test systems.
