  The generalized nonlinear Schr\"odinger equation with full dispersion (FDNLS)
is considered in the semiclassical regime. The Whitham modulation equations are
obtained for the FDNLS equation with general linear dispersion and a
generalized, local nonlinearity. Assuming the existence of a four-parameter
family of two-phase solutions, a multiple-scales approach yields a system of
four independent, first order, quasi-linear conservation laws of hydrodynamic
type that correspond to the slow evolution of the two wavenumbers, mass, and
momentum of modulated periodic traveling waves. The modulation equations are
further analyzed in the dispersionless and weakly nonlinear regimes. The
ill-posedness of the dispersionless equations corresponds to the classical
criterion for modulational instability (MI). For modulations of linear waves,
ill-posedness coincides with the generalized MI criterion, recently identified
by Amiranashvili and Tobisch (New J. Phys. 21 (2019)). A new instability index
is identified by the transition from real to complex characteristics for the
weakly nonlinear modulation equations. This instability is associated with
long-wavelength modulations of nonlinear two-phase wavetrains and can exist
even when the corresponding one-phase wavetrain is stable according to the
generalized MI criterion. Another interpretation is that, while infinitesimal
perturbations of a periodic wave may not grow, small but finite amplitude
perturbations may grow, hence this index identifies a nonlinear instability
mechanism for one-phase waves. Classifications of instability indices for
multiple FDNLS equations with higher order dispersion, including applications
to finite depth water waves and the discrete NLS equation are presented and
compared with direct numerical simulations.
