  The kinetic Sunyaev Zel'dovich (kSZ) and moving lens effects, secondary
contributions to the cosmic microwave background (CMB), carry significant
cosmological information due to their dependence on the large-scale peculiar
velocity field. Previous work identified a promising means of extracting this
cosmological information using a set of quadratic estimators for the radial and
transverse components of the velocity field. These estimators are based on the
statistically anisotropic components of the cross-correlation between the CMB
and a tracer of large scale structure, such as a galaxy redshift survey. In
this work, we assess the challenges to the program of velocity reconstruction
posed by various foregrounds and systematics in the CMB and galaxy surveys, as
well as biases in the quadratic estimators. To do so, we further develop the
quadratic estimator formalism and implement a numerical code for computing
properly correlated spectra for all the components of the CMB
(primary/secondary blackbody components and foregrounds) and a photometric
redshift survey, with associated redshift errors, to allow for accurate
forecasting. We create a simulation framework for generating realizations of
properly correlated CMB maps and redshift binned galaxy number counts, assuming
the underlying fields are Gaussian, and use this to validate a velocity
reconstruction pipeline and assess map-based systematics such as masking. We
highlight the most significant challenges for velocity reconstruction, which
include biases associated with: modelling errors, characterization of redshift
errors, and coarse graining of cosmological fields on our past light cone.
Despite these challenges, the outlook for velocity reconstruction is quite
optimistic, and we use our reconstruction pipeline to confirm that these
techniques will be feasible with near-term CMB experiments and photometric
galaxy redshift surveys.
