  X-ray flashes (XRFs) and X-ray rich gamma-ray bursts (XRGRBs) share many
observational characteristics with long duration GRBs, but the reason for which
their prompt emission peaks at lower photon energies, $E_p$, is still under
debate. Although many different models have been invoked in order to explain
the lower $E_p$ values, their implications for the afterglow emission were not
considered in most cases, mainly because observations of XRF afterglows have
become available only recently. Here we examine the predictions of the various
XRF models for the afterglow emission, and test them against the observations
of XRF 030723 and XRGRB 041006, the events with the best monitored afterglow
light curves in their respective class. We show that most existing XRF models
are hard to reconcile with the observed afterglow light curves, which are very
flat at early times. Such light curves are, however, naturally produced by a
roughly uniform jet with relatively sharp edges that is viewed off-axis (i.e.
from outside of the jet aperture). This type of model self consistently
accommodates both the observed prompt emission and the afterglow light curves
of XRGRB 041006 and XRF 030723, implying viewing angles $\theta_{obs}$ from the
jet axis of $(\theta_{obs}-\theta_0)\sim 0.15\theta_0$ and $\sim \theta_0$,
respectively, where $\theta_0\sim 3$ deg is the jet half-opening angle. This
suggests that GRBs, XRGRBs and XRFs are intrinsically similar relativistic jets
viewed from different angles, corresponding to $\gamma(\theta_{obs}-\theta_0)$
of less than 1, between 1 and a few, and more than a few, respectively, where
$\gamma$ is the Lorentz factor. Future observations with Swift could help test
this unification scheme in which GRBs, XRGRBs and XRFs share the same basic
physics and differ only by their orientation relative to our line of sight.
