  Many autonomous systems forecast aspects of the future in order to aid
decision-making. For example, self-driving vehicles and robotic manipulation
systems often forecast future object poses by first detecting and tracking
objects. However, this detect-then-forecast pipeline is expensive to scale, as
pose forecasting algorithms typically require labeled sequences of object
poses, which are costly to obtain in 3D space. Can we scale performance without
requiring additional labels? We hypothesize yes, and propose inverting the
detect-then-forecast pipeline. Instead of detecting, tracking and then
forecasting the objects, we propose to first forecast 3D sensor data (e.g.,
point clouds with $100$k points) and then detect/track objects on the predicted
point cloud sequences to obtain future poses, i.e., a forecast-then-detect
pipeline. This inversion makes it less expensive to scale pose forecasting, as
the sensor data forecasting task requires no labels. Part of this work's focus
is on the challenging first step -- Sequential Pointcloud Forecasting (SPF),
for which we also propose an effective approach, SPFNet. To compare our
forecast-then-detect pipeline relative to the detect-then-forecast pipeline, we
propose an evaluation procedure and two metrics. Through experiments on a
robotic manipulation dataset and two driving datasets, we show that SPFNet is
effective for the SPF task, our forecast-then-detect pipeline outperforms the
detect-then-forecast approaches to which we compared, and that pose forecasting
performance improves with the addition of unlabeled data.
