  General physical scene understanding requires more than simply localizing and
recognizing objects -- it requires knowledge that objects can have different
latent properties (e.g., mass or elasticity), and that those properties affect
the outcome of physical events. While there has been great progress in physical
and video prediction models in recent years, benchmarks to test their
performance typically do not require an understanding that objects have
individual physical properties, or at best test only those properties that are
directly observable (e.g., size or color). This work proposes a novel dataset
and benchmark, termed Physion++, that rigorously evaluates visual physical
prediction in artificial systems under circumstances where those predictions
rely on accurate estimates of the latent physical properties of objects in the
scene. Specifically, we test scenarios where accurate prediction relies on
estimates of properties such as mass, friction, elasticity, and deformability,
and where the values of those properties can only be inferred by observing how
objects move and interact with other objects or fluids. We evaluate the
performance of a number of state-of-the-art prediction models that span a
variety of levels of learning vs. built-in knowledge, and compare that
performance to a set of human predictions. We find that models that have been
trained using standard regimes and datasets do not spontaneously learn to make
inferences about latent properties, but also that models that encode objectness
and physical states tend to make better predictions. However, there is still a
huge gap between all models and human performance, and all models' predictions
correlate poorly with those made by humans, suggesting that no state-of-the-art
model is learning to make physical predictions in a human-like way. Project
page: https://dingmyu.github.io/physion_v2/
