text: Current algorithmic approaches for piecewise affine motion estimation are based on alternating motion segmentation and estimation. We propose a new method to estimate piecewise affine motion fields directly without intermediate segmentation. To this end, we reformulate the problem by imposing piecewise constancy of the parameter field, and derive a specific proximal splitting optimization scheme. A key component of our framework is an efficient 1D piecewise-affine estimator for vector-valued signals. The first advantage of our approach over segmentation-based methods is its absence of initialization. The second advantage is its lower computational cost, which is independent of the complexity of the motion field. In addition to these features, we demonstrate competitive accuracy with other piecewise-parametric methods on standard evaluation benchmarks. Our new regularization scheme also outperforms the more standard use of total variation and total generalized variation.
  @cite_30: The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. The typical formulation, however, has changed little since the work of Horn and Schunck. We attempt to uncover what has made recent advances possible through a thorough analysis of how the objective function, the optimization method, and modern implementation practices influence accuracy. We discover that "classical" flow formulations perform surprisingly well when combined with modern optimization and implementation techniques. One key implementation detail is the median filtering of intermediate flow fields during optimization. While this improves the robustness of classical methods it actually leads to higher energy solutions, meaning that these methods are not optimizing the original objective function. To understand the principles behind this phenomenon, we derive a new objective function that formalizes the median filtering heuristic. This objective function includes a non-local smoothness term that robustly integrates flow estimates over large spatial neighborhoods. By modifying this new term to include information about flow and image boundaries we develop a method that can better preserve motion details. To take advantage of the trend towards video in wide-screen format, we further introduce an asymmetric pyramid downsampling scheme that enables the estimation of longer range horizontal motions. The methods are evaluated on the Middlebury, MPI Sintel, and KITTI datasets using the same parameter settings.
  @cite_38: This article deals with analysis of the dynamic content of a scene from an image sequence irrespective of the static or dynamic nature of the camera. The tasks involved can be the detection of moving objects in a scene observed by a mobile camera, or the identification of the movements of some relevant components of the scene relatively to the camera. This problem basically requires a motion-based segmentation step. We present a motion-based segmentation method relying on 2-D affine motion models and a statistical regularization approach which ensures stable motion-based partitions. This can be done without the explicit estimation of optic flow fields. Besides these partitions are linked in time. Therefore, the motion interpretation process can be performed on more than two successive frames. The ability to follow a given coherently moving region within an interval of several images of the sequence makes the interpretation process more robust and more comprehensive. Identification of the kinematic components of the scene is induced from an intermediate layer accomplishing a generic qualitative motion labeling. No 3-D measurements are required. Results obtained on several real-image sequences corresponding to complex outdoor situations are reported.
  @cite_37: Abstract : This reprint will introduce and study the most basic properties of three new variational problems which are suggested by applications to computer vision. In computer vision, a fundamental problem is to appropriately decompose the domain R of a function g (x,y) of two variables. This problem starts by describing the physical situation which produces images: assume that a three-dimensional world is observed by an eye or camera from some point P and that g1(rho) represents the intensity of the light in this world approaching the point sub 1 from a direction rho. If one has a lens at P focusing this light on a retina or a film-in both cases a plane domain R in which we may introduce coordinates x, y then let g(x,y) be the strength of the light signal striking R at a point with coordinates (x,y); g(x,y) is essentially the same as sub 1 (rho) -possibly after a simple transformation given by the geometry of the imaging syste. The function g(x,y) defined on the plane domain R will be called an image. What sort of function is g? The light reflected off the surfaces Si of various solid objects O sub i visible from P will strike the domain R in various open subsets R sub i. When one object O1 is partially in front of another object O2 as seen from P, but some of object O2 appears as the background to the sides of O1, then the open sets R1 and R2 will have a common boundary (the 'edge' of object O1 in the image defined on R) and one usually expects the image g(x,y) to be discontinuous along this boundary. (JHD)
  @cite_8: We propose a unified variational formulation for joint motion estimation and segmentation with explicit occlusion handling. This is done by a multi-label representation of the flow field, where each label corresponds to a parametric representation of the motion. We use a convex formulation of the multi-label Potts model with label costs and show that the asymmetric map-uniqueness criterion can be integrated into our formulation by means of convex constraints. Explicit occlusion handling eliminates errors otherwise created by the regularization. As occlusions can occur only at object boundaries, a large number of objects may be required. By using a fast primal-dual algorithm we are able to handle several hundred motion segments. Results are shown on several classical motion segmentation and optical flow examples.
  @cite_41: We present a new approach to integrated motion estimation and segmentation by combining methods from discrete and continuous optimization. The velocity of each of a set of regions is modeled as a Gaussian-distributed random variable and motion models and segmentation are obtained by alternated maximization of a Bayesian a-posteriori probability. We show that for fixed segmentation the model parameters are given by a closed-form solution. Given the velocities, the segmentation is in turn determined using graph cuts which allows a globally optimal solution in the case of two regions. Consequently, there is no contour evolution based on differential increments as for example in level set methods. Experimental results on synthetic and real data show that good segmentations are obtained at speeds close to real-time.
  @cite_48: The purpose of this study is to investigate a variational method for joint segmentation and parametric estimation of image motion by basis function representation of motion and level set evolution. The functional contains three terms. One term is of classic regularization to bias the solution toward a segmentation with smooth boundaries. A second term biases the solution toward a segmentation with boundaries which coincide with motion discontinuities, following a description of motion discontinuities by a function of the image spatio-temporal variations. The third term refers to region information and measures conformity of the parametric representation of the motion of each region of segmentation to the image spatio-temporal variations. The components of motion in each region of segmentation are represented as functions in a space generated by a set of basis functions. The coefficients of the motion components considered combinations of the basis functions are the parameters of representation. The necessary conditions for a minimum of the functional, which are derived taking into consideration the dependence of the motion parameters on segmentation, lead to an algorithm which condenses to concurrent curve evolution, implemented via level sets, and estimation of the parameters by least squares within each region of segmentation. The algorithm and its implementation are verified on synthetic and real images using a basis of cosine transforms.
  @cite_29: We present a novel variational approach for segmenting the image plane into a set of regions of parametric motion on the basis of two consecutive frames from an image sequence. Our model is based on a conditional probability for the spatio-temporal image gradient, given a particular velocity model, and on a geometric prior on the estimated motion field favoring motion boundaries of minimal length. Exploiting the Bayesian framework, we derive a cost functional which depends on parametric motion models for each of a set of regions and on the boundary separating these regions. The resulting functional can be interpreted as an extension of the Mumford-Shah functional from intensity segmentation to motion segmentation. In contrast to most alternative approaches, the problems of segmentation and motion estimation are jointly solved by continuous minimization of a single functional. Minimizing this functional with respect to its dynamic variables results in an eigenvalue problem for the motion parameters and in a gradient descent evolution for the motion discontinuity set. We propose two different representations of this motion boundary: an explicit spline-based implementation which can be applied to the motion-based tracking of a single moving object, and an implicit multiphase level set implementation which allows for the segmentation of an arbitrary number of multiply connected moving objects. Numerical results both for simulated ground truth experiments and for real-world sequences demonstrate the capacity of our approach to segment objects based exclusively on their relative motion.
  @cite_55: In this paper we present a comprehensive energy-based framework for the estimation and the seg- mentation of the apparent motion in image sequences. The robust cost functions and the associated hierarchical minimization techniques that we propose mix efficiently non-parametric (dense) representations, local interacting parametric representations, and global non-interacting parametric representations related to a partition into regions. Experimental comparisons, both on synthetic and real images, demonstrate the merit of the approach on different types of photometric and kinematic contents ranging from moving rigid objects to moving fluids.
  @cite_36: This paper proposes a simple method for estimating dense and accurate optical flow field. It revitalizes an early idea of piecewise parametric flow model. A key innovation is that, we fit a flow field piecewise to a variety of parametric models, where the domain of each piece (i.e., each piece's shape, position and size) is determined adaptively, while at the same time maintaining a global inter-piece flow continuity constraint. We achieve this by a multi-model fitting scheme via energy minimization. Our energy takes into account both the piecewise constant model assumption and the flow field continuity constraint, enabling the proposed method to effectively handle both homogeneous motions and complex motions. The experiments on three public optical flow benchmarks (KITTI, MPI Sintel, and Middlebury) show the superiority of our method compared with the state of the art: it achieves top-tier performances on all the three benchmarks.
  @cite_21: Layered models are a powerful way of describing natural scenes containing smooth surfaces that may overlap and occlude each other. For image motion estimation, such models have a long history but have not achieved the wide use or accuracy of non-layered methods. We present a new probabilistic model of optical flow in layers that addresses many of the shortcomings of previous approaches. In particular, we define a probabilistic graphical model that explicitly captures: 1) occlusions and disocclusions; 2) depth ordering of the layers; 3) temporal consistency of the layer segmentation. Additionally the optical flow in each layer is modeled by a combination of a parametric model and a smooth deviation based on an MRF with a robust spatial prior; the resulting model allows roughness in layers. Finally, a key contribution is the formulation of the layers using an image-dependent hidden field prior based on recent models for static scene segmentation. The method achieves state-of-the-art results on the Middlebury benchmark and produces meaningful scene segmentations as well as detected occlusion regions.
  @cite_19: Motion analysis in computer vision is a well-studied problem with numerous applications. In particular, the tasks of optical flow estimation and tracking are of increasing interest. In this paper, we propose a level set approach to address both aspects of motion analysis. Our approach relies on the propagation of smooth interfaces to perform tracking while using an incremental estimation of the motion models. Implicit representations are used to represent moving objects, and capture their motion parameters. Information from different sources like a boundary attraction term, a background subtraction component and a visual consistency constraint are considered. The Euler-Lagrange equations within a gradient descent method lead to a flow that deforms a set of initial curve towards the object boundaries as well an incremental robust estimator of their apparent motion. Partial extension of the proposed framework to address dense motion estimation and the case of moving observer is also presented. Promising results demonstrate the performance of the method.
  @cite_23: Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos.
  @cite_15: Layered models provide a compelling approach for estimating image motion and segmenting moving scenes. Previous methods, however, have failed to capture the structure of complex scenes, provide precise object boundaries, effectively estimate the number of layers in a scene, or robustly determine the depth order of the layers. Furthermore, previous methods have focused on optical flow between pairs of frames rather than longer sequences. We show that image sequences with more frames are needed to resolve ambiguities in depth ordering at occlusion boundaries; temporal layer constancy makes this feasible. Our generative model of image sequences is rich but difficult to optimize with traditional gradient descent methods. We propose a novel discrete approximation of the continuous objective in terms of a sequence of depth-ordered MRFs and extend graph-cut optimization methods with new “moves” that make joint layer segmentation and motion estimation feasible. Our optimizer, which mixes discrete and continuous optimization, automatically determines the number of layers and reasons about their depth ordering. We demonstrate the value of layered models, our optimization strategy, and the use of more than two frames on both the Middlebury optical flow benchmark and the MIT layer segmentation benchmark.
  @cite_51: We describe a system for representing moving images with sets of overlapping layers. Each layer contains an intensity map that defines the additive values of each pixel, along with an alpha map that serves as a mask indicating the transparency. The layers are ordered in depth and they occlude each other in accord with the rules of compositing. Velocity maps define how the layers are to be warped over time. The layered representation is more flexible than standard image transforms and can capture many important properties of natural image sequences. We describe some methods for decomposing image sequences into layers using motion analysis, and we discuss how the representation may be used for image coding and other applications. >
  @cite_20: Abstract Dynamic analysis of image sequences is an important task in object-oriented video applications. It often relies on the segmentation of each image of the sequence into region entities of apparent homogeneous motion. In this paper, we present an original motion segmentation algorithm based on 2D polynomial motion models, a multiresolution robust estimator to compute these motion models, and appropriate local observations supplying both motion relevant information and their reliability. Motion segmentation is formulated as a contextual statistical labeling problem exploiting multiscale Markov random field (MRF) models. One of its main features is that it avoids time consuming alternate iterations between motion model estimation and spatial support identification. An original detection step allows us to estimate and to update the number of required motion models, and thus to handle the appearance of new objects. Numerous experiments performed with real indoor and outdoor image sequences demonstrate the efficiency of the method.
summary: The differences between existing methods concern mainly the solver for the image-partitioning problem. In a continuous setting, following the approach of Mumford and Shah @cite_37 , the problem has been addressed with an implicit level-set representation of the partitioning curve in @cite_29 @cite_19 @cite_48 . A primal-dual optimization strategy was used in @cite_8 . In a discrete setting, iterated conditional modes and high confidence first approaches were exploited in @cite_38 @cite_20 @cite_55 . Graph-cuts methods have also been used in @cite_41 , and more recently in @cite_36 . Layered models, introduced in @cite_51 , involve a similar optimization problem but add a depth information between the different regions, from which occlusions can be derived. This model has been revitalized in @cite_21 @cite_15 @cite_30 @cite_23 .