Hierarchical retargetting of 2D motion fields to the animation of 3D plant models

Hierarchical retargetting of 2D motion fields to the animation of 3D plant models
of 9
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  Eurographics/ ACM SIGGRAPH Symposium on Computer Animation (2006)M.-P. Cani, J. O’Brien (Editors) Hierarchical retargetting of 2D motion fields to the animationof 3D plant models Julien Diener 1 , Lionel Reveret 1 , Eugene Fiume 2 1 INRIA/GRAVIR, France 2 DGP, Dept. of Computer Science, Univ. of Toronto, Canada Abstract The complexity of animating trees, shrubs and foliage is an impediment to the efficient and realistic depiction of natural environments. This paper presents an algorithm to extract, from a single video sequence, motion fieldsof real shrubs under the influence of wind, and to transfer this motion to the animation of complex, synthetic 3D plant models. The extracted motion is retargeted without requiring physical simulation. First, feature tracking isapplied to the video footage, allowing the 2D position and velocity of automatically identified features to be clus-tered. A key contribution of the method is that the hierarchy obtained through statistical clustering can be used tosynthesize a 2D hierarchical geometric structure of branches that terminates according to the cut-off threshold of a classification algorithm. This step extracts both the shape and the motion of a hierarchy of features groups that are identified as geometrical branches. The 2D hierarchy is then extended to three dimensions using the estimated spatial distribution of the features within each group. Another key contribution is that this 3D hierarchical struc-ture can be efficiently used as a motion controller to animate any complex 3D model of similar but non-identical plants using a standard skinning algorithm. Thus, a single video source of a moving shrub becomes an input de-vice for a large class of virtual shrubs. We illustrate the results on two examples of shrubs and one outdoor tree. Extensions to other outdoor plants are discussed. 1. Introduction Therealisticdepictionofnaturalenvironmentshaslongbeena central problem in computer graphics. While the model-ing and rendering of plants and trees has yielded convinc-ing results in recent years, comparable progress in realisticplant motion under the influence of external forces such aswind remains problematic. Physical simulation is one pos-sibility, but its computational complexity and the lack of direct control makes it desirable to find an alternative thatmay more readily operate at interactive rates, and that canallow direct control over the motion if desired. This paperdemonstrates a method based on the observation of naturethat provides visually convincing results without requiringthe simulation of the exact motion of every branch and leaf.Currently, the standard method for real-time rendering andanimation of moving plants and trees is to animate a hierar-chy of billboards through the addition of  ad hoc oscillatoryor pseudo-random motion. We actually follow this approachstructurally, but we show that more realistic motion and evengeometrical structure can be automatically acquired fromvideo input using careful statistical analysis.The innovative idea of the paper is to illustrate that asparse motion control structure of a tree can be automati-cally extracted from video footage and can be mapped to acomplex 3D geometrical structure of a plant using a simpleskinning framework. Physical phenomena such wind forces,structural elasticity, or inter-collision of the branches are sta-tistically modeled from video, rather than explicitly simu-lated physically, which would be more expensive to com-pute, tune and control.After reviewing previous work, we present an algorithmto extract a hierarchy of animated branches from a statis-tical clustering of features tracked from a single view in avideo sequence. This step provides a set of results embed-ded in the video image plane. We employ a heuristic algo-rithm to project these structures into three dimensions, toachieve real-time rendering and animation of synthetic 3D c  The Eurographics Association 2006.   J. Diener & L. Reveret & E. Fiume / Hierarchical retargetting of 2D motion fields to the animation of 3D plant models replicas of the images found on the input 2D video. We thenshow how the results can be used to animate similar, butnon-identical, complex 3D models of shrubs. We finally dis-cussthelimitationsofourapproachandconcludewithfuturework. 2. Previous work From the pioneering work of Prusinkiewicz and Linden-mayer [PL90], to the most recent advances (e.g., [RFL ∗ 05,WWD ∗ 05]), the modeling and rendering of plants has along, rich literature. By contrast, the graphics literature onplant motion is much smaller, despite the fact that the prob-lem is challenging and clearly essential to the depiction of natural environments.Stam employs modal analysis to simulate the effect of wind as a load force on mechanical plant models. The windforce is numerically solved to account for turbulence effects[Sta97]. Perbet and Cani use a procedural wind model toanimate different geometrical levels of plant representation(grass wisps and billboards) [PC01]. Through video analy-sis, we instead investigate how the visible effect of wind onplants can be modeled without making a priori assumptionsabout the wind force.Beaudoin and Keyser quantify an accurate physical simu-lation of plants to provide a levels-of-detail system for real-time rendering of animated trees [BK04]. By using videoanalysis and statistical clustering, our approach provides analternative to physical simulation. Because our algorithm isinherently hierarchical, our approach also offers a way to au-tomatically generate an LOD structure.Sun etal. proposetheuseofVIDA,or“video-inputdrivenanimation”, a system to estimate the wind velocity fromvideos of the motion of trees or other oscillating systems[SJF03]. They invert the equations of motion of a mechan-ical model of a moving plant to infer a representative windfield that would account for that motion. It allows the intro-duction of additional effects onto the srcinal video, such assynthetic snow, leaves or dust, as well as new trees, all of which would be coherently controlled through interactionswith the estimated wind field. In our case, a video of a plantis analyzed to directly build a model and an animation of a3D synthesized plant without estimating explicitly the windforce.The goal of extracting 3D information from a singlemonocular view has been extensively studied in the Com-puter Vision literature. In the domain of "shape from mo-tion" approaches, factorization methods have proven thatrank constrains allow to extract shape and motion of a col-lection of 3D rigid bodies [CK98] or of a 3D deformablebody[TYAB01].Theseapproaches requiretoestimateinad-vance the rank constraint, which is related to the number of rigid groups in the case of multiple rigid bodies or the num-ber of linear degrees of freedom in the case of deformablebody. In the case of moving trees, motion of branches is notas rigid as motion of different part of an articulated object,making these rank methods not adapted to our case. The goalof our paper is not to recover the exact 3D shape and mo-tion of the srcinal plant, but rather to investigate how stan-dard Computer Vision techniques can provide efficient cuesto create motion controller for complex 3D model of plants.Modeling trees from photographs has been proposed by[RMD04], but their volumetric representation cannot beeasily extended to animation. Video-based animation hasbeen explored to control character animation from car-toons [BLCD02], and the gait of animal locomotion fromlive video documents [FRDC04]. Our work addresses plantsand presents a method to not only predict motion, but also toautomatically build a hierarchical 3D structure. 3. Building a hierarchy of branches from a single videoview In this section, we describe our experimental conditions, andhow the structure and motion of a hierarchy of branches canbe automatically estimated from feature positions tracked inthe video image plane. 3.1. Feature extraction Our method starts with a single view video of the wholeplant (in our case approximately 50 cm height). We use astandard color video camera at the typical scan rate and res-olution (interleaved video format). The first experiments wedescribe were carried out on indoor shrubs under controlledlighting. Aperiodic, non-smooth “wind” was generated bywaving a cardboard sheet near the plant. Using small shrubsand plants in a pot allowed an efficient background subtrac-tion so that outliers are easily rejected from the video analy-sis.Thestepsinthefeatureextractionprocessareasfollows.1.A plant is filmed in front of a uniform white or blue back-ground with a fixed camera.2.A short sequence of the background is filmed withoutthe plant so that the color of each pixel is modeled asa gaussian with known mean and standard deviation tocharacterize the background.3.In the plant video, a pixel is removed (i.e., replaced withblack) if its color is within 95% of the gaussian distribu-tion of the background model.More complex methods exist for background subtraction,but this simple approach has proven to be sufficient for ourexperiments to validate our appraoch.Feature tracking is then applied on the video sequenceof the plant. Feature location is initiated with the methodfrom [ST94]. We use the pyramidal implementation of theLucas-Kanade algorithm [LK81] by [Bou00] for featuretracking. The algorithm has been parameterized so that 200 c  The Eurographics Association 2006.   J. Diener & L. Reveret & E. Fiume / Hierarchical retargetting of 2D motion fields to the animation of 3D plant models features are correctly tracked over the whole sequence, ini-tialized with an average distance of 15 pixels between neigh-boring features.At the end of this process, 200 sequences of 2D featuretrajectories are collected over the entire video sequence of the plant. On the two test sequences, 125 and 250 frameswere analyzed. 3.2. Feature clustering Our hypothesis is that leaf motion tends to be grouped bythe underlying branch structure of the plant, despite their ap-parently random individual motions. This hypothesis is intu-itively illustrated on the video accompanying the paper, inwhich only extracted features are displayed as green dotsover a black background. Instead of a collection of dots withcompletely random motion, the overall structure of the plantplainly emerges. We investigated this hypothesis quantita-tively using clustering analysis.A first approach would be to cluster feature positions foreach frame using an euclidean distance over each feature’simage co-ordinates. Our experiments showed that this ap-proach leads to incorrect feature clustering in the sense thatthe leaves from different branches can be grouped together,since the orientation of branches may place the leaves ondifferent branches in close proximity. Figure 1 compares aground truth clustering made manually and the results of anautomatic clustering based on feature positions only. Figure 1: (a) Ground truth obtained from a manual clus-tering, (b) Automatic clustering based on feature positionsonly. To solve this problem, we consider a composite distancefor the clustering, which integrates both the position and thevelocity of the features. The velocity is computed in 2Dimage co-ordinates as the difference between positions attwo consecutive frames and thus corresponds to the veloc-ity field of the optical flow algorithm used for feature track-ing [Bou00]. Each velocity vector is normalized to have unitlength. We use the product of the euclidean distance betweenpositions and the angular distance between velocities as thenew distance between two features. The distances on posi-tion and velocities are normalized by their respective stan-dard deviation observed over all the features along the wholesequence.As one might expect, our experiments showed that a perframeapproachintroducesinstabilities,becauseincoherencebetween class composition occurs from frame to frame. Toreduce this effect, a criterion is computed for each frame toevaluate its relevance to coherent motion. From a statisti-cal perspective, we need to reject outlier frames that reducethe clustering quality. Such outliers arise due to two edgeconditions as illustrated in Figure 2: (a) when there is onlylow-level “ambient” wind; and (b) when there is a consider-able impulsive wind force. In both cases, there is diminishedcorrelation between leaf motion and branch motion that ismanifested in amplified tracking noise in the first, and overlyhomogeneous motion in the second. To characterize thesetwo cases statistically, we first compute the average of dis-tances between allpairs of points foreach frame. Second,wecompute the mean and standard deviation of this per framevalue over the whole sequence. Frames with a value aboveor below two times the standard deviation with respect tothe mean value are considered as outliers. On the two testsequences, containing respectively 125 and 250 frames, 16and 22 are rejected. Finally, the metric for feature cluster-ing is the average of composite distances between pairs of features computed over the selected inlier frames. Figure 2: (a) Unstructured motion, (b) Strong wind inducinga single overall motion. Finally, we obtain an automatic classification of featuresinto groups with a solution closer to the manual clustering asillustrated in Figure 3. Figure 3: (a) Ground truth manual clustering, (b) Automaticclustering based on distance combining feature position and velocity. c  The Eurographics Association 2006.   J. Diener & L. Reveret & E. Fiume / Hierarchical retargetting of 2D motion fields to the animation of 3D plant models 3.3. Automatic selection of a hierarchy of branches We now discuss details of the clustering algorithm and howit automatically proposes a hierarchy of branches. Most clas-sification approaches fall into three main categories [Cor71,Gor87,HTF01]: • Partition , where all individuals are clustered around rep-resentative data points (typically using k  -means methods). • Division , where an explicit ordering criterion iterativelysplits individuals into hierarchical classes (leading to ahierarchical description of the data set–typically kd  -treemethods). • Aggregation , where individuals are iteratively comparedand combined by closest pairs (leading also to a hierar-chical representation).Because we want to find a hierarchy, and because we haveno inherent, explicit ordering criterion, we adopted the thirdchoice with an aggregative method. At a first level, thesemethods require measuring the distance between two fea-tures and groups the closest pair. We use the distance de-scribed in the previous section.The next step of the algorithm is: given three elements x ,  y , z , where x and y are already grouped into an aggregategroup H  , z is compared to the group H  by evaluating a dis-tance d  (  H  ,  z ) . Several choices are possible for this distance.The most common is the average distance: d  (  H  ,  z ) = d  (  x ,  z )+ d  (  y ,  z ) 2(1)where d  (  x ,  z ) is the features distance described in the previ-ous section.By extension, if  X  and Y  are subgroups gathered into agroup  H  ,andnotjustindividualelements,theequivalentdis-tance of  z to the group H  is given by d  (  H  ,  z ) = n  X  d  (  X  ,  z )+ n Y  d  ( Y  ,  z ) n  X  + n Y  (2)where n  X  and n Y  are the number of leaf elements respec-tively below the subgroups X  and Y  .Finally, when all individuals are already clustered into agroup, further aggregation requires the evaluation of the dis-tance between groups. The following formula is thus used: d  (  X  , Y  ) = 1 n  X  1 n Y  ∑  x i ∈  X  ∑  y  j ∈ Y  d  (  x i ,  y  j ) (3)where X  and Y  are groups containing respectively n  X  and n Y  individual elements.Elements and subgroups are iteratively aggregated intogroups by choosing the smallest distance. This minimumdistance criterion provides a basic approach for aggrega-tive clustering. However, it turned from our experiments thatthis approach lacked of robustness and provided unbalancedgroups. To overcome this limitation, we have used a refinedapproach which considers the concept of inertia, known asthe Ward criterion. This criterion uses the inertia of the sys-tem:  I  = I  inter  +  I  intra = ∑ q n q d  (  H  q , G )+ ∑ q ∑  x i ∈  H  q d  (  H  q ,  x i ) (4)where G is the set of all points, H  q is the subset of points ina subgroup, n q is the number of individual elements of  H  q .The value of  I  , the total inertia of the system, is constantas it is independent of the clustering. At the initial stage allgroups g q contain only one feature, and thus the intra groupinertia, I  intra , is null. Then, at each stage of the clusteringprocess, the inertia of the system will be transferred from the inter  group inertia, I  inter  , to the intra group inertia by an in-crement ∆ i depending on which elements will be aggregatedtogether. The principle of the Ward criterion is to aggregate,at each stage, the pair of elements which will minimize theincrement ∆ i . This criterion has proven to be more robustfor our problem than the minimal distance criterion, at theexpense of an increase in calculation time.These methods induce a hierarchical representationknown as dendrogram . Given a required number of classes,the final clustering is obtained by adjusting a cut-off line onthe dendrogram representation as illustrated in Figure 4. Figure 4: Hierarchical clustering and cut-off line for the de-termination of the number of classes identified as terminalgroups. From Figure 4, once a number of classes is set, all theleaf nodes below the cut-off are grouped as terminal groups .Groups above the cut-off line automatically provides a hi-erarchical structure of the shrub with intermediate groups ,“up” to the base trunk. A branch is identified as joining twogroups. By changing the location of the cut-off line, a differ-ent choice of branches is coherently selected. This approachautomatically produces an LOD mechanism based on thestatistical distribution of data only. 4. Creating 3D shape and motion The algorithm of the previous section yields a topologicalhierarchy of branches that is valid over the entire video se- c  The Eurographics Association 2006.   J. Diener & L. Reveret & E. Fiume / Hierarchical retargetting of 2D motion fields to the animation of 3D plant models quence. This section presents a method to automatically de-duce3Dshapeandmotionfromthisstructure.Theshapeandmotion of the branches (terminal and intermediate groups)are first defined in the 2D video image plane and then con-verted to 3D. 4.1. Creating motion of the terminal groups For each frame of the video sequence, we consider the dis-tribution of the features within a terminal group to deducethe 2D parameters of an ellipse. This ellipse corresponds tothe 95% isocurve of a gaussian distribution defined throughthe covariance matrix of the data set. A third axis is createdperpendicular to the image view plane, with a length equalto the shortest axis of the 2D ellipse. This generates 3D el-lipsoids for each frame and each terminal group. The centerof each ellipsoid provides the position of the group in theimage plane co-ordinate system.This approach is inspired by methods for inflating 3Dshape from 2D structure, typically illustrated by the “Teddy”system [IMT99], where 2D contours drawn by hand are au-tomatically converted into 3D volumes. In our case, we donot create complete continuous polygonal surfaces, but in-stead we project depth values away from the image plane tothe tree nodes. 4.2. Propagating motion to intermediate groups The terminal groups are now geometrically positioned ateach frame. For each frame, the position of an intermedi-ate group is firstly computed as the average of its two childgroups (each group has exactly two children, recalling Fig-ure 4). This initialization process starts with the terminalgroups as their location is already known from the previoussection. Spatial locations are propagated down to the rootnode, providing the position of the intermediate group in theimage-plane co-ordinate system. An additional node is in-troduced at the base of the trunk for visualization only.This first step results in unrealistic T-junctions forbranches. To create more realistic tree shapes, intermediategroups initially at T-junctions are shifted toward their par-ent group. It is realistic to expect that branch lengths donot change over the duration of the animation. We thereforeestimate this shifting coefficient  α by minimizing the stan-dard deviation over all sequences of the distance betweenthe node P i to update and its two child nodes P n and P m (seeFigure 5): α i = argmin α [ σ (  P i ( α ) − P n  )+ σ (  P i ( α ) − P m  )] (5)To keep a coherent shape, this optimization process isstarted from the root node and propagated toward the ter-minal groups.The position of every group (intermediate and terminal)is then transformed from the global co-ordinate system of  Figure 5: Optimizing intermediate group locations. the image plane to a set of local co-ordinate systems alongthe branch using the topology of the hierarchy. Group posi-tions are converted into local polar co-ordinates. The lengthof each branch is fixed at its mean to keep fixed-lengthbranches. The motion of each branch is expressed as a rota-tional animation curve in the 2D image plane. The clusteringand 2D geometrical hierarchy is illustrated in Figure 6. Figure 6: Final geometrical hierarchy of groups. 4.3. Extending groups to 3D shape and motion So far, the shape and motion of the branch structure is em-bedded in the 2D image plane (only leaves are shaped in3D). We propose two methods to finally extend them to 3D:one method relies on user interaction, the other one is fullyautomatic.The first method keeps the geometrical hierarchy obtainedfrom the previous section as is and is presented to the user tomanually edit the tree structure. To make this process intu-itive in 3D, for each branch node, axes are oriented so that: • the branch joining the parent to child group correspondsto the x -axis, • the z -axis is perpendicular to the image plane - this planeis assumed to carry the main motion of the branch, c  The Eurographics Association 2006.
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!