How To, Education & Training

A variational wave acquisition stereo system for the 3-D reconstruction of oceanic sea states

A variational wave acquisition stereo system for the 3-D reconstruction of oceanic sea states
of 10
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  A VARIATIONAL WAVE ACQUISITION STEREO SYSTEM FOR THE 3-DRECONSTRUCTION OF OCEANIC SEA STATES Guillermo Gallego School of Electrical and Computer EngineeringGeorgia Institute of Technology, Atlanta, USA. Anthony Yezzi School of Electrical and Computer EngineeringGeorgia Institute of Technology, Atlanta, USA. Francesco Fedele School of Civil and Environmental EngineeringGeorgia Institute of Technology, Savannah, USA. Alvise Benetazzo CNR-ISMAR, Venice, Italy. ABSTRACT We propose a novel remote sensing technique that infers thethree-dimensional wave form and radiance of oceanic sea statesvia a variational stereo imagery formulation. In this setting, theshape and radiance of the wave surface are minimizers of a com- posite cost functional which combines a data fidelity term and smoothness priors on the unknowns. The solution of a systemof coupled partial differential equations derived from the cost  functional yields the desired ocean surface shape and radiance.The proposed method is naturally extended to study the spatio-temporal dynamics of ocean waves, and applied to three setsof video data. Statistical and spectral analysis are carried out.The results shows evidence of the fact that the omni-directionalwavenumber spectrum S  ( k  )  of the reconstructed waves decays ask  − 2 . 5 in agreement with Zakharov’s theory (1999). Further, thethree-dimensional spectrum of the reconstructed wave surface isexploited to estimate wave dispersion and currents. INTRODUCTION Wind-generated waves play a prominent role at the inter-faces of the ocean with the atmosphere, land and solid Earth.Wavesalsodefineinmanywaystheappearanceoftheoceanseenby remote-sensing instruments. Classical observational methodsrely on time series retrieved from wave gauges and ultrasonic in-struments or buoys to measure the space-time dynamics of oceanwaves. Global altimeters, or Synthetic Aperture Radar (SAR)instruments are exploited for observations of large oceanic ar-eas via satellites, but details on small-scales are lost. Herein,we propose to complement the abovementioned instruments witha novel video observational system which relies on variationalstereo techniques to reconstruct the 3-D wave surface both inspace and time. Such system uses two or more stereo cameraviews pointing at the ocean to provide spatio-temporal data andstatistical content richer than that of previous monitoring meth-ods. Vision systems are non-intrusive and have economical ad-vantages over their predecessors, but they require more process-ing power to extract information from the ocean.Since this work covers both the topics of shape reconstruc-tion and oceanic sea states, it relates to a vast body of litera-ture. The three-dimensional reconstruction of an object’s sur-face from stereo pairs of images is a classical problem in com-puter vision (see, for example [1–4]), and it is still an extremelyactive research area. There are many 3-D reconstruction algo-rithms available in the literature and the reconstruction problemis far from being solved. The different algorithms are designedunder different assumptions and provide a variety of trade-offsbetween speed, accuracy and viability. Traditional  image-based  stereo methods typically consist of two steps: first image pointsor regions are detected and matched across images by optimiz-ing a photometric score to establish local correspondences; thendepth is inferred by combining these correspondences using  tri-angulation  of 3-D points ( back-projection  of image points). Thefirst step, also known as the stereo matching problem, is signif-icantly more difficult than the second one. However, epipolargeometry between image pairs can be exploited to reduce stereo1  matching to a 1-D search along epipolar lines. This is the strat-egy used in recent systems [5,6]. This approach has the advan-tages of being simple and fast. However, it also has some majordisadvantages that motivated the research on improved stereo re-construction methods [7–9]. These disadvantages are: ( i ) Corre-spondences rely on strong textures (high contrast between inten-sities of neighboring points) and image matching gives poor cor-respondences if the objects in the scene have a smooth radiance.Correspondences also suffer from the presence of noise and localminima. ( ii ) Each space point is reconstructed independently andtherefore the recovered surface of an object is obtained as a col-lection of scattered 3-D points. Thus, the hypothesis of the conti-nuity of the surface is not exploited in the reconstruction process.The breakdown of traditional stereo methods in these situationsis evidenced by “holes” in the reconstructed surface, which cor-respond to unmatched image regions [1,5]. This phenomenonmay be dominant in the case of the ocean surface, which, by na-ture, is generally continuous and contains little texture.Modern  object-based   image processing and computer visionmethods that rely on Calculus of Variations and Partial Differ-ential Equations (PDEs), such as Stereoscopic Segmentation [8]and other variational stereo methods [7,9,10], are able to over-come the disadvantages of traditional stereo. For instance, un-matched regions are avoided by building an explicit model of the smooth surface to be estimated rather than representing itas a collection of scattered 3-D points. Thus, variational meth-ods provide dense and coherent surface reconstructions. Sur-face points are reconstructed by exploiting the continuity (co-herence) hypothesis in the full two-dimensional domain of thesurface. Variational stereo methods combine correspondence es-tablishment and shape reconstruction into one single step andthey are less sensitive to matching problems of local correspon-dences. The reconstructed surface is obtained by minimizationof an energy functional designed for the stereo problem. Thesolution is obtained in the context of active surfaces by deform-ing an initial surface via a gradient descent PDE derived fromthe necessary optimality conditions of the energy functional, theso-called Euler-Lagrange (EL) equations.In the context of oceanography, the first experiments withstereo cameras mounted on a ship were by Schumacher [11] in1939. Later, Cot´e et al. [12] in 1960 demonstrated the use of stereo-photographytomeasuretheseatopographyforlongoceanwaves. The study of long waves using stereophotography wasalso discussed by Sugimori [13], based on an optical methodby Barber [14], and by Holthuijsen [15]. Stereography gainedpopularity in studying the dynamics of oceanographic phenom-ena during the 1980s due to advances in hardware. Shemdin etal. [16,17] applied stereography for the directional measurementof short ocean waves. In 1997, Holland et al. [18] demonstratedthe practical use of video systems to measure nearshore physicalprocesses. A more recent integration of stereographic techniquesinto the field of oceanography has been the WAVESCAN project FIGURE 1 . Left: off-shore platform “Acqua Alta” in the NorthernAdriatic Sea, near Venice. Center: pair of synchronized cameras formonitoringtheoceanclimatefromtheplatform. Right: WASShardwareinstalled at the platform for recording stereo videos of ocean waves. of Santel et al. [19].Recently, Benetazzo [5] successfully incorporated epipolartechniques in the Wave Acquisition Stereo System (WASS). Thiswas tested in experiments off the shore of the California Coastand the Venice coast in Italy. Benetazzo was able to estimatewave spectra from the extracted time series of the surface fluc-tuations at one fixed point given the data images. The accu-racy of such spectral estimates is comparable to the accuracy ob-tained from ultrasonic transducer measurements. An example of a WASS system currently installed at the Acqua Alta platform isshown in Fig. 1. An alternative trinocular imaging system (AT-SIS) for measuring the temporal evolution of 3-D surface waveswas proposed in [6]. More recently, in [20] it is shown howa modern variational stereo reconstruction technique pioneeredby [7] can be applied to the estimation of oceanic sea states.Additional references demonstrate that this is an active researchtopic [21–24].Encouraged by the results in [5,20,25], in this paper we pro-pose a novel variational framework for the recovery of the shapeof ocean waves given multi-view stereo imagery. In particular,motivated by the characteristics of the target object in the scene,i.e., the ocean surface, we first introduce the graph surface repre-sentation in the formulation of the reconstruction problem. Then,we present the newvariational stereo method in thecontext of ac-tive surfaces. The performance of the algorithm is validated onexperimental data collected off shore, and the statistics of the re-constructed surface are also analyzed. Concluding remarks andfuture research directions are finally presented. THEVARIATIONALGEOMETRICMETHOD This paper is inspired by the works of [5, 20] and [8]. Inparticular, the variational approach of   Stereoscopic Segmenta-tion  [8] is used to tackle the vision problem: the reconstructedsurface of the ocean is obtained as the minimizer of an energyfunctional designed to fit the measurements of ocean waves. In2  every 3-D reconstruction method, the quality and accuracy of the results depend on the calibration of the cameras. There arestandard camera calibration procedures in the literature to char-acterize accurately the intrinsic and extrinsic parameters of thecameras [1]. We assume cameras are calibrated and synchro-nized, and we focus on the reconstruction of the water surfacefor a fixed time. Multi-image setup. Graph surface representation Let  S   be a smooth surface in  R 3 with generic local coordi-nates  ( u , v ) ∈ R 2 .  Let {  I  i }  N  c i = 1  be a set of images of a static (wa-ter) scene acquired by cameras whose calibration parameters are { P i }  N  c i = 1 . Space points are mapped into image points accordingto the pinhole camera model [2]. The equations of such a per-spective projection mapping are linear if expressed in homoge-neous coordinates of Projective geometry. A surface point (or,in general a 3D point)  X  = (  X  , Y  ,  Z  )  with homogeneous coor-dinates ¯ X  = (  X  , Y  ,  Z  , 1 )  is mapped to point  x i  = (  x i ,  y i )  in the i -th image with homogeneous coordinates ¯ x i  = (  x i ,  y i , 1 )  ∼ P i  ¯ X ,where the symbol ∼ means equality up to a nonzero scale factorand  P i = K i [ R i | t i ]  is the 3 × 4 projection matrix with the intrinsic( K i ) and extrinsic ( R i , t i ) calibration parameters of the  i -th cam-era. These parameters are known under the hypothesis of cal-ibrated cameras. The optical center of the camera is the point C i  = ( C  1 i  , C  2 i  , C  3 i  )  that satisfies  P i  ¯ C i  =  0 . Let  π  i  : R 3 → R 2 notethe projection maps:  x i  = π  i ( X ) . Finally,  I  i ( x i ) ≡  I  i ( π  i ( X ))  is theintensity at  x i .We present a different approach to the reconstruction prob-lem discussed in [7,8] by exploiting the hypothesis that the sur-face of the water can be represented in the form of a graph orelevation map:  Z   =  Z  (  X  , Y  ) ,  (1)where  Z   is the height of the surface with respect to a domainplane that is parameterized by coordinates  X   and  Y  . Indeed,slow varying, non-breaking waves admit this simple represen-tation with respect to a plane orthogonal to gravity direction. Asa natural extension of existing variational stereo methods, energyfunctionals can be tailored to exploit the benefits of this valuablerepresentation. The surface can still be obtained as the minimizerof a suitable energy functional but now with a different geomet-rical representation of the solution.The graph representation of the water surface presents someclear advantages over the more general level set representationof [7–9,20]. Surface evolution is simpler to implement since thesurface is not represented in terms of an auxiliary higher dimen-sional function (the level set function). The surface is evolveddirectly via the height function (1) discretized over a fixed 2-Dgrid defined on the  X  − Y   plane. The latter also implies that forthe same amount of physical memory, higher spatial resolution(finer details) can be achieved in the graph representation thanwith the level set. The  X  − Y   plane becomes the natural com-mon domain to parameterize the geometrical and photometricproperties of surfaces. This simple identification does not ex-ist in the level set approach [8]. Finally, the graph representa-tion allows for fast numerical solvers besides gradient descent,like Fast Poisson Solvers, Cyclic Reduction, Multigrid Methods,Finite-Element Methods (FEM), etc. In the level set framework,the range of solvers is not as diverse.However, there are also some minor disadvantages. A worldframe properly oriented with the gravity direction must be de-fined in advance to represent the surface as a graph with respectto this plane. This is not trivial  a priori  and might pose a problemif only the information from the stereo images is used [5]. Thiscondition may not be so if external gravity sensors provide thisinformation. Surface evolution is constrained to be in the form of a graph and this may not be the same as the evolution describedfor an unconstrained surface. As a result, more iterations may berequired to reach convergence.The reconstruction problem is mathematically stated in thefollowing section. The desired surface is given by the solution of a variational optimization problem. Proposed energy functional Consider the 3-D reconstruction problem from a collectionof   N  c  ≥ 2 input images (we will exemplify with  N  c  =  2). Weinvestigate a generative model of the images that allows for the joint estimation of the shape of the surface  S   and the radiancefunction on the surface  f   as minimizers of an energy functional.Let the energy functional be the weighted sum of a data fidelityterm  E  data  and two regularizing terms: a geometry smoothingterm  E  geom  and a radiance smoothing term  E  rad ,  E  ( S  ,  f  ) =  E  data ( S  ,  f  )+ α   E  geom ( S  )+ β   E  rad (  f  ) ,  (2)where  α  , β   ∈ R + . The data fidelity term measures the photo-consistency of the model: the discrepancy in the  L 2 sense be-tween the observed images  I  i  and the radiance model  f  ,  E  data  =  N  c ∑ i = 1  E  i ,  E  i  =   Ω i φ  i d x i ,  (3)where the photometric matching criterion is φ  i  =  12   I  i ( x i ) −  f  ( x i )  2 .  (4)The region of the image domain where the scene is projectedis denoted by  Ω i . The meaning of   f  ( x i )  will be clear shortly.3  Assuming that the surface of the scene is represented as a graph  Z   =  Z  ( u , v ) , a point on the surface has coordinates X ( u , v ) =  u , v ,  Z  ( u , v )   .  (5)The chain of operations to obtain the intensity  I  i ( x i )  given a sur-face point with world coordinates  X ( u )  ≡  S  ( u ) ,  u = ( u , v )  , is X ( u )  →  ˜ X i  =  M i X + p i 4  → x i  →  I  i ( x i ) ,  (6)where ˜ X i = (  ˜  X  i ,  ˜ Y  i ,  ˜  Z  i )  are related to the coordinates of   X  inthe  i -th camera frame,  x i  = (  x i ,  y i )  = (  ˜  X  i / ˜  Z  i ,  ˜ Y  i / ˜  Z  i )  are thecoordinates of the projection of   X  in the  i -th image plane and P i = [ M i | p i 4 ]  is the projection matrix of the camera correspond-ing to the  i -th image, in world coordinates, i.e.,  M i =  K i R i ≡ ( n i 1 , n i 2 , n i 3 )  and  p i 4  =  K i t i . Also,  | M i |  =  det ( M i ) .The radiance model  f   is specified by a function ˆ  f   definedon the surface  S  . Then,  f   in (4) is naturally defined by  f  ( x i ) = ˆ  f  ( π  − 1 i  ( X )) , where  π  − 1 i  denotes the back-projection operationfrom a point in the  i -th image to the closest surface point withrespect to the camera. With a slight abuse of notation, let us use  f   to denote the parameterized radiance  f  ( u ) , understanding that  f  ( x i )  in (4) reads the back-projected value in ˆ  f  ( X ( u )) =  f  ( u ) .Motivated by the common parameterizing domain of theshape and radiance of the surface and to obtain the simplest dif-fusive terms in the PDEs derived from the necessary optimalityconditions of the energy (2), let the regularizers be  E  geom  =   U  12  ∇  Z  ( u )  2 d u ,  (7)  E  rad  =   U  12  ∇  f  ( u )  2 d u ,  (8)where  ∇  Z   = (  Z  u ,  Z  v )  ,  ∇  f   = (  f  u ,  f  v )  and subscripts indicatethe derivative with respect to that variable. Once all terms in (2)have been specified, some transformations are carried out to ex-press the data fidelity integrals over a more suitable domain: theparameter space. The Jacobian of the change of variables be-tween integration domains is, by applying the chain rule to (6), J i  =  d x i d u  =  −| M i | ˜  Z  − 3 i  ( X − C i ) · ( X u × X v ) ,  (9)where  X u  × X v  is proportional to the outward unit normal  N  tothe surface at  X ( u , v ) , and ˜  Z  i  =  n i 3  · ( X − C i )  >  0 is the depth of the point  X  with respect to the  i -th camera (located at  C i ). Withthis change, energy (3) becomes  E  i  =   Ω i φ  i  d x i  =   U  φ  i J i  d u ,  (10)where the last integral is over  U  : the part of the parameter spacewhose surface projects on  Ω i  in the  i -th image. Observe thatthe Jacobian weights the photometric error  φ  i  proportionally tothe cosine of the angle between the unit normal to the surfaceat  X  and the  projection ray  (the ray joining the optical centerof the camera and  X ):  ( X − C i ) · ( X u  × X v ) . After collectingterms (7), (8), and (10), and noting that the shape X of the surfacesolely depends on the height (Eqn. (5)), energy (2) becomes theintegral of the so-called  Lagrangian L :  E  (  Z  ,  f  ) =   U   L (  Z  ,  Z  u ,  Z  v ,  f  ,  f  u ,  f  v , u , v ) d u .  (11) Energy minimization. Optimality condition The energy (11) depends on two functions: the shape  Z   andthe radiance  f   of the surface. To find a minimizer of such a func-tional, we derive the necessary optimality condition by settingto zero the first variation of the functional. Using standard tech-niques from Calculus of Variations, the first variation (Gˆateauxderivative) of (11) has two terms: one in the interior of the inte-gration region  U   in the parameter space and one boundary term(on  ∂  U  ). Setting the first variation to zero for all possible smoothperturbations yields a coupled system of PDEs (EL equations)along with natural boundary conditions: g (  Z  ,  f  ) − α  ∆  Z   =  0 in U  ,  (12) b (  Z  ,  f  )+ α ∂   Z  ∂ν   =  0 on  ∂  U  ,  (13) − ∑  N  c i = 1 (  I  i −  f  ) J i (  Z  ) − β  ∆  f   =  0 in U  ,  (14) β  ∂   f  ∂ν   =  0 on  ∂  U  ,  (15)where the non-linear terms due to the data fidelity energy are g (  Z  ,  f  ) = ∇  f   · ∑  N  c i = 1 | M i | ˜  Z  − 3 i  (  I  i −  f  )( u − C  1 i  , v − C  2 i  ) ,  (16) b (  Z  ,  f  ) = ∑  N  c i = 1 φ  i | M i | ˜  Z  − 3 i  ( u − C  1 i  ) ν  u +( v − C  2 i  ) ν  v  . The Laplacians ∆  Z   and ∆  f   arise from the regularizing terms (7)and (8), respectively, and  ∂   ∗ / ∂ν   is the usual notation for thedirectional derivative along  ν   = ( ν  u , ν  v )  , the normal to the in-tegration domain U   in the parameter space.A simple classification of the PDEs can be done as follows.For a fixed surface, (14) and (15) form a linear elliptic PDE (of the inhomogeneous Helmholtz type) with Neumann boundaryconditions. On the other hand, for a fixed radiance, (12) and (13)lead to a nonlinear elliptic equation in the height  Z   with nonstan-dard boundary conditions.A common approach to solve difficult EL equations, suchas the EL equation presented in (12)-(15), is to add an artificial4  time marching variable  t   dependency in the unknown functions(height, radiance) and set up a gradient descent flow that willdrive their evolution such that the energy (11) will decrease intime. Thus the solution of the EL equations is obtained as thesteady-state of the gradient descent equations. This is the contextof the so-called active surfaces. The gradient descent PDEs are:  Z  t   =  α  ∆  Z  − g (  Z  ,  f  ) ,  (17)  f  t   =  β  ∆  f  − ∑  N  c i = 1 J i (  Z  )  f   + ∑  N  c i = 1  I  i J i (  Z  ) .  (18)To simplify the equations, we approximate the boundary condi-tion (13) by a simpler, homogeneous Neumann boundary condi-tion. This can be interpreted as if the data fidelity term vanishedclose to the boundary and it is a reasonable assumption since themajor contribution to the energy is given by the terms in  U  , notat the boundary. Numerical solution An iterative, alternating approach is used to find the mini-mum of energy (2) via the evolution of the coupled gradient de-scentPDEs(17)-(18). Duringeachiterationtherearetwophases:( i ) evolve the shape, leaving the radiance fixed, and ( ii ) evolvethe radiance, leaving the shape unchanged. The PDEs are dis-cretized on a rectangular 2-D grid in the parameter space andthen solved numerically using finite-difference methods (FDM).Forward differences in time and central differences in space ap-proximate the derivatives, yielding an  explicit updating scheme .The time step  ∆ t   in the scheme is determined by the stabilitycondition of the resulting PDE. For the linear PDE (18), the timestep for   2 stability satisfies ∆ t  ≤ 1 /  4 β  h 2  +  12  max ∑  N  c k  = 1 J k   , where h  is the spatial step size of the grid,  J k  (  Z  ) ≥ 0 and the maximumis taken over the 2-D discretized Jacobians for the current heightfunction. The time step may change at every iteration, dependingon the value of the evolving height. For the nonlinear PDE (12),thevonNeumannstability analysisofthelinearizedPDEyieldsatime step  ∆ t  ≤ 1 /  4 α  h 2  +  12  max | ˙ g (  Z  ) |  , where ˙ g (  Z  )  is the deriva-tive of (16) and the maximum is taken over the 2-D discretizedgrid at the current time.The previous time-stepping methods are used as relaxationprocedures inside a multigrid method [26] that approximatelysolvestheELequations. Multigridmethodsarethemostefficientnumerical tools for solving elliptic boundary value problems. EXPERIMENTS Experiment 1.  Images of ”Canale della Giudecca” in Venice(Italy).  After validatingthenumericalimplementation of thepro-posed variational stereo method with synthetic data, some exper-iments with real data are carried out. Figs. 2, 3 and 4 showan example of a reconstructed water surface from images of the FIGURE 2 . Experiment I (Venice). Left: projection of the boundaryof the estimated graph, which has been discretized on a grid of 129 × 513 points. Right: modeled image (computed from surface height andradiance) superimposed on srcinal image. X [m]    Y   [  m   ]   02460510152025−0.15−0.1−0.0500. FIGURE 3 . Experiment I (Venice). Left: estimated height function  Z  ( u , v )  (shape of the water surface) in pseudo-color. Center: heightfunction represented by grayscale intensities, from dark (low) to bright(high). Right: estimated radiance function  f  ( u , v ) , i.e., texture on thesurface. Venice Canal. Cropped images in Fig. 2 are of size 600 × 450pixels and show the region of interest to be reconstructed. Fig. 2also displays one of the modeled images created by the genera-tive model within our variational method. The data fidelity termcompares the intensities of the srcinal and modeled images inthe highlighted region, in all images. As observed, the modeledimage is a good match of the original image. Fig. 3 showsthe converged values of the unknowns of the problem (the heightand the radiance of the surface), while Fig. 4 shows the 3-D rep-resentation of the reconstructed surface obtained by combiningboth 2-D functions from Fig. 3. In this experiment, the valuesof the weights of the regularizers were empirically determined: α   =  0 . 035 and  β   =  0 . 01. At the finest of the 5-level multi-grid [26] algorithm, the gradient descent PDEs are discretizedon a 2-D grid with 129 × 513 points. The distance between adja-cent grid points is  h = 5 cm. Therefore, the grid covers an area of 6 . 45 × 25 . 65 m 2 . An example of a surface discretized at the finestgrid level is shown in Fig. 4. Observe the high density of the sur-5
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks