International Journal of Computer Vision 46(3), 201–221, 2002c
2002 Kluwer Academic Publishers. Manufactured in The Netherlands.
Subjective Surfaces: A Geometric Model for Boundary Completion
A. SARTI
DEIS, University of Bologna, Italy
asarti@deis.unibo.it
R. MALLADI AND J.A. SETHIAN
Department of Mathematics, Lawrence Berkeley National Laboratory, University of California, Berkeley, CA 94720, USA
malladi@euphrates.lbl.govsethian@math.berkeley.edu
Received June 14, 2000; Revised August 22, 2001; Accepted August 22, 2001
Abstract.
We present a geometric model and a computational method for segmentation of images with missingboundaries. In many situations, the human visual system ﬁlls in missing gaps in edges and boundaries, building andcompleting information that is not present. Boundary completion presents a considerable challenge in computervision, since most algorithms attempt to exploit existing data. A large body of work concerns completion models,which postulate how to construct missing data; these models are often trained and speciﬁc to particular images. Inthis paper, we take the following, alternative perspective: we consider a given reference point within the image,and then develop an algorithm which tries to build missing information on the basis of the given point of viewand the available information as boundary data to the algorithm. Starting from this point of view, a surface isconstructed. It is then evolved with the mean curvature ﬂow in the metric induced by the image until a piecewiseconstant solution is reached. We test the computational model on modal completion, amodal completion, andtexture segmentation. We extend the geometric model and the algorithm to 3D in order to extract shapes from lowsignal/noise ratio ultrasound image volumes. Results in 3D echocardiography and 3D fetal echography are alsopresented.
Keywords:
subjective surfaces, segmentation, perceptual contours, level sets, differential geometry, Riemanniangeometry, surface evolution
1. Introduction
Consider the images in Fig. 1. The human visual system completes the internal objects exploiting the existingsparsedata.Intheimageontheleft,asolidtrianglein the center of the ﬁgure appears to have welldeﬁnedcontoursevenincompletelyhomogeneousareas;inthecenter ﬁgure, a large rectangle is perceived, and, in theﬁgureontheright,awhitesquarepartiallyoccludedbya gray disk is perceived. These contours, which are notcharacterized by image gradients, may be thought of as “apparent” or “subjective” contours. Following theusualconvention,wedistinguishbetween“modalcompletion”, in which the goal is to construct a perceivedboundary (as in Fig. 1, left and center) and “amodalcompletion”, in which one reconstructs the shape of a partially occluded objects (as in Fig. 1, right); seeKanizsa (1976, 1979).Ourgoalistoextract,i.e.segment,theseinternalob jects. Since some information is missing, one class of algorithms typically complete the segmentation process by building models to postulate what happens
202
Sarti, Malladi and Sethian
Figure 1
. Three images with subjective contours.
between the available data. In this paper we presenta method for segmentation of images with missingboundaries.We take the following perspective:
•
The observer is drawn to a reference point withinthe image, and this is a natural point from which toconstruct a completion process.
•
Starting from this given reference point, we candevise an algorithm which takes advantage of theavailable boundary data to construct a complete segmentation. Thus, the computed segmentation is afunction of the reference point.As in Sarti et al. (2000), we view a segmentationas the construction of a piecewise constant surfacethat varies rapidly across the boundary between different objects and stays
ﬂ
at within it. In Mumfordand Shah (1989) the segmentation is a piecewisesmooth/constant approximation of the image. In ourapproach the segmentation is a piecewise constant approximation of the pointofview or the reference surface. To obtain it we de
ﬁ
ne the following steps:1. Select a
ﬁ
xation point and build an initial surfacecorresponding to this pointofview.2. Detect existing local features in the image.3. Evolve the point of view surface with a
ﬂ
ow thatdepends both on the geometry of the surface andon the image features. The
ﬂ
ow evolves the initialconditiontowardsapiecewiseconstantsurface.Thecompleted object is the
ﬂ
at manifold at the top of the surface.4. (Automatically) Pick the level set that describes thedesired object (that is a closed curve).During the evolution, the pointofview surface isattracted by the existing boundaries and steepens. Thesurface evolves towards a piecewise constant solutionthrough the closing of the boundary fragments and the
ﬁ
lling in the homogeneous regions. A solid object isdelineated as a constant surface bounded by existingand recovered shape boundaries. With this method, theimage completion process depends both on the pointof view and on the geometric properties of the image(Sarti et al., 2000).Humanperceptualorganizationhasbeenextensivelystudied, and the resulting theories of vision are oftenused to construct computational models for image segmentation. For example, properties like boundary continuity of direction, regularity, proximity constraints,maximum homogeneity, closure, and symmetry havebeenappliedasguidelinesformanycomputervisionalgorithms. The perspective and algorithms in this paperare connected to some experimental results on perceptual organization of the human visual system outlinedbyKanizsa(1976,1979,1980).Inparticular,theemergence of the subjective contours depends on the position of the point of view:
“
if you
ﬁ
x your gaze on anapparent contour, it disappears, yet if you direct yourgazetotheentire
ﬁ
gure,thecontoursappeartobereal,
”
see Kanizsa (1976).In the following, we construct a geometrical modelforboundarycompletionbykeepinginmindtheaforementioned characteristics of perceptual organization.Both the mathematical and algorithmic approach inour method relies on a considerable body of recentworkbasedonapartialdifferentialequationsapproachto both front propagation and to image segmentation.Level set methods, introduced by Osher and Sethian(1988), track the evolution of curves and surfaces, using the theory of curve and surface evolution and thelink between front propagation and hyperbolic conservation laws developed by Sethian (1985, 1987). These
Subjective Surfaces: A Geometric Model for Boundary Completion 203methods embed the desired interface as the zero levelset of an implicit function, and then employ
ﬁ
nite differences to approximate the solution of the resultinginitial value partial differential equation. Malladi et al.(1993) and Caselles et al. (1993) used this technologyto segment images by growing trial shapes inside a region with a propagation velocity which depends on theimage gradient and hence stops when the boundary isreached; thus, image segmentation is transformed intoan initial value partial differential equation in which apropagating front stops at the desired edge.Sochen et al. (1998) view image processing as theevolution of an image manifold embedded in a higherdimensional Riemannian space towards a minimal surface. This framework has been applied to processingboth single and vectorvalued images de
ﬁ
ned in twoand higher dimensions (see Kimmel et al., 2000). Inthis paper we view segmentation as the evolution of aninitial reference manifold under the in
ﬂ
uence of localimage features.Our approach takes a more general view of the segmentation problem. Rather than follow a particularfront or level curve which one attempts to steer to thedesired edge (as in Malladi et al., 1993, 1995; MalladiandSethian,1998;Casellesetal.,1993,1997a,1997b),we begin with an initial surface, chosen on the basisof a usersupplied reference
ﬁ
xation point. We then
ﬂ
ow this
entire surface
under a speed law dependenton the image gradient, without regard to any particularlevelset.Suitablychosen,this
ﬂ
owsharpensthesurfacearound the edges and connects segmented boundariesacross the missing information. On the basis of thissurface sharpening, we can identify a level set corresponding to an appropriate segmented edge.InSartietal.(2000),thetheoreticalunderpinningsof this approach were presented. In this paper we concentrate on results for a variety of 2D images (cognitive,medical, photographic, texture images) and we extendthemethodologytotheextractionof3Dshapes.Inparticular we test our algorithm on 2D and 3D ultrasoundimages in which the signal to noise ratio is truly smalland the most part of the edge is missing.Thepaperisorganizedasthefollowing.InSection2we review past work in segmentation of images withmissing boundaries. In Section 3 we discuss the mathematical problem and in Section 4 we present a numerical method to solve it. In Section 5 the extensionof the 2D subjective surface model to 3D manifolds ispresented.InSection6,weshowresultsoftheapplication of the method to different 2D images and discussboth the modal and amodal completion scenarios. InSection7,resultsoftheapplicationofsubjectivemanifolds to 3D image completion are presented and a validation of the method on 3D echocardiographic imagesis provided. In Section 8, a discussion about the accuracy and the ef
ﬁ
ciency of the computational model isalso provided.
2. Past Work and Background
Inthissection,wereviewsomepreviousworkaimedatrecoveringsubjectivecontours.Weareinterestedinrecovering explicit shape representations that reproducethat of the human visual perception, especially in regions with no imagebased constraints such as gradient jump or variation in texture. In Mumford (1994),the distribution of subjective contours are modeledand computed by particles traveling at constant speedsbut moving in directions given by Brownian motion.Williams and Jacobs (1997a, 1997b) introduce the notion of a stochastic completion
ﬁ
eld, the distribution of particle trajectories joining pairs of position and direction constraints, and show how it could be computed.In this approach, a distribution of particles is beingcomputed, rather than an explicit contour/surface,closed or otherwise.A combinatorial approach is considered in Saund(1999). A sparse graph is constructed whose nodes aresalient visual events such as contrast edges and Ltypeand Ttype junctions of contrast edges, and whose arcsare coincidence and geometric con
ﬁ
gurational relations among node elements. An interpretation of thescene consists of choices among a small set of labelsfor graph elements. Any given labeling induces an energy, or cost, associated with physical consistency and
ﬁ
gural interpretation.A common feature of both completion
ﬁ
elds, combinatorialmethods,aswellasvariationalsegmentationmethods(MumfordandShah,1989)istopostulatethatthe segmentation process is independent of observer
’
spoint of focus. On the other hand, methods based onactive contours perform a segmentation strongly dependent on the user/observer interaction. Since theirintroduction in Kass et al. (1988), deformable models have been extensively used to integrate boundariesand extract features from images. An implicit shapemodeling approach with topological adaptability andsigni
ﬁ
cant computational advantages has been introduced in Malladi et al. (1993), Caselles et al. (1993)and Malladi et al. (1995). In these papers, the level set
204
Sarti, Malladi and Sethian
approach (Osher and Sethian, 1988; Sethian, 1999) isusedtoframecurvemotionwithacurvaturedependentspeed. These and a host of other related works rely onedge information to construct a shape representationof an object. In the presence of large gaps and missing edge data, the models can go astray and away fromthe required shape boundary. This behavior is due toa constant speed component in the governing equation that helps the curve from getting trapped by isolated spurious edges. On the other hand, if the constant in
ﬂ
ation term is switched off, as in Caselles et al.(1997a) and Sarti and Malladi (2001), the curve hasto be initialized close to the
ﬁ
nal shape for reasonableresults. We also note a more recent segmentation approach introduced in Chan and Vese (1998), in whichtheauthorsusegeometriccurveevolutionforsegmenting shapes without gradient by imposing a homogeneity constraint. An exhaustive review of algorithms forgrouping is outside the purpose of this paper; we referthe interested reader to Malik et al. (2000) for a largerperspective as well as for the classi
ﬁ
cation of groupingalgorithms in the broad families of regionbased andcontourbased approaches.The approach in this paper rests on view thatsegmentation, regardless of dimensionality, is a
‘
viewpoint
’
dependent computation.
1
The
“
viewpoint
”
orthe userde
ﬁ
ned initial guess to the segmentation algorithm serves as input to the algorithm, and is usedto construct a pointofview surface. Next, this referencesurfaceisevolvedaccordingtoafeatureindicatorfunction. The shape completion aspect relies on twocomponents: (1) the evolution of a surface and (2) a
ﬂ
ow that combines the effects of level set curve evolution with that of surface evolution. In what follows,wepresentageometricframeworkthatmakesthispossible. Computing the
ﬁ
nal segmentation (a contour orsurface) is accomplished by choosing and plotting aparticularlevelsetofahigherdimensionalfunction;aswe shall show, this particular level set may be chosenautomatically.
3. Theory
3.1. Local Feature Detection
Consider an image
I
:
(
x
,
y
)
→
I
(
x
,
y
)
as a real positive function de
ﬁ
ned in some domain
M
⊂
R
2
. Oneinitial task in image understanding is to build a representation of the local structure of the image. This ofteninvolves detection of intensity changes, orientation of structures, Tjunctions and texture. The result of thisstage is a representation corresponding to
the raw primal sketch
(Marr, 1982). Several methods have beenproposed to compute the raw primal sketch, includingmultiscale/multiorientation image decomposition withGabor
ﬁ
ltering (Gabor, 1946), wavelet transform (Lee,1996), deformable
ﬁ
lter banks (Perona, 1995), textons(Julesz,1981;Maliketal.,1999)etc.Forthepurposeof the present paper we consider a simple edge indicator,namely
g
(
x
,
y
)
=
11
+
(
∇
G
σ
(
x
,
y
)
I
(
x
,
y
)

/β)
2
where
G
σ
(ξ)
=
exp
(
−
(ξ/σ)
2
)σ
√
π
(1)The edge indicator function
g
(
x
,
y
)
is a nonincreasing function of
∇
G
σ
(
x
,
y
)
I
(
x
,
y
)

, where
G
σ
(
x
,
y
)
is a Gaussian kernel and
()
denotes the convolution.Thedenominatoristhegradientmagnitudeof asmoothedversionoftheinitialimage.Thus,thevalueof
g
is closer to 1 in
ﬂ
at areas (
∇
I
→
0) and closerto 0 in areas with large changes in image intensity, i.e.the local edge features. The minimal size of the detailsthat are detected is related to the size of the kernel,which acts like a scale parameter. By viewing
g
as apotential function, we note that its minima denotes theposition of edges. Also, the gradient of this potentialfunction is a force
ﬁ
eld that always points in the localedge direction; see Fig. 2.To compute
∇
G
σ
(
x
,
y
)
I
(
x
,
y
)
, we use the convolution derivative property
∇
G
σ
(
x
,
y
)
I
(
x
,
y
)
=∇
(
G
σ
(
x
,
y
)
I
(
x
,
y
))
and perform the convolution bysolving the linear heat equation
∂ϕ∂
t
= ∇ ·
(
∇
ϕ)
(2)in the time interval [0
,σ
] with the initial condition
ϕ(
x
,
y
,
0
)
=
I
(
x
,
y
)
. We conclude this subsection byobserving that there are other ways of both smoothingan image as well as computing an edge indicator function or in general a feature indicator function, see, forexample, Caselles et al. (1997a), Malladi et al. (1995),Malladi and Sethian (1996), Perona and Malik (1987),Sarti et al. (1999), Sochen et al. (1998) and Kimmelet al. (2000).
3.2. Global Boundary Integration
Now,considerasurface
S
:
(
x
,
y
)
→
(
x
,
y
,)
de
ﬁ
nedin the same domain
M
of the image
I
. The differential
Subjective Surfaces: A Geometric Model for Boundary Completion 205
Figure 2
. Local edge detection: The edge map
g
and its spatial gradient
−∇
g
.
area of the graph
S
in the Euclidean space is given by:
dA
=
1
+
2
x
+
2
y
dx dy
(3)We will use the edge indicator
g
to stretch and shrink a metric appropriately chosen so that the edges act asattractors under a particular
ﬂ
ow. With the metric
g
applied to the space, we have
dA
g
=
g
(
x
,
y
)
1
+
2
x
+
2
y
dx dy
.
(4)Now, consider the area of the surface
A
g
=
M
g
(
x
,
y
)
1
+
2
x
+
2
y
dx dy
,
(5)(see Appendix A for derivation) and evolve the surfacein order to reduce it. The steepest descent of Eq. (5)is obtained with usual multivariate calculus techniquesand results in the following
ﬂ
ow
∂∂
t
=
g
1
+
2
x
yy
−
2
x
y
xy
+
1
+
2
y
xx
1
+
2
x
+
2
y
+
(
g
x
x
+
g
y
y
)
(6)(see Appendix B for derivation).
2
The
ﬁ
rst term onthe right hand side is a parabolic term that evolvesthe surface in the normal direction under its mean curvature weighted by the edge indicator
g
. The surfacemotion is slowed down in the vicinity of edges (thatis, where
g
→
0). The second term on the right corresponds to pure passive advection of the surface alongtheunderlyingvelocity
ﬁ
eld
−∇
g
whosedirectionandstrength depend on position. This term pushes/attractsthe surface in the direction of the image edges. Notethat
g
(
I
(
x
,
y
))
is not a function of the third coordinate, therefore the vector
ﬁ
eld
−∇
g
lies entirely on the
(
x
,
y
)
plane.The following characterizes the behavior of the
ﬂ
ow(Eq. (6)):1. In regions of the image where edge information exists,theedgeindicator
g
→
0andthemainequation(Eq. (6)) reduces to a simple advection equation:
∂∂
t
≈
g
x
x
+
g
y
y
(7)drivingthesurfacetowardstheedges.Thelevelsetsof the surface are attracted to the edge and accumulate.Consequently,thespatialgradientincreasesand the surface begins to develop a discontinuity.2. Inside homogeneous regions of the image the edgeindicator
g
=
1,andEq.(6)reducestotheEuclideanmean curvature
ﬂ
ow:
∂∂
t
=
1
+
2
x
yy
−
2
x
y
xy
+
1
+
2
y
xx
1
+
2
x
+
2
y
(8)whosesolutionsarethehorizontalplanescharacterizing the inside of the
ﬁ
gures and the background.3. We now address the regions in the image corresponding to subjective contours. We take the viewthat appropriate subjective contours are continuation of existing edge fragments. As discussedabove, in regions with well de
ﬁ
ned edge information, Eq. (6) causes the level curves to accumulate,therebycausinganincreaseinthespatialgradientof
. Since continuity for the surface is required during the evolution, the edge fragment information