IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1
Beautiﬁcation of Design Sketches UsingTrainable Stroke Clustering and Curve Fitting
Günay Orbay and Levent Burak Kara
Abstract
—We propose a new sketch parsing and beautiﬁcation method that converts digitally created design sketches into beautiﬁedline drawings. Our system uses a trainable, sequential bottomup and topdown stroke clustering method that learns how to parseinput pen strokes into groups of strokes each representing a single curve, followed by pointcloud ordering that facilitates curve ﬁttingand smoothing. This approach enables greater conceptual freedom during visual ideation activities by allowing designers to developtheir sketches using multiple, casually drawn strokes without requiring them to indicate the separation between different stroke groups.With the proposed method, raw sketches are seamlessly converted into vectorized geometric models, thus facilitating downstreamassessment and editing activities.
Index Terms
—sketch based design, conceptual design, sketch parsing, sketch beautiﬁcation, supervised stroke clustering, LaplacianEigenmaps, curve ﬁtting.
✦
1 I
NTRODUCTION
E
ARLY
design ideation and product styling activitiesoften require intensive visual exploration, assessment, and iteration through which the product formeventually evolves [1]. During these activities, designerstypically externalize and communicate their ideas bygenerating a multitude of conceptual sketches, whichhas been recognized as a critically important stage forproduct design and development [2], [3]. In support of these early activities, recent technological advances haveled to a variety of penbased digital design tools whichare becoming increasingly more accessible. However, akey issue that hinders a wide adoption of such toolsfor computer aided design and modeling is the difﬁculty in transforming input pen strokes into geometriccontent that accurately represents designers’ intentions.Due to this challenge, the majority of the current sketch based design interfaces either leave the raw sketchesunprocessed (thus simply serving as digital drawingtools), or otherwise require users’ obtrusive interventionto transform input strokes into usable geometric forms.The latter often forces designers to abandon their naturaldrawing styles and attend to peripheral details, whichseverely inhibits their conceptual freedom.In this study, we describe a new computationalmethod that automatically transforms conceptual design sketches into vectorized drawings using a trainablestroke clustering, point ordering, and curve ﬁtting algorithms. From an input sketch containing an arbitrarynumber of strokes drawn in arbitrary directions andorder, our method identiﬁes the salient stroke groups
•
G. Orbay and L. B. Kara are with the Department of MechanicalEngineering, Carnegie Mellon University, Pittsburgh, PA, 15213.
and beautiﬁes each into a single geometric curve. Thistransformation allows raw design sketches to be seamlessly converted into drawings consisting of parametriccurves, thus facilitating downstream computer aidedmodeling operations. Although this task is rather trivialfor humans, it is often not the case for computationalsystems due to the presence of many strokes exhibitinglarge variations in their intrinsic and extrinsic attributessuch as curvature, angular orientation, intersections, andspatial proximity. The key advantage of the proposedapproach is that it can learn how to group the strokes in agiven sketch by studying a previously created sketch (orsketches). Hence, the system can be readily adapted fordifferent designers and drawing styles without requiringexternal calibration or retuning.At the heart of our method is a set of strokelevelgeometric features that encode and exploit the relationships between input strokes. From a manually clusteredtraining sketch, our system trains a neural network thatencodes a discriminative mapping between the featuresextracted from a pair of strokes, and a decision regardingwhether those strokes should be grouped together. Thisprocess results in a set of raw
stroke groups
each of whichmay contain a single geometric curve, or a set of curvesthat form locally difﬁculttodetect bifurcations. These bifurcations are then detected using a set of global attributes of each group, resulting in distilled clusters eachrepresenting a single salient curve. Next, the focus shiftsfrom the strokes to the coordinate points comprisingthose strokes. From the aggregate set of unorganizedpoints extracted from each stroke cluster, our systemcomputes a spatial ordering of those points using aspectral dimensionality reduction method. It then usesthis point ordering to either generate a natural parameterization for curve ﬁtting, or to directly synthesize ageometric curve. To enhance robustness against non
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2
informative, exploratory strokes, our system additionallyexploits the variations in pen pressure during digitalsketching, thereby generating curve ﬁts that are mostcommensurate with the designer’s intentions.With the proposed approach, drawings with sketchy,overlapping strokes forming open curves, closed curves,intersecting and selfintersecting curves, and curves exhibiting subtle bifurcations can be beautiﬁed into singular geometric curves. In this work, we limit our approachto sketches that contain the
primary
drawing strokesforming the shape communicated in the scene. Hence,input sketches are assumed to be devoid of auxiliarystrokes such as scaffolding lines, shading strokes, orcrosshatches which are common in many professionaldesign sketches. When desirable, however, such set of strokes can be suitably separated into different setsusing the digital layering capabilities of existing drawingsoftware.
2 R
ELATED WORK
Advances in sketchbased design interfaces have led toa variety of novel technologies for 2D and 3D geometricmodeling. However, techniques necessary for transforming input pen strokes into forms suitable for digital modeling operations have not been welldeveloped. Igarashi
et al.
[4] introduced
beautiﬁcation
in which each inputstroke is converted into straight line segments using anumber of visual attributes such as connections to existing segments, parallelism, perpendicularity and othersimilar geometric constraints. This approach has beendesigned to produce purely straight line segments, thusrestricting its use to rectilinear designs. This pioneeringwork has coined the term
beautiﬁcation
of strokes, whichhas since become the de facto term for converting penstrokes into usable geometric entities.Studies [5], [6], [7], [8] proposed starting with an initialstroke and progressively improving it using modiﬁerstrokes. Baudel [5] suggested a strokebased spline editing method which can take as input multiple strokesto modify a single curve. Avro and Novins [6] proposed an active sketch assistance system to “classifythe sketch as one of several predeﬁned basic shapes”and to gradually morph the strokes to the identiﬁed basic shape ontheﬂy. Bae
et al.
[7] suggested developingcurves by converting each stroke into a cubic Béziercurve and averaging all while favoring recently drawnstrokes over previously drawn ones. Kara
et al.
[8] usedactive contours to update existing curves under elasticforces exerted by the modiﬁer strokes. Although thesemethods are useful in curve modiﬁcation, the immediate beautiﬁcation of the strokes requires users’ guidance andconﬁrmation of the results, which can disrupt the naturalﬂow of sketching.In studies [9], [10], researchers suggested using multiple strokes to deﬁne a single smooth curve via curve ﬁtting. Schmidt
et al.
[9] used variational curves to convertthe input strokes into geometric curves via curve fairing,gap ﬁlling and smoothing operations. Kara and Shimada[10] utilized Principal Component Analysis (PCA) todetermine a spatial ordering for points comprising theinput strokes to facilitate Bspline curve ﬁtting. However, similar to the previous methods, these approachesrequire untimely attention to curve beautiﬁcation, thusdisrupting ﬂuidity during sketching.
2.1 Techniques for Drawing Simpliﬁcation
Studies [11], [12], [13], [14], [15] propose extractinggeometric curves from completed sketches. Saund [11]describe a drawing simpliﬁcation algorithm to extractarcs in a drawing at different scales using
curve elementtokens
. The method is concerned with identifying thecurvilinear structures within drawings at various scalescontrolled by the user. Our work on the other hand, aimsto simplify handdrawn sketches into freeform curves.Shesh
et al.
[12] describe a sketch simpliﬁcation methodthat replaces strokes sharing similar slopes and proximate end points, with an average straight line. This approach is designed for sketches composed exclusively of straight line segments. Pusch
et al.
[14] describe a sketch beautiﬁcation method that produce the desired curveelements by ﬁrst subdividing the sketch into rectangularregions until individual strokes are attained, and laterusing this information to construct intended curves. Thesubdivisionbased approach makes the method moresuitable for nonintersecting curves. Shesh and Chen[15], propose a drawing simpliﬁcation method whichuses efﬁcient proximity calculations to facilitate strokegrouping. The amount of simpliﬁcation is determinedthrough parameters that control stroke proximity andoverlap. This approach is concerned with reducing thenumber of geometric elements to be processed duringdynamic rendering at a local level, and is thus not designed to identify the underlying geometric structures ina drawing. Similarly, Barla
et al.
[13] propose a geometricclustering and simpliﬁcation method that sequentiallymerges curves representable by a single curve at aprescribed scale. The merging decision is based on an
ε
line that has a user speciﬁed width enveloping thecandidate curves. As their method uses a greedy clustering and simpliﬁcation approach, the resulting drawingis usually one of many admissible solutions. Moreover,the
ε
line approach favors mergers that exhibit sharpcorners, which in fact may be cues for different geometricstructures. Likewise, it fails to identify curves that may be selfintersecting as often times such curves do not liewithin an
ε
line. Our approach is designed to alleviatethese difﬁculties through a trainable stroke clusteringalgorithm that learns the rules for stroke grouping fromthe users’ sketches. Additionally, our approach is designed to handle selfintersecting and bifurcating curvesthat are often difﬁcult to distinguish using a purely localanalysis.
2.2 Techniques for Curve Fitting to Point Sets
There exists a number of modelbased methods forcurve ﬁtting to unorganized point sets [16], [17], [18].
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 3
Wang
et al.
[16] describe a curvaturebased squared distance minimization method to ﬁt nonselfintersecting,open/closed curves to unorganized point sets. The performance of their method is sensitive to the initializationof the model curve. Additionally, the user has to specifywhether an open or closed curve is to be ﬁt. The methodproposed by Yang
et al.
[17] can ﬁt selfintersectingclosed curves to unorganized point sets. However, theirwork is tailored toward closed curves, which facilitatethe use of encircling curves as the initial curve model.Both methods require a sufﬁciently reasonable startingcurve to make them computationally affordable. Movingleast squares (MLS) [19], [20] have also been used forcurve and surface ﬁtting to unorganized point sets.Although MLS methods can represent a wide variety of shapes, they are limited to manifold geometries devoidof selfintersections. In addition to curve ﬁtting methods,methods based on direct point set parameterization havealso been proposed. Goshtasby [21] propose an image based method to parameterize point sets while automatically determining whether the resulting geometryshould represent an open or closed curve. However, itis not suitable for selfintersecting curves. In this paper,we propose a new parametrization and curve ﬁttingmethod tailored toward sketchbased applications. Ourapproach does not require an initial curve model, andautomatically determines whether the ﬁnal curve should be open or closed. It determines the parametrization onthe point set prior to curve ﬁtting, and uses classicalﬁtting and smoothing methods designed for organizedpoint sets. We also describe a method that exploits penpressure to produce ﬁnal curves that are aligned wellwith the users’ intentions.
3 U
SER
I
NTERACTION AND
O
VERVIEW
In this work, we propose a new sketch parsing and beautiﬁcation method that converts digitally created conceptual design sketches into sketches consisting of beautiﬁed curve elements. Our system is designed to operateon a complete sketch rather than requiring the userto explicitly demarcate the separation between differentcurves during construction. We use a pressure sensitivegraphics tablet for recording the stylus position and tippressure as a function of time. While the temporal orderof the strokes is readily available using this setup, bydesign our system does not rely on this information.This enables strokes to be drawn in an arbitrary orderand allows the user to readily return to a previouslydeveloped portion of a sketch to add new strokes. Ourmethod consists of three main steps: (1) stroke clustering,(2) point reordering, and (3) curve ﬁtting and smoothingand is illustrated in Figure 1.
4 T
RAINABLE STROKE CLUSTERING
In the ﬁrst step, our system uses a trainable clustering method to group input strokes into clusters eachforming a single, unique curve. At the heart of ourapproach is a bottomup stroke fusion method followed by a topdown curve ﬁssion. In the initial bottomupphase, our approach uses a trainable, neural networkmethod that takes as input a set of geometric featuresextracted from each stroke pair and decides whetherthe two strokes should reside in the same group. Oncepairwise decisions are made, our system consolidatesthe connected stroke pairs using a greedy linking algorithm to synthesize compound stroke groups. A compound stroke group may contain perceptually dominantstroke branches that require a global analysis beyondthe pairwise decisions. Such distinct branches are oftenthe cumulative result of small and gradual bifurcationsalong the raw strokes, and are thus undetectable using apurely local analysis. For each initial compound group,the subsequent topdown analysis temporarily revertsto a pointbased representation and uses a spanningtree analysis to split the group at candidate bifurcationpoints. A key advance in this work is the ability to detectsuch bifurcations, which are identiﬁed as the nodes thatsplit the tree into similarly weighted branches. Finally,the branches identiﬁed in the pointbased representationare transformed back to the strokebased representation,thereby resulting in stroke clusters each correspondingto a branchfree, unique curve that can be subsequently beautiﬁed.
4.1 Preprocessing input strokes
Strokes to our system are collected through a digitizingtablet which provides a series of data points sampledalong the trajectory of the stylus. Stylus slippage on thedrawing surface often results in unintended hooks at the beginning of the strokes. To alleviate the difﬁculties infeature extraction caused by this phenomenon, our system initially converts each incoming stroke into a cubicBézier curve similar to [7]. A ﬁxed number of pointsare then extracted along the Bézier curve to produce adensely spaced set of coordinate points approximatingthe srcinal stroke. This intermediate transformation,however, is only used to expedite feature extraction instroke clustering as described in the next section. In thesubsequent curve beautiﬁcation stage, our system reverts back to the srcinal points sampled along the stylusin order to produce ﬁnal curves that accurately matchthe designer’s raw strokes. For scale independence, oursystem normalizes incoming sketches to a unit box whilepreserving their aspect ratios.
4.2 Feature extraction
We deﬁne three geometric features to inform strokeclustering. The ﬁrst feature,
remoteness
, is a measure of spatial distance between a pair of strokes. The secondfeature,
misalignment
, is a measure of angular distancerevealing how well two strokes are aligned with oneanother. The third feature,
discontinuity
, measures thelikelihood of one stroke forming a natural continuationof another stroke. While we have found these featuresto be sufﬁciently informative for stroke clustering, our
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 4
Pen strokes
f o r e a c h c l u s t e r
knownconnectivitynxn matrixManually clustered sketchwith n strokesAutomatically clustered sketchRaw sketch to be beautiﬁedwith m strokesfeaturesnxn matriceschainingPredictedconnectivitymxm matrixNeural Network Featuresmxm matrix
UnorganizedpointsOrdered pointsPoints unfoldPen strokesUnorganizedpointsOrdered pointsPoints unfold
remotenessmisalignmentdiscontinuity
Stroke clustering Training sketchCurve ﬁtting/smoothingPoint reordering
s e l f  i n t e r s e c t i n g n o n s e l f  i n t e r s e c t i n g
t r a i n i n g
Savitzky GolayBspline ﬁttingFFT smoothing
Beautiﬁed sketchCurvature plotremotenessmisalignmentdiscontinuity
Fig. 1: Automatic beautiﬁcation of design sketches: Raw sketches are grouped using a neural network based stroke clusteringmethod which is trained by manually clustered sketches. Resulting stroke clusters are converted into beautiﬁed curves via pointreordering and curve ﬁtting/smoothing.
system can be readily expanded to incorporate othergeometric features if desired.
4.2.1 Remoteness
We deﬁne remoteness as the shortest distance betweentwo strokes:
d
AB
=
x
i
−
x
j
(1)where
x
i
and
x
j
are the position vectors of the pointsresulting in the shortest distance between strokes
A
and
B
. Here,
·
is the Euclidean norm operator. Figure 2a illustrates this measure. Note that since incomingsketches are normalized to the unit box, this distancewill always lie between
[0
,
1]
. For a given pair of strokes,the remoteness will be expectedly large if this distance islarge relative to the size of the scene. However, the samedistance will appear smaller if the constituent strokes aremade longer relative to their gap, or the sketch containsother strokes that enlarge the size of the active scene.This phenomenon is desirably commensurate with human vision, as our perception of proximity between twogeometric entities will vary relative to the size of theobserved scene.
4.2.2 Misalignment
Remoteness, while central to clustering, is not a sufﬁcientmeasure of grouping alone as illustrated in Figure 2b.Here, while remoteness vanishes, a signiﬁcant angularmisalignment suggests a separation between the strokes.Our examination of a large set of sketches and userreactions have revealed that humans tend to view twostrokes to belong to the same group if the strokes are wellaligned at their nearest points. Figure 2c illustrates such
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 5
AB
n
A
n
B
A
θd
B(c) Two well aligned strokes exhibitinga small angular misalignment(d) Illustration of the tangent vectorsat the nearest points(e) Two strokes exhibiting strongcontinuity as well as alignment(small discontinuity and misalignment)(f) Two strokes exhibiting strongalignment but weak continuity(a) Illustration of remoteness andmisalignment features(b) Two strokes exhibiting zeroremoteness but a large angular misalignmentA
θ
BABA
n
B
d
n
AAB
n
B
A
d
B
n
A
n
B
n
s
n
s
ss
Fig. 2: Geometric features deﬁned on a pair of strokes A andB.
a case. We quantify this observation as the misalignment between two strokes, which we formulate as the angulardifference between the two strokes’ tangent vectors atthe points of minimum distance (Fig. 2d):
a
AB
=

(
n
A
,
n
B
)

π/
2
(2)where
n
A
and
n
B
are the unit tangent vectors on strokesA and B respectively and
(
·
,
·
)
maps the angle betweentwo vectors to
[
−
π/
2
,
+
π/
2]
.
4.2.3 Discontinuity
Small values of this feature indicate the likelihood of onestroke naturally continuing another one. Figures 2e andf exhibit cases of strong continuity and weak continuity,respectively. Note that in both cases the strokes arewell aligned, with small misalignment feature values.We deﬁne discontinuity in relation to the unit tangentvectors
n
A
and
n
B
at the end points, and the distancevector
s
between the end points:
c
AB
=
(
n
A
×
n
B
)
+
(
n
A
×
n
s
)
+
(
n
B
×
n
s
)
3

s

(3)where
n
s
is the normalized unit vector along distancevector
s
. The crossproduct operator
×
results in vectorsperpendicular to the drawing plane. When all threevectors
n
A
,
n
B
, and
n
s
are aligned with one another,this measure will attain a small value indicating strongcontinuity. Discontinuity will increase with higher misalignment between any two of the constituent vectors.
4.3 Supervised Bottomup Stroke Fusion
Training:
From a manually clustered training sketch,a neural network is trained that predicts the pairwisestroke groupings. For an arbitrary pair of strokes markedto be in the same group during training, the neuralnetwork must take as input the distances produced bythe three geometric features described above. As shownin Figure 3, however, spatially separated strokes may indeed be perceptually
proximate
in the presence of neigh boring strokes that bridge the two strokes. To reliablyidentify the distance between two arbitrary strokes, wethus take into account the circuitry that exists betweenthe strokes in the form of a principal distance. For eachof the three features, we deﬁne this distance as thetotal distance traveled along the shortest path connectingthe two strokes (
i.e.
the geodesic distance between thestrokes) averaged by the number of strokes visited. Thisdistance is calculated for all three features separately.Mathematically, the training distance between the
i
th
and
j
th
strokes in the
k
th
feature thus becomes:
df
k
(
i,j
) =
d
kG
(
i,j
)
n
kG
(
i,j
)
if
i
,
j
are in same group
d
k
(
i,j
)
otherwise(4)where
d
kG
(
i,j
)
is the geodesic distance,
n
kG
(
i,j
)
is thenumber of strokes traveled along the shortest path,
df
k
(
i,j
)
is the training distance in
k
th
feature, and
d
k
(
i,j
)
is the actual distance. We calculate the geodesic distanceusing the Dijkstra’s algorithm. Note that with this formulation, feature distances between the strokes belongingto the same cluster are attenuated, while features forthe strokes in different clusters remain unchanged. Oncecalculated, the training distances in each feature arecollected into a single vector as:
df
(
i,j
) =
df
1
(
i,j
)
df
2
(
i,j
)
df
3
(
i,j
)
T
(5)where
df
(
i,j
)
is a feature vector composed of the training distances in features,
df
k
(
i,j
)
’s. Each feature vectoris then associated to a binary class decision as
c
(
i,j
) =
1
if
i
,
j
are in same group
0
otherwise (6)This adjustment of the pairwise stroke similaritiesthrough average geodesic distances is analogous to the
average within cluster point distances
presented in [22],which exploit the bridge points to adjust the nominaldistances between point pairs.During training, the neural network takes as inputthe three training distances corresponding to each of the three features described above. All stroke pairs inthe sketch are utilized as training samples thus providing both positive and negative samples. We use a