Beautification of Design Sketches Using Trainable Stroke Clustering and Curve Fitting

Beautification of Design Sketches Using Trainable Stroke Clustering and Curve Fitting
of 17
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1 Beautification of Design Sketches UsingTrainable Stroke Clustering and Curve Fitting Günay Orbay and Levent Burak Kara Abstract  —We propose a new sketch parsing and beautification method that converts digitally created design sketches into beautifiedline drawings. Our system uses a trainable, sequential bottom-up and top-down stroke clustering method that learns how to parseinput pen strokes into groups of strokes each representing a single curve, followed by point-cloud ordering that facilitates curve fittingand smoothing. This approach enables greater conceptual freedom during visual ideation activities by allowing designers to developtheir sketches using multiple, casually drawn strokes without requiring them to indicate the separation between different stroke groups.With the proposed method, raw sketches are seamlessly converted into vectorized geometric models, thus facilitating downstreamassessment and editing activities. Index Terms  —sketch based design, conceptual design, sketch parsing, sketch beautification, supervised stroke clustering, LaplacianEigenmaps, curve fitting. ✦ 1 I NTRODUCTION E ARLY  design ideation and product styling activitiesoften require intensive visual exploration, assess-ment, and iteration through which the product formeventually evolves [1]. During these activities, designerstypically externalize and communicate their ideas bygenerating a multitude of conceptual sketches, whichhas been recognized as a critically important stage forproduct design and development [2], [3]. In support of these early activities, recent technological advances haveled to a variety of pen-based digital design tools whichare becoming increasingly more accessible. However, akey issue that hinders a wide adoption of such toolsfor computer aided design and modeling is the diffi-culty in transforming input pen strokes into geometriccontent that accurately represents designers’ intentions.Due to this challenge, the majority of the current sketch- based design interfaces either leave the raw sketchesunprocessed (thus simply serving as digital drawingtools), or otherwise require users’ obtrusive interventionto transform input strokes into usable geometric forms.The latter often forces designers to abandon their naturaldrawing styles and attend to peripheral details, whichseverely inhibits their conceptual freedom.In this study, we describe a new computationalmethod that automatically transforms conceptual de-sign sketches into vectorized drawings using a trainablestroke clustering, point ordering, and curve fitting al-gorithms. From an input sketch containing an arbitrarynumber of strokes drawn in arbitrary directions andorder, our method identifies the salient stroke groups •  G. Orbay and L. B. Kara are with the Department of MechanicalEngineering, Carnegie Mellon University, Pittsburgh, PA, 15213. and beautifies each into a single geometric curve. Thistransformation allows raw design sketches to be seam-lessly converted into drawings consisting of parametriccurves, thus facilitating downstream computer aidedmodeling operations. Although this task is rather trivialfor humans, it is often not the case for computationalsystems due to the presence of many strokes exhibitinglarge variations in their intrinsic and extrinsic attributessuch as curvature, angular orientation, intersections, andspatial proximity. The key advantage of the proposedapproach is that it can learn how to group the strokes in agiven sketch by studying a previously created sketch (orsketches). Hence, the system can be readily adapted fordifferent designers and drawing styles without requiringexternal calibration or retuning.At the heart of our method is a set of stroke-levelgeometric features that encode and exploit the relation-ships between input strokes. From a manually clusteredtraining sketch, our system trains a neural network thatencodes a discriminative mapping between the featuresextracted from a pair of strokes, and a decision regardingwhether those strokes should be grouped together. Thisprocess results in a set of raw  stroke groups  each of whichmay contain a single geometric curve, or a set of curvesthat form locally difficult-to-detect bifurcations. These bifurcations are then detected using a set of global at-tributes of each group, resulting in distilled clusters eachrepresenting a single salient curve. Next, the focus shiftsfrom the strokes to the coordinate points comprisingthose strokes. From the aggregate set of unorganizedpoints extracted from each stroke cluster, our systemcomputes a spatial ordering of those points using aspectral dimensionality reduction method. It then usesthis point ordering to either generate a natural param-eterization for curve fitting, or to directly synthesize ageometric curve. To enhance robustness against non-  IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2 informative, exploratory strokes, our system additionallyexploits the variations in pen pressure during digitalsketching, thereby generating curve fits that are mostcommensurate with the designer’s intentions.With the proposed approach, drawings with sketchy,overlapping strokes forming open curves, closed curves,intersecting and self-intersecting curves, and curves ex-hibiting subtle bifurcations can be beautified into singu-lar geometric curves. In this work, we limit our approachto sketches that contain the  primary  drawing strokesforming the shape communicated in the scene. Hence,input sketches are assumed to be devoid of auxiliarystrokes such as scaffolding lines, shading strokes, orcross-hatches which are common in many professionaldesign sketches. When desirable, however, such set of strokes can be suitably separated into different setsusing the digital layering capabilities of existing drawingsoftware. 2 R ELATED WORK Advances in sketch-based design interfaces have led toa variety of novel technologies for 2D and 3D geometricmodeling. However, techniques necessary for transform-ing input pen strokes into forms suitable for digital mod-eling operations have not been well-developed. Igarashi et al.  [4] introduced  beautification  in which each inputstroke is converted into straight line segments using anumber of visual attributes such as connections to ex-isting segments, parallelism, perpendicularity and othersimilar geometric constraints. This approach has beendesigned to produce purely straight line segments, thusrestricting its use to rectilinear designs. This pioneeringwork has coined the term  beautification  of strokes, whichhas since become the de facto term for converting penstrokes into usable geometric entities.Studies [5], [6], [7], [8] proposed starting with an initialstroke and progressively improving it using modifierstrokes. Baudel [5] suggested a stroke-based spline edit-ing method which can take as input multiple strokesto modify a single curve. Avro and Novins [6] pro-posed an active sketch assistance system to “classifythe sketch as one of several pre-defined basic shapes”and to gradually morph the strokes to the identified basic shape on-the-fly. Bae  et al.  [7] suggested developingcurves by converting each stroke into a cubic Béziercurve and averaging all while favoring recently drawnstrokes over previously drawn ones. Kara  et al.  [8] usedactive contours to update existing curves under elasticforces exerted by the modifier strokes. Although thesemethods are useful in curve modification, the immediate beautification of the strokes requires users’ guidance andconfirmation of the results, which can disrupt the naturalflow of sketching.In studies [9], [10], researchers suggested using multi-ple strokes to define a single smooth curve via curve fit-ting. Schmidt  et al.  [9] used variational curves to convertthe input strokes into geometric curves via curve fairing,gap filling and smoothing operations. Kara and Shimada[10] utilized Principal Component Analysis (PCA) todetermine a spatial ordering for points comprising theinput strokes to facilitate B-spline curve fitting. How-ever, similar to the previous methods, these approachesrequire untimely attention to curve beautification, thusdisrupting fluidity during sketching. 2.1 Techniques for Drawing Simplification Studies [11], [12], [13], [14], [15] propose extractinggeometric curves from completed sketches. Saund [11]describe a drawing simplification algorithm to extractarcs in a drawing at different scales using  curve elementtokens . The method is concerned with identifying thecurvilinear structures within drawings at various scalescontrolled by the user. Our work on the other hand, aimsto simplify hand-drawn sketches into free-form curves.Shesh  et al.  [12] describe a sketch simplification methodthat replaces strokes sharing similar slopes and proxi-mate end points, with an average straight line. This ap-proach is designed for sketches composed exclusively of straight line segments. Pusch  et al.  [14] describe a sketch beautification method that produce the desired curveelements by first subdividing the sketch into rectangularregions until individual strokes are attained, and laterusing this information to construct intended curves. Thesubdivision-based approach makes the method moresuitable for non-intersecting curves. Shesh and Chen[15], propose a drawing simplification method whichuses efficient proximity calculations to facilitate strokegrouping. The amount of simplification is determinedthrough parameters that control stroke proximity andoverlap. This approach is concerned with reducing thenumber of geometric elements to be processed duringdynamic rendering at a local level, and is thus not de-signed to identify the underlying geometric structures ina drawing. Similarly, Barla  et al.  [13] propose a geometricclustering and simplification method that sequentiallymerges curves representable by a single curve at aprescribed scale. The merging decision is based on an ε -line that has a user specified width enveloping thecandidate curves. As their method uses a greedy cluster-ing and simplification approach, the resulting drawingis usually one of many admissible solutions. Moreover,the  ε -line approach favors mergers that exhibit sharpcorners, which in fact may be cues for different geometricstructures. Likewise, it fails to identify curves that may be self-intersecting as often times such curves do not liewithin an  ε -line. Our approach is designed to alleviatethese difficulties through a trainable stroke clusteringalgorithm that learns the rules for stroke grouping fromthe users’ sketches. Additionally, our approach is de-signed to handle self-intersecting and bifurcating curvesthat are often difficult to distinguish using a purely localanalysis. 2.2 Techniques for Curve Fitting to Point Sets There exists a number of model-based methods forcurve fitting to unorganized point sets [16], [17], [18].  IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 3 Wang  et al.  [16] describe a curvature-based squared dis-tance minimization method to fit non-self-intersecting,open/closed curves to unorganized point sets. The per-formance of their method is sensitive to the initializationof the model curve. Additionally, the user has to specifywhether an open or closed curve is to be fit. The methodproposed by Yang  et al.  [17] can fit self-intersectingclosed curves to unorganized point sets. However, theirwork is tailored toward closed curves, which facilitatethe use of encircling curves as the initial curve model.Both methods require a sufficiently reasonable startingcurve to make them computationally affordable. Movingleast squares (MLS) [19], [20] have also been used forcurve and surface fitting to unorganized point sets.Although MLS methods can represent a wide variety of shapes, they are limited to manifold geometries devoidof self-intersections. In addition to curve fitting methods,methods based on direct point set parameterization havealso been proposed. Goshtasby [21] propose an image- based method to parameterize point sets while auto-matically determining whether the resulting geometryshould represent an open or closed curve. However, itis not suitable for self-intersecting curves. In this paper,we propose a new parametrization and curve fittingmethod tailored toward sketch-based applications. Ourapproach does not require an initial curve model, andautomatically determines whether the final curve should be open or closed. It determines the parametrization onthe point set prior to curve fitting, and uses classicalfitting and smoothing methods designed for organizedpoint sets. We also describe a method that exploits penpressure to produce final curves that are aligned wellwith the users’ intentions. 3 U SER  I NTERACTION AND  O VERVIEW In this work, we propose a new sketch parsing and beautification method that converts digitally created con-ceptual design sketches into sketches consisting of beau-tified curve elements. Our system is designed to operateon a complete sketch rather than requiring the userto explicitly demarcate the separation between differentcurves during construction. We use a pressure sensitivegraphics tablet for recording the stylus position and tippressure as a function of time. While the temporal orderof the strokes is readily available using this setup, bydesign our system does not rely on this information.This enables strokes to be drawn in an arbitrary orderand allows the user to readily return to a previouslydeveloped portion of a sketch to add new strokes. Ourmethod consists of three main steps: (1) stroke clustering,(2) point reordering, and (3) curve fitting and smoothingand is illustrated in Figure 1. 4 T RAINABLE STROKE CLUSTERING In the first step, our system uses a trainable cluster-ing method to group input strokes into clusters eachforming a single, unique curve. At the heart of ourapproach is a bottom-up stroke fusion method followed by a top-down curve fission. In the initial bottom-upphase, our approach uses a trainable, neural networkmethod that takes as input a set of geometric featuresextracted from each stroke pair and decides whetherthe two strokes should reside in the same group. Oncepairwise decisions are made, our system consolidatesthe connected stroke pairs using a greedy linking algo-rithm to synthesize compound stroke groups. A com-pound stroke group may contain perceptually dominantstroke branches that require a global analysis beyondthe pairwise decisions. Such distinct branches are oftenthe cumulative result of small and gradual bifurcationsalong the raw strokes, and are thus undetectable using apurely local analysis. For each initial compound group,the subsequent top-down analysis temporarily revertsto a point-based representation and uses a spanningtree analysis to split the group at candidate bifurcationpoints. A key advance in this work is the ability to detectsuch bifurcations, which are identified as the nodes thatsplit the tree into similarly weighted branches. Finally,the branches identified in the point-based representationare transformed back to the stroke-based representation,thereby resulting in stroke clusters each correspondingto a branch-free, unique curve that can be subsequently beautified. 4.1 Preprocessing input strokes Strokes to our system are collected through a digitizingtablet which provides a series of data points sampledalong the trajectory of the stylus. Stylus slippage on thedrawing surface often results in unintended hooks at the beginning of the strokes. To alleviate the difficulties infeature extraction caused by this phenomenon, our sys-tem initially converts each incoming stroke into a cubicBézier curve similar to [7]. A fixed number of pointsare then extracted along the Bézier curve to produce adensely spaced set of coordinate points approximatingthe srcinal stroke. This intermediate transformation,however, is only used to expedite feature extraction instroke clustering as described in the next section. In thesubsequent curve beautification stage, our system reverts back to the srcinal points sampled along the stylusin order to produce final curves that accurately matchthe designer’s raw strokes. For scale independence, oursystem normalizes incoming sketches to a unit box whilepreserving their aspect ratios. 4.2 Feature extraction We define three geometric features to inform strokeclustering. The first feature,  remoteness , is a measure of spatial distance between a pair of strokes. The secondfeature,  misalignment , is a measure of angular distancerevealing how well two strokes are aligned with oneanother. The third feature,  discontinuity , measures thelikelihood of one stroke forming a natural continuationof another stroke. While we have found these featuresto be sufficiently informative for stroke clustering, our  IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 4 Pen strokes     f   o   r   e   a   c    h    c    l   u   s   t   e   r knownconnectivitynxn matrixManually clustered sketchwith n strokesAutomatically clustered sketchRaw sketch to be beautifiedwith m strokesfeaturesnxn matriceschainingPredictedconnectivitymxm matrixNeural Network Featuresmxm matrix UnorganizedpointsOrdered pointsPoints unfoldPen strokesUnorganizedpointsOrdered pointsPoints unfold remotenessmisalignmentdiscontinuity Stroke clustering Training sketchCurve fitting/smoothingPoint reordering    s   e    l    f  -    i   n   t   e   r   s   e   c   t    i   n   g   n   o   n   s   e    l    f  -    i   n   t   e   r   s   e   c   t    i   n   g        t      r      a        i      n        i      n      g Savitzky GolayBspline fittingFFT smoothing Beautified sketchCurvature plotremotenessmisalignmentdiscontinuity Fig. 1: Automatic beautification of design sketches: Raw sketches are grouped using a neural network based stroke clusteringmethod which is trained by manually clustered sketches. Resulting stroke clusters are converted into beautified curves via pointreordering and curve fitting/smoothing. system can be readily expanded to incorporate othergeometric features if desired. 4.2.1 Remoteness  We define remoteness as the shortest distance betweentwo strokes: d AB  =   x i  − x j   (1)where  x i  and  x j  are the position vectors of the pointsresulting in the shortest distance between strokes  A and  B . Here,   ·   is the Euclidean norm operator. Fig-ure 2a illustrates this measure. Note that since incomingsketches are normalized to the unit box, this distancewill always lie between  [0 , 1] . For a given pair of strokes,the remoteness will be expectedly large if this distance islarge relative to the size of the scene. However, the samedistance will appear smaller if the constituent strokes aremade longer relative to their gap, or the sketch containsother strokes that enlarge the size of the active scene.This phenomenon is desirably commensurate with hu-man vision, as our perception of proximity between twogeometric entities will vary relative to the size of theobserved scene. 4.2.2 Misalignment  Remoteness, while central to clustering, is not a sufficientmeasure of grouping alone as illustrated in Figure 2b.Here, while remoteness vanishes, a significant angularmisalignment suggests a separation between the strokes.Our examination of a large set of sketches and userreactions have revealed that humans tend to view twostrokes to belong to the same group if the strokes are wellaligned at their nearest points. Figure 2c illustrates such  IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 5 AB n A n B A θd B(c) Two well aligned strokes exhibitinga small angular misalignment(d) Illustration of the tangent vectorsat the nearest points(e) Two strokes exhibiting strongcontinuity as well as alignment(small discontinuity and misalignment)(f) Two strokes exhibiting strongalignment but weak continuity(a) Illustration of remoteness andmisalignment features(b) Two strokes exhibiting zeroremoteness but a large angular misalignmentA θ BABA n B d n AAB n B A d B n A n B n s n s ss Fig. 2: Geometric features defined on a pair of strokes A andB. a case. We quantify this observation as the misalignment between two strokes, which we formulate as the angulardifference between the two strokes’ tangent vectors atthe points of minimum distance (Fig. 2d): a AB  =  |  ( n A , n B ) | π/ 2  (2)where  n A  and  n B  are the unit tangent vectors on strokesA and B respectively and   ( · , · )  maps the angle betweentwo vectors to  [ − π/ 2 , + π/ 2] . 4.2.3 Discontinuity  Small values of this feature indicate the likelihood of onestroke naturally continuing another one. Figures 2e andf exhibit cases of strong continuity and weak continuity,respectively. Note that in both cases the strokes arewell aligned, with small misalignment feature values.We define discontinuity in relation to the unit tangentvectors  n A  and  n B  at the end points, and the distancevector  s  between the end points: c AB  =   ( n A  × n B )   +   ( n A  × n s )   +   ( n B  × n s )  3  | s | (3)where  n s  is the normalized unit vector along distancevector  s . The cross-product operator  ×  results in vectorsperpendicular to the drawing plane. When all threevectors  n A ,  n B , and  n s  are aligned with one another,this measure will attain a small value indicating strongcontinuity. Discontinuity will increase with higher mis-alignment between any two of the constituent vectors. 4.3 Supervised Bottom-up Stroke Fusion Training:  From a manually clustered training sketch,a neural network is trained that predicts the pairwisestroke groupings. For an arbitrary pair of strokes markedto be in the same group during training, the neuralnetwork must take as input the distances produced bythe three geometric features described above. As shownin Figure 3, however, spatially separated strokes may in-deed be perceptually  proximate  in the presence of neigh- boring strokes that bridge the two strokes. To reliablyidentify the distance between two arbitrary strokes, wethus take into account the circuitry that exists betweenthe strokes in the form of a principal distance. For eachof the three features, we define this distance as thetotal distance traveled along the shortest path connectingthe two strokes ( i.e.  the geodesic distance between thestrokes) averaged by the number of strokes visited. Thisdistance is calculated for all three features separately.Mathematically, the training distance between the  i th and  j th strokes in the  k th feature thus becomes: df  k ( i,j ) =  d kG ( i,j ) n kG ( i,j )  if   i ,  j  are in same group d k ( i,j )  otherwise(4)where  d kG ( i,j )  is the geodesic distance,  n kG ( i,j )  is thenumber of strokes traveled along the shortest path, df  k ( i,j )  is the training distance in  k th feature, and  d k ( i,j ) is the actual distance. We calculate the geodesic distanceusing the Dijkstra’s algorithm. Note that with this formu-lation, feature distances between the strokes belongingto the same cluster are attenuated, while features forthe strokes in different clusters remain unchanged. Oncecalculated, the training distances in each feature arecollected into a single vector as: df  ( i,j ) =   df  1 ( i,j )  df  2 ( i,j )  df  3 ( i,j )  T  (5)where  df  ( i,j )  is a feature vector composed of the train-ing distances in features,  df  k ( i,j ) ’s. Each feature vectoris then associated to a binary class decision as c ( i,j ) =   1  if   i ,  j  are in same group 0  otherwise (6)This adjustment of the pairwise stroke similaritiesthrough average geodesic distances is analogous to the average with-in cluster point distances  presented in [22],which exploit the bridge points to adjust the nominaldistances between point pairs.During training, the neural network takes as inputthe three training distances corresponding to each of the three features described above. All stroke pairs inthe sketch are utilized as training samples thus pro-viding both positive and negative samples. We use a
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks