Blind removal of lens distortion

Virtually all imaging devices introduce some amount of geometric lens distortion. A technique is presented for blindly removing these distortions in the absence of any calibration information or explicit knowledge of the imaging device. The basic
of 10
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  to appear: Journal of the Optical Society of America A, 2001. Blind Removal of Lens Distortion Hany Farid and Alin C. PopescuDepartment of Computer ScienceDartmouth CollegeHanover NH 03755Virtually all imaging devices introduce some amount of geometric lens distortion. This paperpresents a technique for blindly removing these distortions in the absence of any calibrationinformation or explicit knowledge of the imaging device. The basic approach exploits thefact that lens distortion introduces specific higher-order correlations in the frequency domain.These correlations can be detected using tools from polyspectral analysis. The amount of dis-tortion is then estimated by minimizing these correlations.1  1 Introduction Virtually all medium- to low-grade imaging de-vices introduce some amount of geometric dis-tortion. Thesedistortionsareoftendescribedwitha one-parameter radially symmetric model [2, 8,9]. Given an ideal undistorted image f  u ( x,y ) ,the distortedimage is denotedas f  d (˜ x, ˜ y ) , wherethe distorted spatial parameters are given by: ˜ x = x (1 + κr 2 ) and ˜ y = y (1 + κr 2 ) , (1)where r 2 = x 2 + y 2 , and κ controls the amountof distortion. Shown in Figure 1 are the resultsof distorting a rectilinear grid with positive andnegative values of  κ .While these distortions may be artistically in-teresting it is oftendesirable to remove these ge-ometric distortions for many applications in im-age processing and computer vision (e.g., struc-ture estimation, image mosaicing). The amountof distortion is typically determined experimen-tally by imaging acalibration targetwith knownfiducial points. The deviation of these pointsfrom their srcinal positions is used to estimatethe amount of distortion (e.g., [9]). But oftensuch calibration is not available or direct accessto the imaging device is not possible, for exam-ple when down-loading an image from the web.Inaddition, thedistortionparameterscanchangeas other imaging parameters are varied (e.g., fo-callengthorzoom),thus requiring repeatedcali- brationforallpossible camerasettings. Analter-native calibration technique relies on the pres-ence of straight lines in the scene (e.g., [1, 7]).These lines, mapped to curves in the image dueto the distortion, are located or specified by theuser. The distortions are estimated by findingthemodelparametersthatmapthesecurvedlinesto straight lines. While this technique is moreflexible than those based on imaging a calibra-tion target, it still relies on the scene containingextended straight lines.In this paper a technique is presented for es-timating the amount of lens distortion in the ab-senceofanycalibrationinformationorscenecon-tent. The basic approach exploits the fact that κ < 0 κ = 0 κ > 0 Figure 1: One-parameter radially symmetriclens distortion, Equation (1). lens distortion introduces specific higher-ordercorrelations in the frequency domain. These cor-relationscanbedetectedusing toolsfrompolyspec-tral analysis. The amount of distortion is thendeterminedbyminimizing thesecorrelations. These basic principles were used in a related paper inwhich we introduced a technique for the blindremoval of luminance non-linearities [3].Insight is gained into the proposed technique by first considering what effect a geometric dis-tortion has on a one-dimensional signal. Con-sider, for example, a pure sinusoid with ampli-tude a and frequency b : f  u ( x ) = a cos( bx ) . (2)For purposes of exposition, consider a simpli-fied version of the lens distortiongiven in Equa-tion (1), where the spatial parameter is squared: f  d ( x ) = a cos( bx 2 ) . (3)This signal is composed of a multitude of har-monics. This canbeseenbyconsidering itsFouriertransform: F  d ( ω ) =   ∞−∞ f  d ( x ) e − iωx dx = 2   ∞ 0 a cos( bx 2 )cos( ωx ) dx. (4)Because the signal is symmetric (a cosine), theFourier integral may be expressed from 0 to ∞ andwithrespecttoonlythecosinebasis(i.e., thesine component of the complex exponential in-tegrates to zero). This integral has a closed form2  solution [4] given by: F  d ( ω ) = 2 a   π 2 b  cos  ω 2 2 b  + sin  ω 2 2 b  . (5)Unlike the undistorted signal, with: F  u ( ω ) =  1 | ω | = b 0 | ω |  = b (6)theFouriertransformofthedistortedsignal con-tains a multitude of harmonics. Moreover, theamplitude andphase ofthese harmonics arecor-related to the srcinal signal. Here the phasesaretriviallycorrelatedasallfrequencies arezero-phase. Nevertheless,iftheinitialsignal consistedofmultiplefrequencies withnon-zerophases, thenthe resulting distorted signal would have simi-lar amplitude correlations and non-trivial phasecorrelations.In what follows we will show that this obser-vation is not limited to the specific choice of sig-nal or distortion. We will also show empiricallythat when an image is geometrically distorted,higher-ordercorrelations inthefrequency domainincrease proportional to the amount of distor-tion. As such, the amount of distortion can bedeterminedby simply minimizing these correla-tions. We first show how toolsfrom polyspectralanalysis can be used to measure these higher-order correlations, and then show the efficacy of this technique to the blind removal of lens dis-tortion in synthetic and natural images. 2 BispectralAnalysis Considerastochastic one-dimensional signal f  ( x ) ,and its Fourier transform: F  ( ω ) = ∞  k = −∞ f  ( k ) e − iωk . (7)It is commonpracticetouse the powerspectrumto estimate second-order correlations: P  ( ω ) = E { F  ( ω ) F  ∗ ( ω ) } , (8)where E{·} is the expected value operator, and ∗ denotescomplexconjugate. Howeverthepowerspectrum is blind to higher-order correlations of thesortintroducedbyanon-linearity, Equation(1).Thesecorrelations canhoweverbeestimatedwithhigher-order spectra (see [6] for a thorough sur-vey). Forexamplethebispectrumestimatesthird-order correlations and is defined as: B ( ω 1 ,ω 2 ) = E { F  ( ω 1 ) F  ( ω 2 ) F  ∗ ( ω 1 + ω 2 ) } . (9)Notethat unlike the powerspectrum the bispec-trum of a real signal is complex-valued.The bispectrum reveals correlations betweenharmonically related frequencies, for example, [ ω 1 ,ω 1 , 2 ω 1 ] or [ ω 1 ,ω 2 ,ω 1 + ω 2 ] . If it is assumedthat the signal f  ( x ) is ergodic, then the bispec-trum can be estimated by dividing f  ( x ) into N  (possibly overlapping) segments, computing Fouriertransforms of each segment, and then averagingthe individual estimates: ˆ B ( ω 1 ,ω 2 ) =1 N  N   k =1 F  k ( ω 1 ) F  k ( ω 2 ) F  ∗ k ( ω 1 + ω 2 ) , (10) where F  k ( · ) denotes the Fourier transform of the k th segment. This arithmetic averageestima-toris unbiased and of minimum variance. How-ever, it has the undesired property that its vari-ance at each bi-frequency ( ω 1 ,ω 2 ) depends on P  ( ω 1 ) , P  ( ω 2 ) , and P  ( ω 1 + ω 2 ) (see e.g., [5]). Wedesire an estimator whose variance is indepen-dent of the bi-frequency. To this end, we employthe bicoherence, a normalized bispectrum, de-fined as: b 2 ( ω 1 ,ω 2 ) = | B ( ω 1 ,ω 2 ) | 2 E{| F  ( ω 1 ) F  ( ω 2 ) | 2 }E{| F  ( ω 1 + ω 2 ) | 2 } . (11) Itisstraight-forwardtoshowusing theSchwartzinequality thatthis quantity isguaranteedtohavevalues in the range [0 , 1] . As with the bispec-trum, the bicoherence can be estimated as: ˆ b ( ω 1 ,ω 2 ) = | 1 N   k F  k ( ω 1 ) F  k ( ω 2 ) F  ∗ k ( ω 1 + ω 2 ) |   1 N   k | F  k ( ω 1 ) F  k ( ω 2 ) | 21 N   k | F  k ( ω 1 + ω 2 ) | 2 . (12) Note that the bicoherence is now a real-valuedquantity.Shown in Figure 2 is an example of the sen-sitivity of the bicoherence to higher-order corre-lations that are invisible to the power spectrum.3  power bicoherence 001 π ω 1 ω 2 ω 3 001 π ω 1 ω 2 ω 3 Figure 2: Top: the normalized power spec-trum and bicoherence for a signal with ran-dom amplitudes and phases. Bottom: thesamesignalwithone frequency, ω 3 = ω 1 + ω 2 ,whose amplitude and phase are correlated to ω 1 and ω 2 . The horizontal axis of the bicoher-ence corresponds to ω 1 , andthe vertical to ω 2 .The srcin is in the center, and the axis rangefrom [ − π,π ] . A signal of length 4096 with random amplitudeand phase is divided into N  = 128 overlappingsegments of length 64 each. Shown in the toprow of Figure 2is the estimatedpower spectrumandthebicoherence estimatedasspecified inEqua-tion (12). Shown below is the same signal where ω 3 = ω 1 + ω 2 has been coupled to ω 1 and ω 2 .That is, ω 3 has amplitude a 3 = a 1 · a 2 andphase φ 3 = φ 1 + φ 2 . Note that the remain-ing frequency content of the signal remains un-changed, butthatthebicoherence issignificantlymore active (increasing from 0.08 to 0.20) at the bi-frequency ω 1 ,ω 2 , as seen by the peaks in Fig-ure 2. The multiple peaks aredue totheinherentsymmetries in the bicoherence.As a measure of overall correlations, the bico-herence can be averaged across all frequencies: 1 N  2 N/ 2  ω 1 = − N/ 2 N/ 2  ω 2 = − N/ 2 ˆ b  2 πω 1 N , 2 πω 2 N   . (13)This quantity is employed throughout this pa-per as a measure of higher-order correlations. 3 LensDistortionsandCorrelations Shownin Figure 3is a 1-Dsignal f  u ( x ) , oflength4096, with a 1 /ω power spectrum and randomphase. Also shown is the log of its normalizedpowerspectrum P  ( w ) anditsbicoherence ˆ b ( ω 1 ,ω 2 ) .The bicoherence was estimated from 128 over-lapping segments each of length 64 each. Alsoshownin Figure3isthesamesignal passedthrougha 1-D version of the lens distortion, f  d ( x ) , givenin Equation (1): f  d ( x ) = f  u ( x (1 + κx 2 )) (14)where κ controls the amount of distortion. No-tice that while the distortion leaves the powerspectrum largely unchanged there is a signifi-cant increase in the bispectral response: the bi-coherenceaveragedacross allfrequencies, Equa-tion (13), nearly doubles from 0.08 to 0.14. Thisexample illustrates that when an arbitrary sig-nal is exposed to a geometric non-linearity, cor-relations between triples of harmonics are intro-duced.For our purposes, what remains to be shownis that these correlations are proportional to theamount of distortion, κ . To illustrate this rela-tionship a 1-D signal f  u ( x ) is subjected to a fullrange of distortions as in Equation (14). ShowninFigure 4istheaveragebicoherence, Equation(13),plottedasa function of theamount of distortion.Notice that this function has a single minimumat κ = 0 , i.e., no distortion.Theseobservations leadto asimple algorithmforblindly removing lens distortions. Beginningwith a distorted signal:1. select a range of possible κ values,4  f  u ( x ) f  d ( x ) Figure 3: Shown in theleft column is a fractalsignal, the log of its normalized power spec-trum and its bicoherence. Shown in the rightcolumn is a distorted version of the signal.While the distortion leaves the power spec-trum largely unchanged there is a significantincrease in the average bispectral response. 2. for each value of  κ apply the inverse dis-tortion to f  d yielding a provisional undis-torted image f  κ ,3. compute the bicoherence of  f  κ ,4. select the value of  κ that minimizes the bi-cohereneceaveragedacrossallfrequencies.5. remove the distortion according to the in-verse distortion modelThis basic algorithm extends naturally to 2-Dimages. However in order to avoid the memoryand computational demands of computing an −0.6 −0.4 −0.2 0 0.2 0.400. ( κ  )         b        i      c      o        h      e      r      e      n      c      e Figure4: Shownisthebicoherence computedfora range of lens distortion( κ ). The bicoher-ence is minimal when κ = 0 , i.e., no distor-tion. image’sfull 4-D bicoherence, welimit our analy-sis to one-dimensional radial slices through thecenter of the image. This is reasonable assum-ing a radially symmetric distortion and that thedistortion emanates from the center of the im-age. If the image center drifts, then a more com-plexthree-parameterminimization wouldbe re-quired to jointly determinethe image center andamount of distortion. The amount of distortionfor an image is then estimated by averaging overtheestimatesfromasubsetofradialslices (e.g., ev-ery 10 degrees), as described above.In the results that follow in the next section,we assume a one-parameter radially symmetricdistortion model. Denoting the desired undis-torted image as f  u ( x,y ) , the distorted image isdenoted as f  d (˜ x, ˜ y ) , where ˜ x = x (1 + κr 2 ) and ˜ y = y (1 + κr 2 ) , (15)and r 2 = x 2 + y 2 , and κ controls the amountof distortion. Given an estimate of the distor-tion, the image is undistorted by solving Equa-tion (15) for the srcinal spatial coordinates x and y , andwarping thedistortedimageontothissampling lattice. Solving for the srcinal spatial5
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!