to appear:
Journal of the Optical Society of America A, 2001.
Blind Removal of Lens Distortion
Hany Farid and Alin C. PopescuDepartment of Computer ScienceDartmouth CollegeHanover NH 03755Virtually all imaging devices introduce some amount of geometric lens distortion. This paperpresents a technique for blindly removing these distortions in the absence of any calibrationinformation or explicit knowledge of the imaging device. The basic approach exploits thefact that lens distortion introduces speciﬁc higherorder correlations in the frequency domain.These correlations can be detected using tools from polyspectral analysis. The amount of distortion is then estimated by minimizing these correlations.1
1 Introduction
Virtually all medium to lowgrade imaging devices introduce some amount of geometric distortion. Thesedistortionsareoftendescribedwitha oneparameter radially symmetric model [2, 8,9]. Given an ideal undistorted image
f
u
(
x,y
)
,the distortedimage is denotedas
f
d
(˜
x,
˜
y
)
, wherethe distorted spatial parameters are given by:
˜
x
=
x
(1 +
κr
2
)
and
˜
y
=
y
(1 +
κr
2
)
,
(1)where
r
2
=
x
2
+
y
2
, and
κ
controls the amountof distortion. Shown in Figure 1 are the resultsof distorting a rectilinear grid with positive andnegative values of
κ
.While these distortions may be artistically interesting it is oftendesirable to remove these geometric distortions for many applications in image processing and computer vision (e.g., structure estimation, image mosaicing). The amountof distortion is typically determined experimentally by imaging acalibration targetwith knownﬁducial points. The deviation of these pointsfrom their srcinal positions is used to estimatethe amount of distortion (e.g., [9]). But oftensuch calibration is not available or direct accessto the imaging device is not possible, for example when downloading an image from the web.Inaddition, thedistortionparameterscanchangeas other imaging parameters are varied (e.g., focallengthorzoom),thus requiring repeatedcali brationforallpossible camerasettings. Analternative calibration technique relies on the presence of straight lines in the scene (e.g., [1, 7]).These lines, mapped to curves in the image dueto the distortion, are located or speciﬁed by theuser. The distortions are estimated by ﬁndingthemodelparametersthatmapthesecurvedlinesto straight lines. While this technique is moreﬂexible than those based on imaging a calibration target, it still relies on the scene containingextended straight lines.In this paper a technique is presented for estimating the amount of lens distortion in the absenceofanycalibrationinformationorscenecontent. The basic approach exploits the fact that
κ <
0
κ
= 0
κ >
0
Figure 1:
Oneparameter radially symmetriclens distortion, Equation (1).
lens distortion introduces speciﬁc higherordercorrelations in the frequency domain. These correlationscanbedetectedusing toolsfrompolyspectral analysis. The amount of distortion is thendeterminedbyminimizing thesecorrelations. These basic principles were used in a related paper inwhich we introduced a technique for the blindremoval of luminance nonlinearities [3].Insight is gained into the proposed technique by ﬁrst considering what effect a geometric distortion has on a onedimensional signal. Consider, for example, a pure sinusoid with amplitude
a
and frequency
b
:
f
u
(
x
) =
a
cos(
bx
)
.
(2)For purposes of exposition, consider a simpliﬁed version of the lens distortiongiven in Equation (1), where the spatial parameter is squared:
f
d
(
x
) =
a
cos(
bx
2
)
.
(3)This signal is composed of a multitude of harmonics. This canbeseenbyconsidering itsFouriertransform:
F
d
(
ω
) =
∞−∞
f
d
(
x
)
e
−
iωx
dx
= 2
∞
0
a
cos(
bx
2
)cos(
ωx
)
dx.
(4)Because the signal is symmetric (a cosine), theFourier integral may be expressed from
0
to
∞
andwithrespecttoonlythecosinebasis(i.e., thesine component of the complex exponential integrates to zero). This integral has a closed form2
solution [4] given by:
F
d
(
ω
) = 2
a
π
2
b
cos
ω
2
2
b
+ sin
ω
2
2
b
.
(5)Unlike the undistorted signal, with:
F
u
(
ω
) =
1

ω

=
b
0

ω

=
b
(6)theFouriertransformofthedistortedsignal contains a multitude of harmonics. Moreover, theamplitude andphase ofthese harmonics arecorrelated to the srcinal signal. Here the phasesaretriviallycorrelatedasallfrequencies arezerophase. Nevertheless,iftheinitialsignal consistedofmultiplefrequencies withnonzerophases, thenthe resulting distorted signal would have similar amplitude correlations and nontrivial phasecorrelations.In what follows we will show that this observation is not limited to the speciﬁc choice of signal or distortion. We will also show empiricallythat when an image is geometrically distorted,higherordercorrelations inthefrequency domainincrease proportional to the amount of distortion. As such, the amount of distortion can bedeterminedby simply minimizing these correlations. We ﬁrst show how toolsfrom polyspectralanalysis can be used to measure these higherorder correlations, and then show the efﬁcacy of this technique to the blind removal of lens distortion in synthetic and natural images.
2 BispectralAnalysis
Considerastochastic onedimensional signal
f
(
x
)
,and its Fourier transform:
F
(
ω
) =
∞
k
=
−∞
f
(
k
)
e
−
iωk
.
(7)It is commonpracticetouse the powerspectrumto estimate secondorder correlations:
P
(
ω
) =
E {
F
(
ω
)
F
∗
(
ω
)
}
,
(8)where
E{·}
is the expected value operator, and
∗
denotescomplexconjugate. Howeverthepowerspectrum is blind to higherorder correlations of thesortintroducedbyanonlinearity, Equation(1).Thesecorrelations canhoweverbeestimatedwithhigherorder spectra (see [6] for a thorough survey). Forexamplethebispectrumestimatesthirdorder correlations and is deﬁned as:
B
(
ω
1
,ω
2
) =
E {
F
(
ω
1
)
F
(
ω
2
)
F
∗
(
ω
1
+
ω
2
)
}
.
(9)Notethat unlike the powerspectrum the bispectrum of a real signal is complexvalued.The bispectrum reveals correlations betweenharmonically related frequencies, for example,
[
ω
1
,ω
1
,
2
ω
1
]
or
[
ω
1
,ω
2
,ω
1
+
ω
2
]
. If it is assumedthat the signal
f
(
x
)
is ergodic, then the bispectrum can be estimated by dividing
f
(
x
)
into
N
(possibly overlapping) segments, computing Fouriertransforms of each segment, and then averagingthe individual estimates:
ˆ
B
(
ω
1
,ω
2
) =1
N
N
k
=1
F
k
(
ω
1
)
F
k
(
ω
2
)
F
∗
k
(
ω
1
+
ω
2
)
,
(10)
where
F
k
(
·
)
denotes the Fourier transform of the
k
th
segment. This arithmetic averageestimatoris unbiased and of minimum variance. However, it has the undesired property that its variance at each bifrequency
(
ω
1
,ω
2
)
depends on
P
(
ω
1
)
,
P
(
ω
2
)
, and
P
(
ω
1
+
ω
2
)
(see e.g., [5]). Wedesire an estimator whose variance is independent of the bifrequency. To this end, we employthe bicoherence, a normalized bispectrum, deﬁned as:
b
2
(
ω
1
,ω
2
) =

B
(
ω
1
,ω
2
)

2
E{
F
(
ω
1
)
F
(
ω
2
)

2
}E{
F
(
ω
1
+
ω
2
)

2
}
.
(11)
Itisstraightforwardtoshowusing theSchwartzinequality thatthis quantity isguaranteedtohavevalues in the range
[0
,
1]
. As with the bispectrum, the bicoherence can be estimated as:
ˆ
b
(
ω
1
,ω
2
) =

1
N
k
F
k
(
ω
1
)
F
k
(
ω
2
)
F
∗
k
(
ω
1
+
ω
2
)

1
N
k

F
k
(
ω
1
)
F
k
(
ω
2
)

21
N
k

F
k
(
ω
1
+
ω
2
)

2
.
(12)
Note that the bicoherence is now a realvaluedquantity.Shown in Figure 2 is an example of the sensitivity of the bicoherence to higherorder correlations that are invisible to the power spectrum.3
power bicoherence
001
π
ω
1
ω
2
ω
3
001
π
ω
1
ω
2
ω
3
Figure 2:
Top: the normalized power spectrum and bicoherence for a signal with random amplitudes and phases. Bottom: thesamesignalwithone frequency,
ω
3
=
ω
1
+
ω
2
,whose amplitude and phase are correlated to
ω
1
and
ω
2
. The horizontal axis of the bicoherence corresponds to
ω
1
, andthe vertical to
ω
2
.The srcin is in the center, and the axis rangefrom
[
−
π,π
]
.
A signal of length 4096 with random amplitudeand phase is divided into
N
= 128
overlappingsegments of length 64 each. Shown in the toprow of Figure 2is the estimatedpower spectrumandthebicoherence estimatedasspeciﬁed inEquation (12). Shown below is the same signal where
ω
3
=
ω
1
+
ω
2
has been coupled to
ω
1
and
ω
2
.That is,
ω
3
has amplitude
a
3
=
a
1
·
a
2
andphase
φ
3
=
φ
1
+
φ
2
. Note that the remaining frequency content of the signal remains unchanged, butthatthebicoherence issigniﬁcantlymore active (increasing from 0.08 to 0.20) at the bifrequency
ω
1
,ω
2
, as seen by the peaks in Figure 2. The multiple peaks aredue totheinherentsymmetries in the bicoherence.As a measure of overall correlations, the bicoherence can be averaged across all frequencies:
1
N
2
N/
2
ω
1
=
−
N/
2
N/
2
ω
2
=
−
N/
2
ˆ
b
2
πω
1
N ,
2
πω
2
N
.
(13)This quantity is employed throughout this paper as a measure of higherorder correlations.
3 LensDistortionsandCorrelations
Shownin Figure 3is a 1Dsignal
f
u
(
x
)
, oflength4096, with a
1
/ω
power spectrum and randomphase. Also shown is the log of its normalizedpowerspectrum
P
(
w
)
anditsbicoherence
ˆ
b
(
ω
1
,ω
2
)
.The bicoherence was estimated from 128 overlapping segments each of length 64 each. Alsoshownin Figure3isthesamesignal passedthrougha 1D version of the lens distortion,
f
d
(
x
)
, givenin Equation (1):
f
d
(
x
) =
f
u
(
x
(1 +
κx
2
))
(14)where
κ
controls the amount of distortion. Notice that while the distortion leaves the powerspectrum largely unchanged there is a signiﬁcant increase in the bispectral response: the bicoherenceaveragedacross allfrequencies, Equation (13), nearly doubles from 0.08 to 0.14. Thisexample illustrates that when an arbitrary signal is exposed to a geometric nonlinearity, correlations between triples of harmonics are introduced.For our purposes, what remains to be shownis that these correlations are proportional to theamount of distortion,
κ
. To illustrate this relationship a 1D signal
f
u
(
x
)
is subjected to a fullrange of distortions as in Equation (14). ShowninFigure 4istheaveragebicoherence, Equation(13),plottedasa function of theamount of distortion.Notice that this function has a single minimumat
κ
= 0
, i.e., no distortion.Theseobservations leadto asimple algorithmforblindly removing lens distortions. Beginningwith a distorted signal:1. select a range of possible
κ
values,4
f
u
(
x
)
f
d
(
x
)
Figure 3:
Shown in theleft column is a fractalsignal, the log of its normalized power spectrum and its bicoherence. Shown in the rightcolumn is a distorted version of the signal.While the distortion leaves the power spectrum largely unchanged there is a signiﬁcantincrease in the average bispectral response.
2. for each value of
κ
apply the inverse distortion to
f
d
yielding a provisional undistorted image
f
κ
,3. compute the bicoherence of
f
κ
,4. select the value of
κ
that minimizes the bicohereneceaveragedacrossallfrequencies.5. remove the distortion according to the inverse distortion modelThis basic algorithm extends naturally to 2Dimages. However in order to avoid the memoryand computational demands of computing an
−0.6 −0.4 −0.2 0 0.2 0.400.10.20.30.40.5distortion (
κ
)
b i c o h e r e n c e
Figure4:
Shownisthebicoherence computedfora range of lens distortion(
κ
). The bicoherence is minimal when
κ
= 0
, i.e., no distortion.
image’sfull 4D bicoherence, welimit our analysis to onedimensional radial slices through thecenter of the image. This is reasonable assuming a radially symmetric distortion and that thedistortion emanates from the center of the image. If the image center drifts, then a more complexthreeparameterminimization wouldbe required to jointly determinethe image center andamount of distortion. The amount of distortionfor an
image
is then estimated by averaging overtheestimatesfromasubsetofradialslices (e.g., every 10 degrees), as described above.In the results that follow in the next section,we assume a oneparameter radially symmetricdistortion model. Denoting the desired undistorted image as
f
u
(
x,y
)
, the distorted image isdenoted as
f
d
(˜
x,
˜
y
)
, where
˜
x
=
x
(1 +
κr
2
)
and
˜
y
=
y
(1 +
κr
2
)
,
(15)and
r
2
=
x
2
+
y
2
, and
κ
controls the amountof distortion. Given an estimate of the distortion, the image is undistorted by solving Equation (15) for the srcinal spatial coordinates
x
and
y
, andwarping thedistortedimageontothissampling lattice. Solving for the srcinal spatial5