A variancebased sensitivity index function for factor prioritization
Douglas L. Allaire
, Karen E. Willcox
Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
a r t i c l e i n f o
Article history:
Received 15 October 2010Received in revised form15 August 2011Accepted 22 August 2011Available online 3 September 2011
Keywords:
Global sensitivity analysisFactor prioritizationVariance reductionRejection sampling
a b s t r a c t
Among the many uses for sensitivity analysis is factor prioritization
—
that is, the determination of which factor, once ﬁxed to its true value, on average leads to the greatest reduction in the variance of anoutput. A key assumption is that a given factor can, through further research, be ﬁxed to some point onits domain. In general, this is an optimistic assumption, which can lead to inappropriate resourceallocation. This research develops an srcinal method that apportions output variance as a function of the amount of variance reduction that can be achieved for a particular factor. This variancebasedsensitivity index function provides a main effect sensitivity index for a given factor as a function of theamount of variance of that factor that can be reduced. An aggregate measure of which factors would onaverage cause the greatest reduction in output variance given future research is also deﬁned andassumes the portion of a particular factors variance that can be reduced is a random variable. Anaverage main effect sensitivity index is then calculated by taking the mean of the variancebasedsensitivity index function. A key aspect of the method is that the analysis is performed directly on thesamples that were generated during a global sensitivity analysis using rejection sampling. The methodis demonstrated on the Ishigami function and an additive function, where the rankings for futureresearch are shown to be different than those of a traditional global sensitivity analysis.
&
2011 Elsevier Ltd. All rights reserved.
1. Introduction
Sensitivity analysis of model output has been deﬁned as thedetermination of how uncertainty in the output of a model can beapportioned to different sources of uncertainty in the modelfactors [1]. Sensitivity analysis deﬁned in this manner is oftenreferred toas
global sensitivity analysis
, owing to the fact that entirefactor distributions are considered in the apportionment process.Since what is meant by the term ‘‘uncertainty’’ is typically casedependent, several indicators have been developed to apportiondifferent measures of uncertainty among model factors. Theseindicators are often based on screening methods [2], variancebased methods [3–6], entropybased methods [6,7], nonpara
metric methods [6,8], and momentindependent approaches
[9–11]. This paper focuses on the development and demonstration
of an extension of traditional variancebased global sensitivityanalysis that considers the change in output variance caused by achange in factor variance that may arise from researching a factorfurther. In general, variancebased global sensitivity analysis is thestandard practice for determining how each factor contributes tooutput uncertainty when output variance is considered sufﬁcientto describe output variability [4,5].
Variancebased global sensitivity analysis is a rigorous methodfor apportioning output variance [3,12]. The method has been
applied in a wide variety of applications including hydraulicmodeling [13], aviation environmental modeling [14], nuclear
waste disposal [4], robust mechanical design practices [15], and
many others. The two main metrics computed in variancebasedglobal sensitivity analysis are the main effect sensitivity indicesproposed by Sobol’ [16] and the total effect sensitivity indicesproposed by Homma and Saltelli [5]. One of the primary uses of global sensitivity analysis is in the context of
factor prioritization
[3]. In this setting, the objective is to determine which factor,on average, once ﬁxed to its true value, will lead to the greatestreduction in output variance. It has been established by Saltelliet al. [3] and Oakley and O’Hagan [17] that the main effect
sensitivity indices are appropriate measures for ranking factorsin this setting, however, as noted in Oakley and O’Hagan [17], it israrely possible to learn the true value of any uncertain factor, andthus these sensitivity indices only suggest the potential forreducing uncertainty in an output through new research on afactor. Given that it is rarely possible to obtain the true value of any uncertain factor, the assumption that a given factor will beﬁxed to some point on its domain is a major limitation in the useof main effect sensitivity indices for use in allocating resourcesaimed at reducing output variance.To account for the inherent limitations in using global sensitivity analysis results for directing future research, a new method
Contents lists available at SciVerse ScienceDirectjournal homepage: www.elsevier.com/locate/ress
Reliability Engineering and System Safety
09518320/$see front matter
&
2011 Elsevier Ltd. All rights reserved.doi:10.1016/j.ress.2011.08.007
Corresponding author.
Email address:
dallaire@mit.edu (D.L. Allaire).Reliability Engineering and System Safety 107 (2012) 107–114
that apportions output variance as a function of the amount of variance reduction that can be achieved for a particular factor hasbeen developed. This function is called the variancebased sensitivity index function. By assuming the portion of a particularfactor’s variance that can be reduced is a random variable, themean of this function can be taken to provide average main effectsensitivity indices for ranking purposes. A key aspect of themethod is that the analysis is performed directly on the factorand output samples that were generated during a global sensitivity analysis using a rejection sampling technique for MonteCarlo simulation proposed in Beckman and McKay [18]. Thederivation of the method is given in Section 2, which is followedby a demonstration of the method on the wellknown Ishigamifunction [19] and a purely additive function in Section 3. Conclu
sions are presented in Section 4.
2. Methodology
In the following subsections, the method is derived using maineffect sensitivity indices from global sensitivity analysis. Thenotion of a reasonable distribution that could arise through futureresearch on a given factor is considered as well as the rejectionsampling technique and how it can be employed to reuse modelevaluations from a Monte Carlo simulation.
2.1. Derivation
Consider a generic model,
Y
¼
f
ð
x
Þ
, where
x
¼ ½
X
1
,
. . .
,
X
k
T
, and
X
1
,
. . .
,
X
k
are random variables on the measurable space
ð
R
,
B
Þ
,and
f
:
R
k

R
is
ð
B
k
,
B
Þ
measurable. Then
Y
is a random variable,and by deﬁnition, the variance of
Y
can be decomposed accordingtovar
ð
Y
Þ ¼
E
½
var
ð
Y
9
X
i
Þþ
var
ð
E
½
Y
9
X
i
Þ
,
ð
1
Þ
for any
X
i
, where
i
A
f
1
,
. . .
,
k
g
. According to Saltelli et al. [3], thegoal of factor prioritization is the identiﬁcation of which factor,once ﬁxed at its true value, would reduce the variance of
Y
themost. Since it is not known
a priori
a given factor’s true value,factor prioritization is carried out by identifying the factorswhich, on average, once ﬁxed, would cause the greatest reductionin the variance of
Y
. The average amount of variance remainingonce a given factor is ﬁxed is just
E
½
var
ð
Y
9
X
i
Þ
for any factor
X
i
.Thus, according to Eq. (1), the average amount of the variance of
Y
that could be reduced through ﬁxing factor
X
i
somewhere on itsdomain is var
ð
E
½
Y
9
X
i
Þ
. Global sensitivity analysis uses this fact forfactor prioritization by considering main effect sensitivity indices,which take the form
S
i
¼
var
ð
E
½
Y
9
X
i
Þ
var
ð
Y
Þ
,
ð
2
Þ
where
S
i
is the main effect sensitivity index of factor
X
i
. The maineffect sensitivity index can then be used as a measure of theproportion of the variance of
Y
that is expected to be reduced oncefactor
X
i
is ﬁxed to its true value.The calculation of main effect sensitivity indices in a globalsensitivity analysis is most commonly done using either theFourier Amplitude Sensitivity Test (FAST) method or the Sobol’method [5,12,20,21]. The FAST method is based on Fourier trans
forms, while the Sobol’ method utilizes Monte Carlo simulation.The Sobol’ method is employed in this work.The Sobol’ method is welldeveloped and in wide use in thesensitivity analysis ﬁeld. Following Homma and Saltelli [5], themain effect sensitivity indices may be estimated via the Sobol’method for a given factor
X
i
by ﬁrst estimating the mean
f
0
of thefunction
Y
¼
f
ð
x
Þ
as
^
f
0
¼
1
N
X
N m
¼
1
f
ð
x
m
Þ
,
ð
3
Þ
where
x
m
represents the
m
th realization of the random vector
½
X
1
,
. . .
,
X
k
T
and
N
denotes the total number of realizations of therandom vector. Then estimating the variance of the function as
^
V
¼
1
N
X
N m
¼
1
f
ð
x
m
Þ
2
^
f
20
,
ð
4
Þ
and the singlefactor partial variances as
^
V
i
¼
1
N
X
N m
¼
1
f
ð½
x
m
1
,
. . .
,
x
mi
,
. . .
,
x
mk
T
Þ
f
ð½
~
x
m
1
,
. . .
,
x
mi
,
. . .
,
~
x
mk
T
Þ
^
f
20
,
ð
5
Þ
where
x
jm
and
~
x
m j
denote different samples of factor
X
j
and thepartial variances can be computed for
i
A
f
1
,
. . .
,
k
g
. The main effectsensitivity index for factor
X
i
can then be computed according to
^
S
i
¼
^
V
i
^
V
:
ð
6
Þ
Here it should be noted that improvements in estimating maineffect indices using samplingbased methods have been developed by Saltelli et al. [22] and using regression or emulatorbasedmethods by Lewandowski et al. [23], Oakley and O’Hagan [17],
Tarantola et al. [24], Ratto et al. [25], Storlie and Helton [26]. Our
focus in this work is on samplingbased approaches, which arecommonly used in situations where both main effect and totaleffect indices are desired [22]. The methodology developed in thispaper can readily be applied to the samplingbased techniques of Saltelli et al. [22], however, the development is more accessible inthe context of the traditional Sobol’ method. Adapting the work of this paper to regression and emulatorbased methods is a topicfor future work.As noted previously, these main effect sensitivity indices maybe used for factor prioritization by ranking inputs according totheir main effect indices, which give the percentage of how muchoutput variability can be expected to be eliminated by ﬁxing aparticular input somewhere on its domain. However, this use of global sensitivity analysis for factor prioritization relies on theassumption that a given factor, through future research, can beﬁxed to some point on its domain. The key contribution of thiswork is to relax that assumption by considering the amount of variance that can be reduced for a given factor as a randomvariable rather than assuming the variance to be completelyreducible. More precisely, we assume that for a given amount of variance reduction for a factor
X
i
, there is a corresponding familyof allowable distributions, and we calculate an average change inthe variance of the model output over this family.Let
X
io
be the random variable deﬁned by the srcinal distribution for some factor
X
i
, and
X
0
i
be the random variable deﬁned by anew distribution for factor
X
i
after some further research has beendone.
X
io
and
X
0
i
have corresponding main effect sensitivity indices
S
io
and
S
0
i
respectively. Then we can deﬁne the ratio of the varianceof factor
X
i
that is not reduced and the total variance of thesrcinal distribution of factor
X
i
as
l
i
¼
var
ð
X
0
i
Þ
=
var
ð
X
oi
Þ
. Assumingfurther research reduces the variance of factor
X
i
,
1
it is clear that
l
i
A
½
0
,
1
. Since it cannot be known in advance how much variancereduction for a given factor is possible through further research,
l
i
is cast as a uniform random variable
L
i
on [0,1], which corresponds to a maximum entropy distribution given that all weknow is the interval in which
l
i
will take a value [27].
1
It is possible that further research could increase the variance of a factor,however, this would suggest that the srcinal characterization of uncertainty wasﬂawed.
D.L. Allaire, K.E. Willcox / Reliability Engineering and System Safety 107 (2012) 107–114
108
Given that the variance of factor
X
i
that may be reduced is arandom percentage, 100
ð
1
L
i
Þ
%
, of the total srcinal variance of factor
X
i
, the variancebased sensitivity index function can bedeﬁned as
z
i
ð
l
i
Þ ¼
var
ð
Y
o
Þ
S
oi
E
½
var
ð
Y
0
Þ
S
0
i
9
L
i
¼
l
i
var
ð
Y
o
Þ
,
ð
7
Þ
where
S
io
is the srcinal main effect sensitivity index of factor
X
i
,var
ð
Y
o
Þ
is the srcinal output variance, and
E
½
var
ð
Y
0
Þ
S
0
i
9
L
i
¼
l
i
isthe expected value of the product of the variance of the outputand the main effect global sensitivity index of factor
X
i
taken overall
reasonable
distributions of factor
X
i
with 100
l
i
%
of thevariance of the srcinal distribution for factor
X
i
. The reasonabledistributions are a prespeciﬁed, parameterized family of distributions. These distributions are discussed further in the followingsubsection.The variancebased sensitivity index function given by Eq. (7)provides the main effect sensitivity index for factor
X
i
if it isknown that exactly 100
ð
1
l
i
Þ
%
of the factor’s variance can bereduced. This can be seen by noting that var
ð
Y
o
Þ
S
oi
is the expectedvalue of the variance of
Y
o
that is due to factor
X
i
, and var
ð
Y
0
Þ
S
0
i
isthe expected value of the variance of
Y
0
that is due to factor
X
i
after 100
ð
1
l
i
Þ
%
of factor
X
i
’s variance has been reduced. Sincethere are many ways to reduce the variance of factor
X
i
by100
ð
1
l
i
Þ
%
, the expected value of var
ð
Y
0
Þ
S
0
i
is taken over all thereasonable distributions for which 100
ð
1
l
i
Þ
%
has been reduced.Thus, var
ð
Y
o
Þ
S
oi
E
½
var
ð
Y
0
Þ
S
0
i
9
L
i
¼
l
i
is the amount of variance in
Y
o
that cannot be reduced further if factor
X
i
’s variance can only bereduced by 100
ð
1
l
i
Þ
%
.If it is assumed that all of the variance of a particular factor canbe reduced, then
l
i
¼
0, and for a given factor
X
i
, this means that
E
½
var
ð
Y
0
Þ
S
0
i
9
L
i
¼
0
¼
0, since once all of the variance of factor
X
i
hasbeen reduced, factor
X
i
will simply become a constant, and thus,
S
0
i
¼
0. Therefore, when
l
i
¼
0,
z
i
ð
0
Þ ¼
S
oi
, and the index reduces tothe speciﬁc case of global sensitivity analysis. However, as notedpreviously, since it is not likely known what value
l
i
will take priorto further research on a given factor,
l
i
is considered to be auniform random variable,
L
i
, on the interval
½
0
,
1
.The expected value of
z
i
ð
L
i
Þ
can thus be taken to give an
average main effect sensitivity index
(
Z
), as shown in Eq. (8) forsome factor
X
i
,
Z
i
¼
E
L
i
½
z
i
ð
L
i
Þ
:
ð
8
Þ
The average main effect sensitivity index for each factor in amodel is then an index that can be used to quantitatively rankfactors based on the average amount of output variance that canbe reduced when further research is done on a particular factor.
2.2. Deﬁning reasonable distributions
In the discussion of the development of the variancebasedsensitivity index function it was noted that reasonable new factordistributions, which represent the result of further research on afactor, be used in the estimation of the function. This is becausegiven some initial distribution for a factor and some
l
i
, there willgenerally not be a unique new distribution with 100
l
i
%
of thevariance of the srcinal factor distribution. For example, if a factorhas an srcinal distribution that is uniform on the interval [0,1],and
l
i
¼
0
:
5, there are an inﬁnite number of new distributions,such as
U
½
0
,
ﬃﬃﬃ
2
p
=
2
,
U
½
1
ﬃﬃﬃ
2
p
=
2
,
1
,
U
½
ﬃﬃﬃ
2
p
=
4
,
1
ﬃﬃﬃ
2
p
=
4
, etc., that allhave variances equal to
l
i
times the srcinal variance. The newdistributions could also be from a different family of distributions,such as triangular. Therefore, a set of reasonable distributionswith 100
l
i
%
of the variance of any given srcinal distributionmust be deﬁned. Here we present a procedure for identifyingreasonable distributions for factors that are srcinally uniformlydistributed or normally distributed. These distributions tend to beused in the absence of full distributional information and thus areoften candidates for future research. In both cases, it is assumedthat future research will only impact the parameters of a givendistribution, thus the impact of future research that could lead toa change in the underlying distribution family (e.g. from auniform distribution to a triangular distribution) is not consideredhere. However, if the distribution family of a given factor wereexpected to change through further research, then reasonabledistributions from the new family, given that the srcinal distribution was from another family, could be deﬁned.
2.2.1. Uniform distributions that may arise through future research
Consider an arbitrary uniform distribution,
U
½
a
,
b
. The varianceof this distribution is given as var
ð
X
Þ ¼ ð
b
a
Þ
2
=
12. Thus,
l
i
for thisfamily of distributions can be written as
l
i
¼ ð
b
0
a
0
Þ
=
ð
b
o
a
o
Þ
2
,
where
a
0
and
b
0
are the endpoints of a new distribution and
a
o
and
b
o
are the endpoints of the srcinal distribution. In this case, agiven
l
i
implies all new uniform distributions are intervals of thesame width, which is
l
1
=
2
i
ð
b
o
a
o
Þ
. A reasonable method forsampling from the set of intervals on
½
a
o
,
b
o
with width
l
1
=
2
i
ð
b
o
a
o
Þ
is given in Algorithm 1.
Algorithm 1.
Sampling uniform distributions.1: Sample
l
i
from a uniform distribution on the interval [0,1].2: Sample
b
0
from a uniform distribution on the interval
½
a
o
þ
l
1
=
2
i
ð
b
o
a
o
Þ
,
b
o
.3:Let
a
0
¼
b
0
l
1
=
2
i
ð
b
o
a
o
Þ
.This method of sampling ensures the new parameters,
a
0
and
b
0
, for a given
l
i
, will be such that
a
0
U
½
a
o
,
b
o
l
1
=
2
i
ð
b
o
a
o
Þ
, and
b
0
U
½
a
o
þ
l
1
=
2
i
ð
b
o
a
o
Þ
,
b
o
. Thus the set of possible uniform distributions with
ð
1
l
i
Þ
the variance of the srcinal distribution issampled uniformly.
2.2.2. Normal distributions that may arise through future research
Consider an arbitrary normal distribution,
N
ð
m
o
,
s
2
o
Þ
, where
m
o
is the mean and
s
2
o
is the variance of the distribution. For thenormal family of distributions,
l
i
is written as
l
i
¼
s
0
2
=
s
2
o
, where
s
0
2
is the variance of a new distribution and
s
2
o
is the srcinalvariance. Here a procedure is presented where the mean value of the srcinal distribution is also the mean value of any newdistributions after further research has been undertaken. However, if the mean value is expected to change, other procedurescan be developed to take that into account. Given that here themean does not change, a speciﬁc
l
i
uniquely deﬁnes a newdistribution
N
ð
m
o
,
l
i
s
2
o
Þ
, where
m
o
is the mean of the srcinaldistribution. Thus, the proposed procedure for sampling normaldistributions is simply selecting a
l
i
and using that
l
i
to calculatethe variance of the new distribution. This procedure is given inAlgorithm 2.
Algorithm 2.
Sampling normal distributions.1: Set
m
0
¼
m
o
.2: Sample
l
i
from a uniform distribution on the interval [0,1].3: Set
s
0
2
¼
l
i
s
2
o
.
2.3. Rejection sampling
The evaluation of Eq. (7) and subsequently of Eq. (8) requiresconsideration of a large number of different distributions for eachfactor. If a global sensitivity analysis is carried out for each newdistribution for each factor, the computational expense would bemassive and estimating values of the variancebased sensitivity
D.L. Allaire, K.E. Willcox / Reliability Engineering and System Safety 107 (2012) 107–114
109
index function would likely be too costly to ever carry out.However, if a global sensitivity analysis with the srcinal distributions for each factor is completed, rejection sampling can beused to estimate the values of the variancebased sensitivityindex function without any further model evaluations.Rejection sampling is a method for generating samples from adesired distribution by sampling from a different distribution. Themethod has been developed for both random samples [28] andquasirandom samples such as low discrepancy sequences [29].For random samples, following DeGroot and Schervish [28], let
f
Z
(
z
) be a probability density function of a desired distribution forsome random variable,
Z
. Let
f
X
(
x
) be some other probabilitydensity function for a random variable,
X
, with the property thatthere exists a constant,
k
, such that
kf
X
ð
x
Þ
Z
f
Z
ð
x
Þ
for all
x
, where
x
is a realization of
X
. The rejection method can then be used togenerate
J
samples from
f
Z
(
z
) as shown in Algorithm 3. For lowdiscrepancy sequences, following Wang [29], a similar procedurefor sampling from
Z
can be developed as shown in Algorithm 4.Both of these algorithms are rigorous techniques for performingrejection sampling on nested uniform distributions. However, inpractice, we may use the bounds of the desired new interval to dorejection sampling directly by rejecting all points outside thebounds.
Algorithm 3.
Rejection sampling for 1D random samples.1: Draw a sample,
x
, from
X
.2: Draw a sample,
u
, from a uniform random variable on
½
0
,
1
.3: If
f
Z
ð
x
Þ
=
f
X
ð
x
Þ
Z
ku
let
z
j
¼
x
,
j
¼
j
þ
1
,
If
j
¼
J
,
STOP.Else return to 1.4: Else, discard
x
and
u
and return to 1.
Algorithm 4.
Rejection sampling for 1D low discrepancysequences.1: Generate a low discrepancy sequence of points
ð
x
i
,
n
i
Þ
A
½
0
,
1
2
,
i
¼
1
,
2
,
. . .
,
M
.2: Map
x
i
to
x
i
for all
i
(e.g., using inverse CDF method).3: Let
u
i
¼
n
i
.4: For
i
¼
1 to
M
If
f
Z
ð
x
i
Þ
=
f
X
ð
x
i
Þ
Z
ku
i
,
let
z
j
¼
x
i
,
j
¼
j
þ
1
,
Else reject
x
i
.As an example of how rejection sampling is used in this work,consider a function
Y
¼
f
ð
X
1
,
X
2
Þ
, where
X
1
,
X
2
, and
Y
are randomvariables and
X
1
,
X
2
U
½
0
,
1
. Suppose we care about some quantity
Q
ð
Y
Þ
(e.g. an integral, a mean, a sensitivity index, etc.). Toevaluate
Q
ð
Y
Þ
, we ﬁrst sample from the random variables
X
1
and
X
2
using a Sobol’ quasirandom sequence [30]. The points for asequence of size 1024 are shown in Fig. 1a.Following Beckman and McKay [18], if we would now like to
evaluate
Q
ð
Y
Þ
with a different distribution on say
X
1
, we can do soby reusing previous samples of
X
1
and
X
2
, and hence samples of
Y
,by performing the appropriate rejection sampling on
X
1
. The use of rejection sampling for this task only requires that the support of the new distribution of
X
1
be contained within the support of thesrcinal distribution of
X
1
and the existence of a uniform bound
k
as in Algorithms 3 and 4. For example, if we would like to evaluate
Q
ð
Y
Þ
, where
Y
¼
f
ð
X
1
,
X
2
Þ
and
X
1
U
½
1
=
5
,
3
=
5
and
X
2
U
½
0
,
1
, wecould do so without reevaluating
Y
by using the evaluations of
Y
associated with the samples of
X
1
that have been accepted inrejection sampling as samples of the new distribution of
X
1
. Thesesamples are shown Fig. 1b, which presents the srcinal samples of
X
1
and
X
2
and the accepted samples after rejection sampling isapplied to
X
1
. It should be noted here that this use of rejectionsampling for the case of normal distributions in which the meanmay change poses computational difﬁculties due to the potentialsmall number of previous function evaluations near the new meanof the distribution. This difﬁculty may also arise if we wish to userejection sampling on a small subset of the srcinal interval in thecase of both uniform and normal distributions.
2.4. Application of rejection sampling to global sensitivity analysis
Rejection sampling can be employed to reuse the results froma global sensitivity analysis to calculate values of the variancebased sensitivity index functions and average main effect sensitivity indices as follows. Consider again a generic model,
Y
¼
f
ð
x
Þ
,where
x
¼ ½
X
1
,
. . .
,
X
k
T
, and
X
1
,
. . .
,
X
k
and
Y
are random variables.Suppose we have conducted a global sensitivity analysis tocalculate the main effect sensitivity index of factor
X
i
using the
0 0.2 0.4 0.6 0.8 100.20.40.60.81
x
1
x
1
x
2
x
2
0 0.2 0.4 0.6 0.8 100.20.40.60.81Original Accepted
Fig. 1.
(a) Original set of 1024 quasirandom Sobol’ points for
X
1
and
X
2
, which are used to calculate samples of
Y
. (b) Accepted samples of
X
1
and
X
2
after rejectionsampling has been performed on
X
1
. Originally
X
1
U
½
0
,
1
and is now
X
1
U
½
1
=
5
,
3
=
5
. The srcinal samples are presented for comparison.
D.L. Allaire, K.E. Willcox / Reliability Engineering and System Safety 107 (2012) 107–114
110
Sobol’ method [5] and quasirandom Sobol’ points. Thus, we havethe model evaluations corresponding to
f
ð½
x
m
1
,
. . .
,
x
mi
,
. . .
,
x
mk
T
Þ
and
f
ð½
~
x
m
1
,
. . .
,
x
mi
,
. . .
,
~
x
mk
T
Þ
, as in Eq. (5), where
m
¼
1
,
. . .
,
N
, where
N
isthe number of samples of each factor. Consider some newdistribution,
f
X
0
i
ð
x
0
Þ
for factor
X
i
, with srcinal distribution
f
X
oi
ð
x
i
Þ
,for which we would like to determine the value of the variancebased sensitivity index function for a particular amount of variance reduction in
X
i
. To do this, we must estimate the maineffect sensitivity index,
S
0
i
, and the new variance of the output,var
ð
Y
0
Þ
, for use in the estimation of a variancebased sensitivityindex function value for factor
X
i
given by Eq. (7). The estimationof these quantities using the function evaluations from globalsensitivity analysis can be achieved as shown in Algorithm 5.
Algorithm 5.
Rejection sampling for the Sobol’ method forfactor
X
i
.1: Use Algorithm 3 or 4 to choose a subsample of the srcinalpoints.2: Calculate the new variance of the output var
ð
Y
0
Þ
usingEq. (4) and the output samples associated with thesubsample from Step 1.3: Calculate the singlefactor partial variance for
X
i
using thesamples from Step 1 and Eq. (5).4: Calculate the new Sobol’ main effect sensitivity index
S
0
i
using Eq. (6) with the variance from Step 2 and the partialvariance from Step 3.The values of the variancebased sensitivity index function forfactor
X
i
may then be calculated as shown in Algorithm 6.
Algorithm 6.
Evaluating
z
i
ð
l
i
Þ
for factor
X
i
.1: Estimate the srcinal output variance var
ð
Y
o
Þ
.2: Perform a global sensitivity analysis for factor
X
i
toestimate the srcinal main effect sensitivity index
S
io
.3: Use Algorithm 5 to estimate var
ð
Y
0
Þ
and
S
0
i
. Repeat over allreasonable distributions for factor
X
i
with 100
l
i
%
of thevariance of the srcinal distribution for factor
X
i
.4: Calculate
z
i
ð
l
i
Þ
using Eq. (7) with the quantities from Steps2 and 3.Given the variancebased sensitivity index function for factor
X
i
, we may estimate the average main effect sensitivity index forfactor
X
i
as shown in Algorithm 7.
Algorithm 7.
Evaluating
Z
i
for factor
X
i
.1: Discretize the interval [0,1] and estimate
z
i
ð
l
i
Þ
at thediscretization points (e.g., 0, 0.1, 0.2,
y
,1.0) usingAlgorithm 6.2: Estimate the average main effect sensitivity index
Z
i
usingEq. (8) with the values of
z
i
from Step 1.
3. Test function analysis
To demonstrate the methodology developed in Section 2, theapproach is applied here to the Ishigami function and an additivefunction. The Ishigami function was ﬁrst introduced by Ishigamiand Homma [19]. It is commonly used to test sensitivity anduncertainty analysis techniques. For example, it was used byRatto et al. [31] to demonstrate the use of statedependentparameter modeling in the estimation of conditional momentsfor sensitivity analysis, by Homma and Saltelli [5] to demonstratethe performance of importance measures for sensitivity analysis,by Saltelli et al. [32] to demonstrate the calculation of highdimensional model representation [16] for use in variancebasedsensitivity analysis, by Eldred and Swiler [33] to explore reﬁnement approaches for nonintrusive polynomial chaos expansionand stochastic collocation for uncertainty quantiﬁcation techniques, and by Storlie et al. [34] to investigate the use of metamodels and bootstrap conﬁdence intervals for sensitivity analysis of computationally demanding models. The second example, anadditive function, was created for this work to fully demonstratethe beneﬁts of a variancebased sensitivity index function and toprovide an example that does not contain strong interactions asthe Ishigami function does.
3.1. Ishigami function
The Ishigami function is given as follows:
Y
¼
sin
X
1
þ
a
sin
2
X
2
þ
bX
43
sin
X
1
,
ð
9
Þ
where the
X
i
are independent and uniformly distributed on
½
p
,
p
. The constants are set as
a
¼
5 and
b
¼
0.1 as in Rattoet al. [31]. A global sensitivity analysis was carried out using the
Sobol’ method and a Sobol’ quasirandom sequence of size 4096.The computed main effect sensitivity indices for each factor are
S
1
¼
0.40,
S
2
¼
0
:
28 and
S
3
¼
0.00. For factor prioritization purposesthen, the conclusion that is drawn from the analysis is to focusfuture research efforts on factor
X
1
, since on average once ﬁxed,factor
X
1
is expected to reduce the variance of
Y
by the largestamount.If the analysis for factor prioritization were to be concluded atthis point, a great deal of information regarding the impact of future research on factors
X
1
and
X
2
is missed and an inappropriate decision regarding how to allocate future resources forreducing the variance of
Y
may be made. It should be noted herethat factor
X
3
is not considered further because it does not have amain effect and thus only affects the variance of
Y
throughinteractions with the other factors. Thus, to demonstrate thebeneﬁts of the variancebased sensitivity index function, ananalysis for factors
X
1
and
X
2
was carried out following themethodology presented in Section 2.The results of the analysis are presented in Fig. 2a. Indices of each factor for values of
l
i
, that is the variance that cannot bereduced, for
l
i
¼
0
:
0
,
0
:
05
,
0
:
10
,
. . .
,
1
:
0 are provided. Withoutaggregating the results, the ﬁgure shows that the effects of futureresearch on a given factor on the variance of
Y
is highly nonlinear.Thus, depending on the expected returns of future research on afactor, the main effect global sensitivity indices, which are therightmost points on the ﬁgure, could be misleading if used fordetermining how to best allocate resources aimed at reducing thevariance of
Y
. As an extreme example of how misleading theinformation from the main effect indices computed via globalsensitivity analysis could be, consider directing future research atfactor
X
1
(which is supported by factor prioritization analysis) andachieving a 25% reduction in the variance of the factor (thus1
l
1
¼
0.25). As can be seen in Fig. 2a, on average, a 25% reductionin the variance of
X
1
will actually lead to an
increase
in thevariance of
Y
, and thus not have been an appropriate use of resources. In fact, according to the results of the distributionalsensitivity analysis, unless it is believed that future research willlead to a reduction in the variance of
X
1
of more than 50%, it doesnot make sense to direct research at factor
X
1
at all if the goal is toreduce the variance of
Y
. This behavior can be explained byconsidering the Ishigami function output plotted against input
X
1
and
X
2
as shown in Fig. 3a and b respectively. In Fig. 3a, the
dark points are accepted samples for a 1
l
1
¼
0.25 case for
X
1
andthe light points are the rejected points from the full model. It isclear from the ﬁgure that the variance of the dark points is greaterthan the variance of the combination of the dark and light points,
D.L. Allaire, K.E. Willcox / Reliability Engineering and System Safety 107 (2012) 107–114
111