A model for R&D performance measurement
Valentina Lazzarotti
n
, Raffaella Manzini, Luca Mari
Universit
a Carlo Cattaneo
—
LIUC, Corso Matteotti, 22, 21053 Castellanza, Varese, Italy
a r t i c l e i n f o
Article history:
Received 10 November 2008Accepted 28 June 2011Available online 5 July 2011
Keywords:
R&D performance measurementMeasurement in soft systemsModels in measurementBalanced scorecard approach
a b s t r a c t
R&D activities are increasingly costly and risky and, as a consequence, measuring their performanceand contribution to value becomes critical. This paper illustrates a formal model for measuring R&Dperformance, based upon a balanced and synthetic evaluation of quantitative indicators from ﬁvedifferent perspectives of performance: ﬁnancial, customer, innovation and learning, internal business,alliances, and networks. The model is built in coherence with the suggestions coming from the theoryof measurement in soft systems, which gives relevant guidelines for ensuring validity, objectivity andintersubjectivity of the model. Then, an application in a real R&D setting is described, which helps tounderstand the model and to enlighten its main advantages and limits.
&
2011 Elsevier B.V. All rights reserved.
1. Introduction
Measuring performance and contribution to value of Research &Development (R&D) has become a fundamental concern for R&Dmanagers and executives in the last decades (KerssenvanDrongelen and Bilderbeek, 1999). Since the 1990s, several phenomena have encouraged the development and adoption of speciﬁcmethodologies and techniques for assessing the performance of R&D. First of all, the technological and competitive environment hasdramatically changed. Market arenas have become more turbulentand dynamic, with customer needs, competitors and businessmodels changing over time with a frequency much higher thanever (Wolf, 2006; Mohr et al., 2005). New knowledge has been
developed and applied to products and services faster and faster(Bayus, 1998; Wind and Mahajan, 1997). Consequently, lifecycles
have shortened in some product categories (Nevens et al., 1990), ahigher number of new products and services have been introducedover time and the time between subsequent innovations hasdecreased. Moreover, radical innovations have often come out fromthe conﬂuence of technologies belonging to traditionally separatedisciplines (Kodama, 1995) and the complexity, costs and variety of technical and scientiﬁc knowledge incorporated into products andservices have raised (Tidd et al., 2005).This paradigmatic change has signiﬁcantly ampliﬁed the importance of R&D to the ﬁrm’s competitiveness, especially in technologyintensive industries (Germeraad, 2001). However it is stillacknowledged that the measurement of R&D performance is achallenging task, since effort levels may not be easily observable,success is uncertain and inﬂuenced by uncontrollable factors, and itcan be usually assessed only after long delays. The deﬁnition itself of the property under measurement – ‘R&D performance’ – isusually loose and very context dependent. As a consequence, inthe last years many studies have been written aimed at discussingthe subject and suggesting possible approaches in the performancemeasurement, innovation and R&D management literature (Pappasand Remer, 1985; Brown and Svenson, 1988; Sivathanu and
Srinivasa, 1996; Werner and Souder, 1997; Hauser, 1998; Driva
and Pawar, 1999; Loch and Tapper, 2002; Godener and Soderquist,
2004; Ojanen and Vuola, 2006; JimenezZarco et al., 2006; Kunz,
2010; MolinaCastillo and MunueraAleman, 2009; Chiesa et al.,
2008, 2009; Bassani et al., 2010; Merschmann and Thonemann,
2011) and in the practitioners’ contribution as well. In spite of thehuge amount of work in the ﬁeld, the problem of deﬁning arigorous model for measuring R&D performance has not beensolved yet, although some notable and interesting attempts havebeen recently published (Tohumcu and Karasakal, 2010; Carayannis
and Provance, 2008).Relevant insight can be drawn from the literature on measurement in soft systems, an increasingly important subject in Measurement Science (Finkelstein, 2003; Finkelstein, 2005; Cecconi
et al., 2006). The underlying hypothesis is that, although in thesesituations physical sensors cannot be exploited as data acquisitiondevices (or however the measurand, R&D performance, cannot bemeasured by means of a physical effect of transduction), somelessons learned in designing and performing ‘‘hard measurement’’processes (in terms of, e.g., modeling of the relation betweenmeasurand and inﬂuence quantities, calibration and traceability tostandards, repeatability and stability, etc.) can be exported to softsystems, thus emphasizing the structure of the process instead of its implementation details.
Contents lists available at ScienceDirectjournal homepage: www.elsevier.com/locate/ijpe
Int. J. Production Economics
09255273/$see front matter
&
2011 Elsevier B.V. All rights reserved.doi:10.1016/j.ijpe.2011.06.018
n
Corresponding author. Tel.:
þ
39 0331 572 233; fax:
þ
39 0331 483 447.
Email addresses:
vlazzarotti@liuc.it (V. Lazzarotti),rmanzini@liuc.it (R. Manzini), lmari@liuc.it (L. Mari).
Int. J. Production Economics 134 (2011) 212–223
This paper aims at contributing to this growing body of knowledge, focusing on the deﬁnition of a model for R&Dperformance measurement. The paper is organized as follows.The next section introduces some basics of measurement in softsystems, in view of interpreting them in the case of that speciﬁcnonphysical quantity, which is R&D performance. Section 3presents the proposed model of R&D performance measurement,and that in Section 4 is applied to an exemplary case. Section 5synthesizes the highlights of this work, outlines some of itslimitations, and suggests some directions for future research.
2. Measurement in soft systems
An increasingly important subject of research in MeasurementScience is the analysis of measurability conditions (Rossi, 2007;Mari, 2007; Mari et al., 2009) for nonphysical properties, to
which physical transducers cannot be applied, by transferring tosuch ‘‘soft’’ properties what have been learned in measurement of physical quantities in many centuries of scientiﬁc and technological development. In the current literature this borderline ﬁeld of analysis is termed ‘‘measurement in soft systems’’, or sometimes(more appropriately) ‘‘measurement of soft quantities’’, or evensimply ‘‘soft measurement’’. Recently, an authoritative contribution to the analysis of measurement in soft systems has comefrom the ‘‘Guide to the expression of uncertainty in measurement’’ (GUM) (BIPM, 1995), which has thrown some new lighton the classical distinction between ‘‘direct’’ and ‘‘derived’’(or ‘‘indirect’’) measurement. The basic hypothesis is that theproperty intended to be measured, called in this context the‘‘measurand’’, must be characterized by a suitable model describing, in particular, the relations between the measurand itself andother properties, generically called ‘‘input quantities to themeasurement model’’ and including in particular all relevantinﬂuence quantities that could affect the measurand value. Hence,it is acknowledged that several components generally contributeto the measurand value and uncertainty, so that any measurement in which such components must be combined shouldbe dealt with as an indirect process that includes an informationprocessing stage. The considered measurand is indeed the outputquantity obtained by processing one or more input quantities by afunctional relationship that the GUM calls the (mathematical)measurement model.In principle, such measurement models have thus the samestructure for both hard and soft systems: what makes thedifference is the lack of a generally agreed theory embedding asystem of relations among soft quantities, analogous to theInternational System of Quantities (ISO, IEC, 2007) for physicalquantities. That is why measurement in soft systems is mainlyconcerned with the problem of suitably selecting input quantities(in this context usually called ‘‘indicators’’, plausibly to emphasizetheir role of codetermining the measurand) and algorithmicallycombining them to obtain a value for the searched quantity, i.e.,the measurand. In this context the fundamental issue arises of how to characterize measurement with respect to generic assignment of numerical values to quantities, as it could be performedby, e.g., estimation, guess, etc., so to guarantee the epistemicsigniﬁcance of the results. Accordingly, our attempt here is toapply some general principles of measurement in soft systems toR&D,in order to identify a model able to give as much as possiblearobust and reliable measurement to R&D performance. Such amodel should be able to operatively support the identiﬁcation of the conditions for an objective and intersubjective numericcharacterization of R&D performance, such as they are requiredto consider it a ‘‘proper case’’ of measurement (see, e.g., (Mari,2003, 2007):
Objectivity
: measurement results should convey informationon the considered system and not the surrounding environment (which typically includes the subject who is measuring).In physical measurement systems objectivity is obtained byguaranteeing a sufﬁcient stability and selectivity of the system, so to make its output invariant to the effects of theenvironment, i.e., to the variations of the inﬂuence quantities.Hence, objectivity is a condition of reliability for the information produced by the evaluation process.
Intersubjectivity
: measurement results should be interpretedin the same way by different subjects. In physical measurement systems intersubjectivity is obtained by calibration, thatmakes the system output traceable to a standard, so thatdifferent systems traced to the same standard produce comparable results. Hence, intersubjectivity is a condition of public interpretability for the information produced by theevaluation process.Furthermore, the problem of characterizing measurement ismade complex by its polysemy, as the following diagram highlights (see Fig. 1).A data acquisition process (1) applied to an empirical object,i.e., the system under measurement (
s
), produces an informationentity (
x
), which is in turn processed (2) leading to a furtherinformation entity (
y
). Hence, the concept of (physical) measurement can be recognized as twofold:
measurement as data acquisition (1): this is traditionallycalled
fundamental
(or also: direct)
measurement
;
measurement as data acquisition
þ
data processing (1
þ
2): thisis called
derived
(or also: indirect)
measurement
.Furthermore, when taking into account some, usually nonphysical, quantities a third meaning is adopted:
measurement as data processing (2), to obtain the value (
y
) fora property of the object of interest (
s
) from some raw data (
x
),under the hypothesis that such raw data actually wereobtained from that object in some reliable way.R&D performance is not generally considered a physicalproperty, so that no physical transducers sensitive to performancecan be exploited. Some analysis on the concept of derivedmeasurement can be useful at this regards, also aimed atidentifying the structural elements on which objectivity andintersubjectivity could be obtained in this case.According to a black box model, measurement processes canbe formalized as represented in Fig. 2, i.e., the measurand isthought of as an entity that allows mapping system states tosymbolic values,
p
:
S

X
, so that measurement results are interpreted as measurand values. The case of derived measurement is just a specialization of this general model (as presented in Fig. 3),where thus
p
i
:
S

X
i
and
f
:
X
1
. . .
X
n

Y
.Accordingly, the measurement of the measurand
Y
implies that avalue to all input quantities
X
i
has been previously assigned. Sincethis is generally a demanding requirement, redundancies in theargumentof the model
f
are avoided,,the quantities
X
i
arechosensoto be empirically independent and thus reduce the information
Fig. 1.
Measurement as data acquisition and possibly data processing.
V. Lazzarotti et al. / Int. J. Production Economics 134 (2011) 212–223
213
acquisition costs. A critical issue in this context is the
validation
of the measurand model and the measurement process: how can it beguaranteed that the obtained result actually expresses a value forthe modeled measurand? Two general strategies can be envisioned:(1) the sameness of results obtained from empirically independentmeasurement processes, and (2) effectiveness of a measurementdriven decision process. Let us introduce and brieﬂy comment them.
First strategy
: let us suppose that two empirically independentmeasurement processes are available to measure what is deemedto be the same measurand. The fact that the two processessystematically produce comparable results when applied to thesame system under measurement can be interpreted as a validation for both the measurand model implied by the processes andthe processes themselves (see Fig. 4).As a classical example from physics, inertial mass
m
i
andgravitational mass
m
g
are measured in different experimentalsettings and by means of different systems, so that whenever it isascertained that
m
i
E
m
g
the models deﬁning the two measurandsand the two measurement processes are mutually corroborated intheir validity. It should be noted that the critical requirement hereis empirical independence of the two processes, and therefore of the two models on which the processes are deﬁned.
Second strategy
: let us suppose that a problem must be solvedwhich somehow involves what is deemed to be the measurandunder consideration, and that the solution for such a problemimplies an empirical transformation of the system, so that themeasurand can be evaluated before and after the transformation.Let us also hypothesize that an inferential process is availablewhich produces a decision about how to operate the transformationon the basis of, among other pieces of information, the measurandvalue (e.g., ‘‘if the measurand value is
y
thenoperatethis way’’). Thewhole process can be diagrammatically represented as follows (seeFig. 5).The fact that the state resulting from the transformation iscomparable to the intended one increases the plausibility of thehypothesis that what has been measured was actually themeasurand, and thus can be interpreted as a validation for boththe measurand model and the whole decision process.
3. A model for performance measurement in R&D
The aim here is to analyze the measurability of performance inR&D processes of a company as the system under measurement.Such a system is considered as the set of activities and resourcesthat the company devotes to innovation, thus including
activities such as basic research, applied research, development and the relative supporting activities, such as, forexample, technology intelligence, technology scouting, marketanalysis; and
resources such as people, tangible resources (laboratories,machinery, etc.), intangible resources (knowhow and competences, patents, etc.).These activities and resources can be either concentrated in asingle and well deﬁned organizational entity (the ‘‘R&D unit’’) ordispersed inside the company, and may be intertwined to otheractivities and resources, not speciﬁcally devoted to R&D (as veryoften it is the case in small and medium enterprises). According tothe well established ‘‘balanced scorecard’’ approach, R&D performance has ﬁve ‘‘aspects’’, or ‘‘perspectives’’ as we will call them(Mari et al., 2009; Kaplan and Norton, 1992; Bremser and Barsky,
2004; Sandstrom and Toivanen, 2002; Kerssensvan Drongelen
and Cook, 1997; Chiesa et al., 1996; Eccles and Pyburn, 1992;
Gregory, 1993; Berg et al., 2002; Lebas and Euske, 2002; Kaplan
and Norton, 2006; Lee and Lai, 2007):
ﬁnancial perspective;
customer perspective;
innovation and learning perspective;
internal business perspective;
alliances and networks perspective.The logic according to which such perspectives contribute toperformance is complex and at least partially contextdependent,deﬁned as a (suitably calibrated) combination of two generalstrategies: (i) mutual balancing, i.e., a low performance in aperspective is compensated by a high value in another perspective; (ii) chain strength determined by the weakest link, i.e., a lowperformance in a perspective hinders the global performance.Furthermore, in this context performance values are generallyacknowledged to be meaningful only relatively to a reference(e.g., a market benchmark or an internally deﬁned target), as alsousual when measuring quality as conformance to speciﬁcations.These assumptions together constitute the underlying model forthe concept of performance of the R&D system.The R&D overall performance, i.e., the property
p
whosemeasurability is under analysis, derives from these perspectives,which thus operate as the input measurands,
p
1
,
y
,p
5
, in themeasurement process, i.e.,
p
(
s
)
¼
g
(
p
1
(
s
),
y
,
p
5
(
s
)) for some function
g
(). On the other hand, such perspectives are still too complexto allow the direct acquisition of their values. Rather, severalindicators
indj
(
i
) can be used to give each measurand
pi
a value,
pi
(
s
)
¼
h
i
(
ind
1
(
i
)(
s
),
y
,
indni
(
i
)(
s
)). Hence, each
pi
is in its turn the
Fig. 3.
A functional model of derived measurement.
Fig. 4.
First validation strategy.
Fig. 5.
Second validation strategy.
Fig. 2.
A black box model of measurement.
V. Lazzarotti et al. / Int. J. Production Economics 134 (2011) 212–223
214
result of a derived measurement process, whose inputs are calledindicators:
p
ð
s
Þ ¼
g
f
h
1
½
ind
1
ð
1
Þ
ð
s
Þ
,
. . .
,
indn
1
ð
1
Þ
ð
s
Þ
,
. . .
,
h
5
½
ind
1
ð
5
Þ
ð
s
Þ
,
. . .
,
indn
5
ð
5
Þ
ð
s
Þgð
1
Þ
So far, there is not any model in literature deﬁning how:
to obtain a value for the perspectives
pi
¼
h
i
(
ind
1
(
i
),
y
,
indni
(
i
))as a function of one or more indicators
indj
(
i
): we will denote itas
Measurement Model
1;
to express the R&D overall performance as a synthesis of thevalues
pi
, i.e., such that
p
¼
g
(
p
1
,
y
,
p
5
): we will denote it as
Measurement Model
2.In the following, these two models, whose role is synthesizedin Fig. 6, are explored in some further detail.
3.1. Measurement Model 1
Many different indicators can be used to evaluate performancefrom each of the ﬁve considered perspectives. Each indicator can bethought of as related to a speciﬁc perspective
pi
, but no singleindicator is semantically rich enough to convey information onperformance as a whole, nor on any of the ﬁve perspectives.Furthermore, it is not clear how a set of indicators can be identiﬁedso to convey a (sufﬁciently reliable) information on R&D performance. To this aim, together with the distinction by ‘‘perspective’’(i.e., ﬁnancial, customer, etc.), the classiﬁcation ‘‘input vs. processvs. output’’ indicators, also well known in literature and in practice,can be useful. A list of possible indicators, according to both thetypes of classiﬁcation is reported in the appendix. This list, whichhas absolutely no claim to be comprehensive, has been developedby authors on the basis of literature suggestions (in particular, thecontributions focused on balanced scorecard perspectives (Bremserand Barsky, 2004; Sandstrom and Toivanen, 2002; Kerssensvan
Drongelen and Cook, 1997), and contributions that distinguish theindicators by type (Brown and Svenson, 1988; Werner and Souder,
1997; Hauser, 1998; Chiesa et al., 1996), analyzing speciﬁcally input
(Baruk, 1997; Hagedoorn and Cloodt, 2003; Parthasarthy and
Hammond, 2002), process (Howells, 1995; Kahn, 2002; Koen and
Kohli, 1998), output (Parthasarthy and Hammond, 2002; Michalisin,
2001; Coombs et al., 1997)), and experience in companies.
Accordingly, performance in R&D is evaluated considering: (i) thequantity and quality of input resources available for R&D activities;(ii)theefﬁciencyandeffectivenessofR&Dprocesses;(iii)thequantityand quality of results actually achieved. Hence, a minimal set of indicatorstoevaluateperformanceineachperspectivecanbebuiltbyselecting three indicators: one input indicator, one process indicatorand one output indicator. The hypothesis is that each companyshould be able to choose, among the wide set of indicators availablefor each perspective, those speciﬁcally coherent with its speciﬁcactivity, organizational structure, information and management control systems, culture, managerial competences, etc. To each selectedindicatoraquantitativevalueisthenassignedinreferencetothetimespan under consideration, typically by extracting the related informationfrom companydatabases. Sucha value isthencomparedtosomepossible references in order to measure performance in relativeterms. Three references canin particular be takenintoaccount,whichare useful to grasp different aspects of performance evaluation:
the
previous
indicator value, as evaluated at a previous time;
the
target
indicator value, as planned objective for the sameperiod;
the
benchmark
indicator value, as mean value computed fromthe available sample of competitor data.Under the following assumptions:
the perspectives
pi
,
i
¼
1,
y
,5, can be measured independentlyof each other (therefore, for the sake of notation simplicity inthe following the index
i
is left implicit when the reference isto a generic perspective);
among the high number of potential indicators, the candidatesas input measurands are chosen so to be measurable in aratio scale.Measurement Model 1 can be formalized by a two stageprocedure as follows:
the value
indj
(
s
(
t
)) of each indicator
indj
for the R&D unit
s
under consideration at the time
t
is compared to a reference
refj
, i.e., the value of the same indicator either (i) measured for
s
at a previous time
t
D
t
,
refj
¼
indj
(
s
(
t
D
t
)), or (ii) hypothesized as the target for
s
,
refj
¼
targetj
(
s
(
t
)), or (iii) computed as amarket benchmark,
refj
¼
benchj
(mkt); hence, the ﬁrst stage of Measurement Model 1 computes the comparative indicator
cindj
as
cindj
ð
s
Þ ¼ ð
indj
ð
s
Þ
refj
Þ
=
refj
ð
2
Þ
thus making the comparative indicators dimensionless quantities, that can be compared with one another in their turn andsynthesized by statistical operators (the deﬁnition of
cindj
(
s
) isthe basic justiﬁcation of the previous requirement for indicators to be measurable in a ratio scale
1
);
the second stage of Measurement Model 1 computes theperspectiverelated performance
pi
as
pi
ð
s
Þ ¼
h
½
cind
1
ð
s
Þ
,
. . .
,
cind
n
ð
s
Þ ð
3
Þ
being
h
¼
E
W
1
½
min
ð
cind
1
,
. . .
,
cindn
Þ
,
E
W
2
ð
cind
1
,
. . .
,
cindn
Þ ð
4
Þ
where
E
W
() is the weighted mean operator, being
W
the vectorof its weights, and min() is the minimum value operator; thefunction
h
formalizes the twofold logic of mutual balancingamong indicators, obtained by taking their mean value, andacknowledging that the weakest link of the chain hinders the
Fig. 6.
Models of R&D performance measurement.
1
Of course, Eq. (2) must be suitably modiﬁed in the case
refj
¼
0.
V. Lazzarotti et al. / Int. J. Production Economics 134 (2011) 212–223
215
global performance, formalized by ﬁnally taking the meanvalue of the worst and the average performance.The two vectors of weights
W
1
and
W
2
allow the ﬁne tuning of the evaluation process by taking into account:
[
W
1
] the relative importance of the worst performance withrespect to the average one, which depends on the speciﬁcsituation of the company and on the features of its industry(for example, the importance of the minimum is higher if theunderlying indicator reveals a status of particular weakness forthe company with respect to the competitors and the indicatoritself represents a critical factor to successfully compete in thespeciﬁc industry);
[
W
2
] the relative importance of each considered (input, process,output) indicator, which depends on several factors, such as
the company phase in its life cycle; for example, in a phaseof startup with respect to a maturity stage, the input andprocess indicators are typically more important than theoutput ones;
the industry phase in its life cycle; for example, in a highinnovative sector with respect to a mature one, the inputand process indicators are typically more important thanthe output ones;
the levers of control established by the management control system emphasizing the relevance of an indicatorrather than another to foster internal motivation.
3.2. Measurement Model 2
The achieved values for the perspectives
p
1
,
y
,
p
5
lead to ameasurement result
y
¼
p
(
s
) for the performance of the R&Dsystem as a whole by means of Measurement Model 2. Such anoverall performance
p
is computed structurally in the same wayeach perspectiverelated performance
pi
is obtained in the secondstage of Measurement Model 1, i.e.,
p
ð
s
Þ ¼
g
½
p
1
ð
s
Þ
,
. . .
,
p
5
ð
s
Þ ð
5
Þ
being
g
¼
E
W
3
½
min
ð
p
1
,
. . .
,
p
5
Þ
,
E
W
4
ð
p
1
,
. . .
,
p
5
Þ ð
6
Þ
thus keeping into account once more the mentioned twofoldeffects of mutual balancing and weakest link of the chain. Also inthis case some weights can assigned to take into account therelative importance of the minimum value with respect to theaverage one (
W
3
) and the relative importance of each perspective(
W
4
). Such vectors formalize the available information on thetypology of company, industry, management control features, etc.It should be noted that, as computed by Measurement Models1 and 2, the value
p
(
s
) expresses the overall performance asrelated to a single reference, either previous value, or target, orbenchmark. An even more synthetic performance value, aggregating the three referencerelated performance values, could be maybe looked for but is not discussed here. Indeed, our claim is thatintegrating such different aspects of performance into a singlevalue does not lead to any signiﬁcant better indications aboutcompany state and behavior.
3.3. Validation
From a managerial point of view the relevant feature of aperformance measurement system is its ability of eliciting information on the state of the system. Accordingly, the validationprocess of the two proposed models can be designed as follows.Measurement Model 1 can be validated by means of anempirical study, by taking into account a statistically signiﬁcantnumber of randomly selected R&D systems to verify, for eachperspective
p
i
and once the reference
ref
and the weight vectors
W
1
and
W
2
have been ﬁxed, whether different triples (input,process, and output) of indicators lead to consistent results. Let
cindi
j
,
cindp
j
, and
cindo
j
be the
j
th input, process, and outputcomparative indicators, respectively, then the model for a system
s
is validated if for all
j
,
k
:
h
ð
cindi
j
ð
s
Þ
,
cindp
j
ð
s
Þ
,
cindo
j
ð
s
ÞÞ
4
0
2
h
ð
cindi
k
ð
s
Þ
,
cindp
k
ð
s
Þ
,
cindo
k
ð
s
ÞÞ
4
0
ð
7a
Þ
h
ð
cindi
j
ð
s
Þ
,
cindp
j
ð
s
Þ
,
cindo
j
ð
s
ÞÞ
o
0
2
h
ð
cindi
k
ð
s
Þ
,
cindp
k
ð
s
Þ
,
cindo
k
ð
s
ÞÞ
o
0
ð
7b
Þ
According to this criterion, Measurement Model 1 is valid if the evaluation of perspectiverelated good (
h
4
0) or bad (
h
o
0)performance is invariant with respect to the chosen input,process, and output indicators (a stronger criterion includes thefurther condition:
h
ð
cindi
j
ð
s
Þ
,
cindp
j
ð
s
Þ
,
cindo
j
ð
s
ÞÞ ¼
0
2
h
ð
cindi
k
ð
s
Þ
,
cindp
k
ð
s
Þ
,
cindo
k
ð
s
ÞÞ ¼
0
ð
8
Þ
on invariance of stability (
h
¼
0) in performance; on the otherhand, such a condition would critically depend on the assessmentof the uncertainty degree of the various properties involved in theevaluation process, a task whose empirical meaningfulness isdisputable in this case). In reference to the previous generalanalysis on validation in measurement, such logic is thus basedon the ﬁrst case of validation, as discussed in Section 2, thesameness of results obtained from empirically independentprocedures.Measurement Model 2 can also be validated by means of anempirical study. The underlying assumption (Chiesa, 2001) isthat, in the long run, an efﬁcient and effective R&D activity shouldincrease the company economic value. The economic value isdeﬁned as the sum of the expected earnings in the long periodand it is measured (Bini and Guatri, 2007) as
X
T t
¼
0
R
t
v
t
þ
V
T
v
T
ð
9
Þ
where
R
t
is the average expected earning from 0 and
T
(until thismoment an analytical forecast is possible),
V
T
the averageexpected earning from
T
and
N
, calculated as perpetuity factor
R
T
/
k
,
v
¼
1/(1
þ
k
), and the actualization factor deﬁned on the basisof average cost of capital (
k
).Accordingly, a statistically signiﬁcant number of R&D cases canbe randomly selected and evaluated by Measurement Model 2 for atleast ﬁve years (such a period is required because R&D results arenot meaningful in the short run and require a continuous andcoherent long term activity). Then, if the observation relates toperiod [
t
,
t
þ
5], the economic value of the selected company shouldbe computed for the period [
t
þ
D
t
,
t
þ
D
t
þ
5], the value of
D
t
depending on the speciﬁc sector of activity of the company (indeed,in different sectors the time needed for innovating products and/orprocesses is signiﬁcantly different. Just to make some examples:
in pharmaceutical industry
D
t
is around 13–15 years;
in consumer electronics industry
D
t
is around 2 years;
in aviation industry
D
t
is around 7–9 years.)Again, ﬁve years of evaluation for the economic value arerequired because of the uncertainty of the time delay of R&Deffects on the innovation. Measurement Model 2 is thus validatedwhen
p
ð
s
ð
t
ÞÞ ¼
g
ð
p
1
ð
s
ð
t
ÞÞ
,
. . .
,
p
5
ð
s
ð
t
ÞÞÞ
4
0 for years
½
t
,
t
þ
5

E
ð
ev
ð
s
ð
t
þ
D
t
ÞÞ
,
. . .
,
ev
ð
s
ð
t
þ
D
t
þ
5
ÞÞÞ
4
ev
ð
s
ð
t
ÞÞ ð
10a
Þ
V. Lazzarotti et al. / Int. J. Production Economics 134 (2011) 212–223
216