Kohli Devaraj ISR LR DFA OLS in IT Payoff

Measuring Information Technology Payoff
of 20
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  Measuring Information Technology  Payoff A Meta-Analysis of Structural Variables inFirm-Level Empirical Research Rajiv Kohli ã Sarv Devaraj* Department of Management, University of Notre Dame, Notre Dame, Indiana ã P ayoffs from information technology (IT) continue to generate interest and debate bothamong academicians and practitioners. The extant literature cites inadequate sample size,lack of process orientation, and analysis methods among the reasons some studies have shownmixed results in establishing a relationship between IT investment and firm performance.In this paper we examine the structural variables that affect IT payoff through a meta-analysis of 66 firm-level empirical studies between 1990 and 2000. Employing logistic regres-sion and discriminant analyses, we present statistical evidence of the characteristics that dis-criminate between IT payoff studies that observed a positive effect and those that did not. Inaddition, we conduct ordinary least squares (OLS) regression on a continuous measure of ITpayoff to examine the influence of structural variables on the result of IT payoff studies.The results indicate that the sample size, data source (firm-level or secondary), and industryin which the study is conducted influence the likelihood of the study finding greater improve-ments on firm performance. The choice of the dependent variable(s) also appears to influencethe outcome (although we did not find support for process-oriented measurement), the typeof statistical analysis conducted, and whether the study adopted a cross-sectional or longitu-dinal design. Finally, we present implications of the findings and recommendations for futureresearch.  Meta-Analysis) Information Teehnology  Payoff Business Value IT; Firm-Level; Discriminant Anal-ysis; Logistic Regression; Process-Orientation) 1. Introduction Researchers and business managers consider infor-mation technology (IT) investment as an enabler forimproved organizational efficiency and competitive- ness.  Measurable performance improvements result-ing from IT investment can help sustain investment infuture IT initiatives. However, as demand for IT in-vestment increases, the assumed payoff is likely tocome under scrutiny. Although the much talked about productivity paradox (Ahituv and Giladi 1993,  Both authors contributed equally to the manuscript. Roach 1987, Strassmann 1985) has largely been put torest by recent studies (Brynjolfsson and Hitt 2000,Jorgenson 2001, Jorgenson and Stiroh 2000, Kraemerand Dedrick 2001, Oliner and Sichel 2000), not allstudies have demonstrated clear payoff from ITinvestment.Among the reasons attributed to equivocal results  o past studies are structural issues such as inadequatemeasurement and analysis methodologies (Brynjolfsson 1993,  Robey and Boudreau 1999) and time lags in mea-suring payoff (Devaraj and Kohli 2000a). Suggestionsto examine IT payoffs include improving the quality 1047-7047/03/1402/0127$05.001526-5536 electronic ISSN INFORMATION SYSTEMS RESEARCH,  © 2003 INFORMSVol. 14, No. 2, June  2003,  pp. 127-145  KOHLI AND DEVARAJ Measuring nformation Technology Payoff of data and the analytical rigor Robey and Boudreau1999), examining valuation  and  conversion barriers Chircu  and  Kauffman 2000, Davern  and  Kauffman2000), applying improved modeling techniques Hittand Brynjolfsson 1996),  and  examining intermediateand context-related variables {Barua  et  al. 1996, Baruaand Mukhopadhyay 2000). Similar calls for improvingexecution  of  payoff studies and  to  improve reliabilityof results have been made by past review papers andsyntheses  of  IT payoff studies Brynjolfsson and Yang 1996,  Mahmood  et  al. 1999, Sircar  et  al. 1998). There-fore, there  is  need  for a  systematic analysis to under-stand  the  structural characteristics  of  past  IT  payoffstudies and how they affect their outcomes. Such find-ings will  not  only help critically view results  of  paststudies, they will also serve  as a  guide  for  futureresearch.We report  a  meta-analysis  of  firm-level  IT  payoffstudies  to  catalog  the  extant literature, examine vari-ables that influence the findings, and provide sugges-tions  for  conducting future studies. In conducting thismeta-analysis, we examine  the  structural dimensionsalong which IT payoff studies differ. The analysis  at- tempts  to  observe  a  pattern,  if  any, which discrimi-nates between studies that result  in  positive IT payoffand those that do not. Furthermore, we investigate theextent  of  payoff resulting from such structuraldimensions.The paper  is  organized  as  follows. In §2, we brieflyreview  the  meta-analysis studies  in  the IT payoff lit-erature and present  a  literature-guided framework  to review the IT payoff studies.  In  §3, we present detailsof the procedure adopted  for  the meta-analysis. In §4,we present results of the statistical analysis of  IT  payoffstudies. Finally, §5 presents our findings  of  the influ-ence  of  structural variables, followed  by  limitations,contributions, and areas for future research. 2.  Review  of  Literature 2.1.  Review  of  IT Payoff LiteraturePast firm-level studies  of  IT payoff can  be  viewed  as addressing three general questions—  hat  is mea- sured,  hoxv  is it  measured,  and  zvhere  is it  measured? Banker  et  al. 1993, Berger  et  al. 1988, Mahmood  and Szewczak 1999). In  wh t  is measured, past studies pro-pose that IT performance  is  associated with variablesthat transcend traditional measures and include mea-sures  of  productivity,  in  addition  to  profitability Mahmood  and  Mann 2000). Although there  is  not  a consistent  set of  performance variables, even whenmeasurement variables are identified, the quality andcompleteness of data and the subsequent robustness  of the analysis appears  to  impact  the  outcome  of  the  IT payoff studies Brynjolfsson and Yang  1996,  Mahmoodet al. 1999).  In  other words, the data source and anal-ysis approach have  a  bearing on the IT payoff result.Study characteristics, such  as  duration  of  data col-lection and the process  of  IT investment, describe  how the data are gathered. We find that some studies gatherdata at one point in time Prattipati and Mensah 1997),while others collect three  to  five annual data points Barua  et al.  1995, Dewan  and Min  1997, Hitt  and Brynjolfsson 1996, Prasad  and  Harker 1997). The  du- ration  of  studies, combined with the number  of  firms,determines the sample size or data points captured  in a study. The process of IT investment can be examinedto assess  if  appropriate  IT  assets  and  impacts weremeasured Soh  and  Markus 1995). This process mea-surement view proposes that IT expenditures have  to be converted into appropriate IT assets. The appropri-ate use  of  IT assets leads to IT impacts, and IT impactswhen positioned competitively, lead to impacts on or-ganizational performance Lucas 1993, Mooney  et al. 1996,  Soh and Markus 1995). Even when IT spendingis shown  to  improve intermediate variables  of  orga-nizational productivity, such  as  improved communi-cation leading  to the  need  for  reduced inventories Dudley  and  Lasserre 1989),  it  does  not  necessarilylead  to  improvements  in  productivity Barua  et al. 1991).On the question of  lohere  measurements for  IT  payoffshould occur, prior studies indicate that payoff  has been harder to measure in some industries than others.Furthermore, studies that use firms as the data sourceare likely  to  show  a  positive relationship between  IT investment and firm performance because  of  the com-pleteness  and  availability  of  required variables Hittand Brynjolfsson 1996, Sircar  et  al. 1998).Although greater payoffs among firm-level studiesare generally expected Brynjolfsson  and  Yang 1996,Devaraj and Kohli 2000a, Sircar et al. 1998), our review 128 INFORMATION SYSTEMS RESEARCH/VOI.  14 No. 2 June 2003  KOHLI AND DEVARAJ Measuring Information Technology of the literature finds prevalent differences  in  the con-texts, characteristics, data sources,  and  variables  em- ployed  in  firm-level studies.  To  examine  the  differ-ences  in  the execution  of  IT payoff studies, we derivea  set of  structural variables  to  develop  a  frameworkalong which studies vary Figure  1). The  categorieswith the relevant literature are cited in Table   and  are discussed thereafter. Appendix  A  provides  the  codesassigned  to  each  of the  dimensions  in our  meta-analysis.The categories and subcategories  in  our  frameworkexpand past work that recommends taking into  ac- count research designs  and the  process  of  conversionof IT expenditure into benefits  in  examining IT payoff McKeen  and  Smith 1993). Resulting from  tbe  discus-sion  of  categories  of  structural variables  in  the frame-work,  we  present propositions  to  examine  the  influ-ence  of  each structural variable on IT  payoff.  To assistfuture  IT  payoff studies,  we  present  a  summary  of studies witb their dimensions data source, method,and dependent variables selected  in  Appendix B. 2.2.  Structural Variables and Propositions 2.2.1.  Context. Industry sector studies can differby the type  of  industry  of  tbe IT investment and sub-sequent payoff measurement.  IT s  role  and  intensityare often affected  by  the competitive nature  of  the  in- dustry. Furthermore, technology applied  in  manufac-turing,  for  instance, computer aided design and com-puter aided manufacturing CAD/CAM) or electronicdata interchange EDI),  can  yield tangible  and mea- surable efficiency outcomes Mukbopadhyay  et al. 1995b). The measurable impact of such outcomes  is  sig-nificantly different from  IT  investment  in  healthcare Figure  A  Framework  tor  the Structural Categories InfluencingIT Payott ContexiStudyCharacteristicsData Soiirce(s) f Djtii Analysis ã Results  of  IT Pavolf i VariablesEmployed services,  for  instance,  to  detect drug interactionsthrough  a  clinical information system, which  are  lesstangible.  The  issue  of  measurement  can  also becomecomplicated when  the  industry type  is a  state  or fed- eral government where traditional profitability mea-sures may  not  apply. Consequently, studies  in  differ-ent industries, even when measured  for  similar  IT systems, can lead  to  different results Irani  et  al. 1997,Kelley 1994, Sohai  et  al. 2001). PROPOSITION  1.  IT payoff  loill differ mong  the  industrysector  of  the  firms 2.2.2. Study Characteristics.  The  studies  re- viewed  in  this paper have varying levels  of  data  ag- gregation, namely, month, quarter,  or  year. Similarly,the studies cover varying periods over which data  are collected. Most firm-level studies  use  aggregation  of data  at the  annual level.  It  can  be  hypothesized thataggregation  at  the quarterly  or  monthly level may bet-ter locate  the  payoff Kohli  and  Devaraj 2000),  as  op-posed  to  annual aggregation where gains  in  part  of a year can be offset by low-gain months in the same year.From an analysis standpoint, more frequent data helpsin identifying appropriate time lags  to  detect the pay-off from investments.The duration  for  which studies capture and analyzedata also varies widely. About half  of the  studies  in our meta-analysis have collected data  for  less than  a five-year period. Finally, studies may simply vary  in the number of firms that were included in the analysis.We created a variable  s mple size  to capture the numberof observations  as  well  as  to  account  for  the aggrega-tion  and  duration  of  the study when  it  was longitudi-nal  or  panel.  A  small sample size increases standarderrors  and  thus makes  it  more difficult  to  isolate  the effects  of  IT investment from random noise. Further, asmall number  of  data points may not be sufficient  for establishing  a  trend  for IT  payoff,  especially whenthere  are lag  effects resulting from IT investment andmeasurable payoff Mukhopadhyay  et  al. 1997a). PROPOSITION  2.  Studies using  l rger s mple sizes  will show gre ter  IT  payoff 2.2.3.  Data Source.  The  accuracy  of IT  payoffstudies is contingent upon tbe quality of data utilized.Therefore, the source  of  data is critical to the IT payoff INFORMATION SYSTEMS RESEARCH/VOI.  14, No. 2, June 2003 129  KOHLI AND DEVARAj Measuring Information Technology  Payoff Table 1 Categories Subcategories and Dimensions in iT Payoff Studies CategoriesSubcategories and DimensionsContext Industry Sector—f^ianufacturing, Services, Government, Nonprofit Brynjolfsson and Yang 1996, Robey and Boudreau 1999)Study Characteristics Sample SizeAggregation: Month, Quarter, Year Devaraj and Kohli 2000a)Duration: Number of Years Brynjolfsson 1993, Mahmood and Szewczak 1999)Data Source Firm, Commercial Databases Brvnjoltsson and Yang 1996, Devaraj and Kohli 2000a)Variables Employed Dependent Classification Robey and Boudreau 1999}Data Analysis Statistical Analysis Lee and Barua 1999)Method: Cross-sectional, Longitudinal Brynjolfsson and Hitt 1998, Devaraj and Kohli 2000a, Mahmood and Mann 1997)Level of Detail—IT Assets, IT Impact, Organizational Impact Brynjolfsson and Hitt 1998, Soh and Markus 1995, Ward et al,1996)Result Positive, Negative, Neutral, Partial; Percent + ve and-ve significant variables analysis. Data gathered from secondary sources is eas-ier to obtain and generally objective. In addition, sec-ondary sources can provide access to a greater numberof firms and thus improve the generalizability of con-clusions. Such wider access to data also allows re-searchers to replicate and verify past results by usingnovel analytical approaches. On the other hand, sec-ondary or public data sources may be limited in detailand may not include data considered competitor-sensitive by contributing organizations. Further, datafields in commercial databases are predetermined andmay not match the exact needs of the researcher. Datafrom the firms can overcome several of the above lim-itations by providing greater access, additional detail,and supplementary variables for triangulation of re-sults. When data are accurate, they exhibit fewer errorsin variable bias. Although it may be difficult to deter-mine the direction of the bias, in practice accurate datatend to bias coefficients toward zero. Thus, from anestimation perspective, it is always desirable to haveas accurate data as possible.In addition, it can be difficult to conduct a consistentanalysis across firms without uniform data definitions.Firm-level contexts also provide greater detail throughaccess to contextual variables Brynjolfsson and Hitt1998, Devaraj and Kohli 2000a, Harris and Katz 1991,Morrison and Berndt 1991). PROPOSITION  3.  Studies using primary  data sources  willshow greater IT payoff than those using secondary datasources. 2.2A.  Dependent Variables Employed. Paststudies have employed various types of dependentvariables in examining firm performance. The mostcommonly used dependent variables are financial,such as return on investment ROI) and return on as-sets ROA) Barua et al. 1995, Byrd and Marshall 1997,Lai and Mahapatra 1997, Mahmood and Mann 1993b,Rai and Patnayakuni 1997, Tam 1998a) and revenue Lichtenberg 1995). Productivity- or output-basedmeasures captured as dependent variables includemanagement output Prattipati and Mensah 1997),milk production Van Asseldonk et al. 1988), and totalmail sorted Mukhopadhyay et al. 1997a). Some re-searchers have used expense-based measures, such aslabor hours Mukhopadhyay et al. 1997a), expenses Francalanci and Galal 1998), capacity utilization Barua et al. 1995), and inventory turnover Mukhopadhyay et al. 1995a). Given the wide distri-bution of measures, we created a classification schemeto capture whether the study measured productivity,profitability, or both as dependent measures of firmperformance. The following proposition aims to as-sess if outcomes varied depending upon the type ofdependent measure utilized. PROPOSITION  4.  Studies with  profitability based  depen dent variables will have different IT payoff tban those thatmeasure productivity or both. 2.2.5. Data Analysis  Analytical Approach. The analytical ap-proach to measure IT payoff has varied widely among 130 INFORMATION SYSTEMS RESEARCH/VOI.  14, No. 2, June 2003

ISIS Review

Jul 23, 2017
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!