A Research Synthesis of the Evaluation Capacity Building Literature

A Research Synthesis of the Evaluation Capacity Building Literature
of 33
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
    http://aje.sagepub.com/  American Journal of Evaluation  http://aje.sagepub.com/content/early/2012/01/27/1098214011434608The online version of this article can be found at: DOI: 10.1177/1098214011434608 published online 27 January 2012 American Journal of Evaluation  LesesneSusan N. Labin, Jennifer L. Duffy, Duncan C. Meyers, Abraham Wandersman and Catherine A. A Research Synthesis of the Evaluation Capacity Building Literature  Published by:  http://www.sagepublications.com On behalf of:  American Evaluation Association  can be found at: American Journal of Evaluation  Additional services and information for http://aje.sagepub.com/cgi/alerts Email Alerts:  http://aje.sagepub.com/subscriptions Subscriptions:  http://www.sagepub.com/journalsReprints.nav Reprints:  http://www.sagepub.com/journalsPermissions.nav Permissions:  What is This? - Jan 27, 2012OnlineFirst Version of Record >>  at American Evaluation Association on February 9, 2012aje.sagepub.comDownloaded from   A Research Synthesis of the Evaluation CapacityBuilding Literature Susan N. Labin 1 , Jennifer L. Duffy 2 , Duncan C. Meyers 2 ,Abraham Wandersman 2 , and Catherine A. Lesesne 3 Abstract The continuously growing demand for program results has produced an increased need for eva-luation capacity building (ECB). The  Integrative ECB Model   was developed to integrate concepts fromexisting ECB theory literature and to structure a synthesis of the empirical ECB literature. The studyused a broad-based research synthesis method with systematic decision rules and demonstrates theviability of the method for producing a reliable analysis of disparate data from a variety of designs.There was a high degree of consistency in what was reported in the empirical literature and thetheoretical literature in terms of strategies and outcomes. Reported outcomes at the individual levelincluded attitudes, knowledge, and behaviors and at the organizational level included practices,leadership, culture, mainstreaming, and resources. Collaborative processes and programmaticoutcomes emerged as important issues for ECB models and practice. The consistency between theempirical and the theoretical literature indicates that the field is ready to develop common mea-sures, use stronger designs, and report more systematically. This synthesis provides an overview of existing data and an empirical basis for refining strategies and common measures for enhancing theresearch and practice of ECB to achieve ECB and programmatic goals and outcomes. Keywords ECB, evaluation capacity building, research synthesis, capacity building, systematic review For at least the past decade, evaluation capacity building (ECB) has been attracting the interest of evaluators committed to increasing stakeholder understanding of evaluation and building evaluationcultureandpracticeinorganizations(Boyle,Lemaire,&Rist,1999;Compton,Baizerman,&Stockdill,2002; Fetterman, Kaftarian, & Wandersman, 1996; Milstein & Cotton, 2000). Consequently, there is agrowing theoretical and empirical ECB literature (Compton et al., 2002; Cousins, Goh, Clark, & Lee, 1 Washington, DC, USA 2 Department of Psychology, University of South Carolina, Columbia, SC, USA 3 ICF Macro, Atlanta, GA, USA Corresponding Author: Susan N. Labin, 8517 Rayburn Road, Bethesda, MD 20817, USAEmail: susan@susanlabin.com American Journal of Evaluation00(0) 1-32 ª The Author(s) 2012Reprints and permission:sagepub.com/journalsPermissions.navDOI: 10.1177/1098214011434608http://aje.sagepub.com  at American Evaluation Association on February 9, 2012aje.sagepub.comDownloaded from   2004;Preskill&Boyle,2008).However,todatetherehasnotbeenasystematicreviewoftheempiricalECB literature. The purpose of this article is to address this gap.Synthesizing and taking stock of the existing empirical literature on ECB has a number of intended benefits. A systematic synthesis provides an evidence base about how ECB is being prac-ticed in the field. The results can illuminate the current landscape of ECB by describing the contextsand settings where it occurs, the strategies used, the outcomes reported, and how the strategies have been evaluated. Such a descriptive base is an important step in the development of ECB and cancontribute to the improvement of ECB practice by increasing information sharing for the manyevaluators who have incorporated ECB into their practice (according to an American EvaluationAssociation [AEA] membership survey, at least half of those responding reported they included ECB in their practice; AEA, 2008). This descriptive base can also be of value to organizations and funders that embark on the journey of ECB for improving programs and organizations.Building on the work of others who have contributed to developing a language of and indicatorsfor ECB (Compton et al., 2002; Preskill & Boyle, 2008), this synthesis systematically codifies theliterature. Common terminology and indicators are the basis for describing ECB processes and are prerequisites for evaluating ECB efforts; all of which will enhance both the practice and science of ECB. As with all syntheses (Labin, 2008), this synthesis is intended to clarify existing knowledge,raise questions, and reveal gaps to inform future practice and research. Defining ECB Developing a working definition of ECB was an essential first step for synthesizing the literature onECB. Various definitions of ECB have been proposed (Boyle et al., 1999; Preskill & Boyle, 2008;Schaumberg-Muller, 1996; Stockdill, Baizerman, & Compton, 2002). Our review of thesedefinitions identified common features. For example, each identifies ECB as an activity separatefrom actually conducting evaluations. There were also differences among definitions-some focus primarily on ECB as an activity at the organizational level (Stockdill et al., 2002), while others areconcerned with capacity building at the individual and organizational levels (Preskill & Boyle, 2008;Schaumberg-Muller, 1996).In addition to examining explicit definitions of ECB, we drew on the work of collaborative, participatory, and empowerment evaluation, which are precursors or approaches that include aspectsof ECB. For example, in empowerment evaluation, building evaluation capacity in order to improve program outcomes has been a central and explicit principle since its inception in the early 1990s(Fettermanetal.,1996).Empowermentevaluationandotherparticipatoryandcollaborativeapproachesemphasize how goals similartothose ofECB could beachievedthrough participatorymeans and thus, provided much of the foundation for what has became known as ECB (Cousins & Whitmore, 1998;Fetterman & Wandersman, 2005; Love, 2006; O’Sullivan, 2004; Rodriguez-Campos, 2005). Based on our review of these frameworks and approaches, we developed the following working definitionof ECB: Evaluation capacity building (ECB) is an intentional process to increase individual motivation,knowledge, and skills, and to enhance a group or organization’s ability to conduct or use evaluation. The Integrative ECB Model This working definition was used to identify the types of cases to include in the synthesis. In order todetermine what information would be systematically extracted from the cases, we developed an  Integrative ECB Model   (Figure 1) based on existing ECB frameworks and a review of both the 2  American Journal of Evaluation 00(0)  at American Evaluation Association on February 9, 2012aje.sagepub.comDownloaded from   theoretical and empirical ECB literature (Baizerman, Compton, & Stockdill, 2002b; Cousins et al.,2004; Duffy & Wandersman, 2007; Milstein & Cotton, 2000; Owen, 2003; Preskill & Boyle, 2008;Suarez-Balcazar et al., 2010). In particular, we used Preskill and Boyle’s (2008)  Multidisciplinary Model of ECB . We developed the Integrative ECB Model to ensure that the synthesis reflected and integrated key elements in existing theory and empirical literature on ECB and did not restrictinformation to that from any one existing framework or subset of the literature. Our intention wasto maximize the breadth of information to be extracted, advance the development and use of common terminology, and operationalize the study’s working definition of ECB.A basic logic model of Needs-Activities-Outcomes (Kellogg Foundation, 2001; United Way,1996, 2008; University of Wisconsin-Extension, 2003) was used to organize and portray the keycircumstances, activities, processes, and outcomes of ECB. The logic model structure implies acausal direction from left to right, that is, needs will affect the strategies and strategies will affectoutcomes achieved. Furthermore, implementation and evaluation descriptors may mediate theeffects of strategies on outcomes. Individual level o  Attitudes o  Knowledge o  Skills/BehaviorsOrganizational level o  Processes, Policies, and Practices (PPP) o  Leadership o  Organizational Culture o  Mainstreaming o  ResourcesProgram Outcomes o  Development o  Implementation o  Results  Negative outcomesLessons Learned  Short  Long-Term & Sustainable Strategies o  Theory o  Mode oo  Type o  ContentImplementation o  Target Population OrganizationDomain o  Timing, Frequency,Dosage o  Mid−CourseCorrectionsBarriersEvaluation of ECB o  Approach o  Design o  Measures o  Data type o  Timeframe o  Internal or ExternalReasons–Motivations o  Audience: Internal−External o  Assumptions o  ExpectationsGoals–ObjectivesContext o  Needs Assessment o  Tailored Resources and Strengths o  Individual−Attitudes o  OrganizationalResources: staff/time/moneyEvaluationexpertisePractices,leadership, streaming *   III. RESULTS: OUTCOMES • ooo • ooooo • ooo •• II. ACTIVITIES: WHAT and HOW • ooo  Level: Individual-Organizational oo • oooo • oooooo I. NEED:WHY • ooo •• oo • oo culture, main- Figure 1.  Integrative evaluation capacity-building model.*Collaborative and participatory aspects and processes should be included in defining and operationalizingnearly all elements of the model. Labin et al.  3  at American Evaluation Association on February 9, 2012aje.sagepub.comDownloaded from   Our goal was to create a model that included key activities and processes in ECB. We realize thatour model is a simplification and may not have identified every interactive process that is part of ECB. However, we believe we have captured the major activities and processes, thus allowing for systematic extraction and coding of data. Need for ECB––Why  The first column of the model relates to the need for conducting ECB and who and what motivatesthe interest in ECB (Milstein & Cotton, 2000; Preskill & Boyle, 2008). ECB may be driven byfactors internal to the organization (such as a leader’s desire to increase evaluation within the orga-nization), by external factors (such as funder requirements), or by a combination of internal or exter-nal factors. Preskill and Boyle emphasize the importance of considering three elements related to theneed for ECB: (a) motivation for ECB; (b) assumptions and expectations about ECB; and (c) iden-tification of goals and objectives for ECB. They point out that ‘‘Understanding the organization’smotivation forengaging in ECB  . . .  provides insight into whoshould participate and which teachingand learning strategies might be most beneficial’’ (p. 446). Related to the motivation for ECB areassumptions that may underlie the desire to engage in ECB. Preskill and Boyle suggest that whenthese assumptions are not shared among the key people involved in ECB, the success of the effortmay be inhibited. They also note that the explication of specific objectives is important for the suc-cessful design and implementation of ECB efforts.Conducting a needs assessment and tailoring ECB efforts to the particular population and con-text can affect the selection and implementation of the ECB strategies. The existing characteristicsof an organization have also been hypothesized to affect the type of strategies utilized and their efficacy. Some of these specific factors noted in the literature include attitudes toward evaluation,availability of resources for ECB (staff, time, and financial), internal evaluation expertise, and organizational practices and capacities such as support for evaluation and ECB from leadership,from the organizational culture, and through mainstreaming or making evaluation a routine partof the organization (Milstein & Cotton, 2000; Owen, 2003; Preskill & Boyle, 2008; Suarez-Balcazar et al., 2010). These factors have been hypothesized to affect organizational learning and the extent to which the outcomes of ECB will become sustainable (Preskill & Boyle, 2008). In theIntegrative ECB Model, many of these preexisting characteristics that can facilitate the ECB pro-cess are defined as strengths and resources. ECB Activities––What and How  The second column of the Integrative ECB Model categorizes the activities of the ECB strategies,implementation specifics, and evaluation of the ECB efforts. Various aspects of strategies aredefined to capture important dimensions that define their nature and effectiveness. Some ECBefforts are justified by an underlying theory or approach such as empowerment evaluation or orga-nizational learning. Preskill and Boyle (2008) identified a number of types of theories which caninform the design and implementation of ECB strategies, including theories about evaluation, adultlearning theory, and theories related to organizational change and development.ECB strategies may be provided through multiple modes, such as face-to-face meetings, telecon-ferences or phone calls, e-mail or other web-based mechanisms, and through written materials suchas evaluation manuals. Some may use a mode of exclusively face-to-face efforts, while others mightutilize only distal modes such as phone or web. Strategies can be directed at the individual level for learning and behavior change and at the organizational level. The ECB literature addresses how thelevel at which strategies are directed affects outcomes (Preskill & Boyle, 2008; Stockdill et al.,2002). Types of strategies refer to the mechanisms of delivery, that is, training, technical assistance(TA), and experiential involvement or participation in evaluation activities (Duffy & Wandersman, 4  American Journal of Evaluation 00(0)  at American Evaluation Association on February 9, 2012aje.sagepub.comDownloaded from 
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks