Nature & Wildlife

An initial reliability and validity study of the Interaction, Communication, and Literacy Skills Audit

An initial reliability and validity study of the Interaction, Communication, and Literacy Skills Audit
of 13
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110not included (Gott, Purcell, McCabe, & Williams, 2012). Language and literacy development is broadly defined here as spoken structural and interactional aspects of language (e.g., vocabulary, peer  – peer interaction) as well as emergent literacy outcomes such as oral narrative, print concepts, and phono-logical awareness. The effect of quality childcare is as large as the effect for parenting quality or socio-economic status (e.g., Greenwood et al., 1993; Loeb, Fuller, Kagan, & Carrol, 2004; Magnuson & Shager, 2010; McTaggart, 1991). While most work in this area has been focused on children from low socio-economic status groups, a recent large longitudinal study from Northern Ireland shows that pre-school experiences continue to influence school performance in literacy and numeracy to the age of 11 years, regardless of socio-economic background (Melhuish, Quinn, Sylva, Sammons, Siraj-Blatchford, & Taggart, 2013). It has been repeatedly demonstrated that attending quality childcare increases school readiness (e.g., Skibbe, Connor, Morrison, & Jewkes, 2011). A growing body of literature focuses on the importance of school readiness for academic achievement (Farrar, Goldfield, Introduction Quality early childcare and its impact on children ’ s language and literacy development High-quality out-of-home childcare is consistently reported to facilitate improved language and literacy outcomes using a range of standardized measures prior to school entry (e.g., Greenwood, Whyte, & Harkavy, 1993; McTaggart, 1991). The quality of childcare is related to structural (e.g., physical space, group size) and process-related features (e.g., adult  –  child interaction and curriculum) (Harrison, 2008). This paper focuses on process features, specifically the verbal interactions of ECEs that support lan-guage and literacy development of children enrolled in childcare settings. For the purposes of this paper, the term childcare describes formal care and early education by trained staff of children covering the years before formal primary school attendance. Childcare includes government pre-schools, com-munity pre-schools, long daycare centres, kindergar-tens (that are not part of formal school), and early childhood learning centres. Services such as family daycare and informal care, such as with a nanny, are Correspondence: Patricia McCabe, Speech Pathology, Faculty of Health Sciences, The University of Sydney, PO Box 170, Lidcombe NSW 1825, Australia. Email: International Journal of Speech-Language Pathology , 2014; Early Online: 1–13 ISSN 1754-9507 print/ISSN 1754-9515 online © 2014 The Speech Pathology Association of Australia LimitedPublished by Informa UK, Ltd.DOI: 10.3109/17549507.2014.882988  An initial reliability and validity study of the Interaction, Communication, and Literacy Skills Audit NISRINE EL-CHOUEIFATI , ALISON PURCELL , PATRICIA MCCABE , ROBERT HEARD & NATALIE MUNRO The University of Sydney, Sydney, Australia Abstract Early childhood educators (ECEs) have an important role in promoting positive outcomes for children ’ s language and literacy development. This paper reports the development of a new tool, The Interaction Communication and Literacy (ICL) Skills Audit, and pilots its reliability and validity. Intra- and inter-rater reliability was examined by three speech-language pathologists (SLPs). Five skill areas relating to ECE language and literacy practice were rated. The face and content validity of the ICL Skills Audit was examined by expert SLPs ( n     8) and expert ECEs ( n     4) via questionnaire. The overall intra-rater reliability for the ICL Skills Audit was excellent with percentage close agreement (PCA) of 91  – 94. Inter-rater agreement was PCA 68  – 80. Expert SLPs and ECEs agreed that the content was comprehensive and practical. Based on this preliminary study, the ICL Skills Audit appears to be a promising tool that can be used by SLPs and ECEs in collaboration to measure the skills of ECEs in the areas of language and literacy support. Future psychometric and outcome research on the revised ICL Skills Audit is warranted. Keywords:  Early childhood professionals  , measurement tools  , validity  , reliability  , professional development  , language development  , literacy.  1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191202  N. El-Choueifati et al. & Moore, 2007; McTurk, Nutton, Lea, Robinson, & Carapetis, 2008). School readiness encompasses many factors, including physical and mental health, speech and language skills, social and cooperative behav-iours, and family capacity to engage with schools. The impact of high-quality childcare on language and literacy is even greater for younger children (3 year olds) (McTaggart, 1991) and children from disadvantaged environments (e.g., Melhuish, 2004). The importance of childcare to improve outcomes for children is recognized by many governments, for example the Australian Government and its imple-mentation of the National Partnership for Early Childhood Education for universal access for all Australian 4-year-old children to childcare (Depart-ment of Education Employment and Workplace Relations, 2009). In the Longitudinal Study of Australian Children, 95% of 4-year-old children attended some form of childcare with various factors such as family characteristics and SES influencing childcare attendance (see Harrison, Ungerer, Smith, Zubrick, & Wise, 2009 for a comprehensive review of childcare within Australia). Verbal interactions and high quality childcare High quality childcare occurs as a result of the verbal interactions and instructional skills used by early childhood educators (ECEs) among other factors (Hartas, 2004; O ’ Toole & Kirkpatrick, 2007). What ECEs say and do in their interactions with children is most associated with improved child outcomes in language and emergent literacy, more so than, for example, if children have more books and have bet-ter print and writing materials in their environment (Mroz, 2006; Paradice, Bailey-Wood, Davies, & Sol- omon, 2007; Zaslow, Tout, Halle, Whittaker, & Lavelle, 2010). These ECE acts include the ability of ECEs to actively engage in responsive interactions and provide explicit instructional support (Chiovitti & Piran, 2003; Glaser, 1965; Hewitt-Taylor, 2001). A summary of literature regarding the specific interactional and instructional skills of ECEs that support language and emergent literacy develop-ment and the evidence to support their use is provided in Table I and reported in more detail in a recent systematic review (El-Chouefaiti, Purcell, McCabe, & Munro, 2012). Despite this knowledge base, it is also known that these desired high quality ECEs ’ interactions and instructions are not observed as frequently as desir-able (Mroz, 2006; Waniganayake, Harrison, Cheese-man, De Gioia, Burgess, & Press, 2008). Additionally, even ECEs with university qualifications often do not provide the level of instruction and interactions that lead to best outcomes for children (Hartas, 2004; O ’ Toole & Kirkpatrick, 2007). These findings indicate a potential need to improve the quality of ECE and child language and literacy interactions. Evaluating the verbal interaction and instructional skills of ECEs Measuring ECE ’ s skills relating to language and lit-eracy practice is challenging because of the dynamic nature of their interactions with children. It is beyond the scope of this paper to provide a comprehensive evaluation of existing tools that could be used to measure ECE ’ s interactional skill sets that relate to children ’ s language and literacy development. There are a number of recent and comprehensive reviews identifying tools, for example see Halle, Vick, and Anderson (2010) and Neuman and Carta (2011). However, three key issues arise from this literature and are summarized below. First, there is no tool that currently measures ECE ’ s interactional skills in communicating with families about children ’ s language and literacy. The importance of communicating and involving fami-lies is highlighted by the longitudinal intervention study, Effective Provision of Preschool Education study (Sylva, Melhuish, Sammons, Siraj-Blatchford, Table I. Interactional and instructional skills used by early childhood professionals to facilitate language and literacy. ECP skillsExample of supportive research evidenceActively engage in small group and individual interactions, responding and extending on the language used by children.Girolametto and Weitzman (2002); Justice, Mashburn, Pence, and Wiggins (2008)Introduce and model new and rare vocabulary in a range of contexts.Wasik and Bond (2006); Wasik and Hindman (2011)Make explicit references, linking letter names with corresponding sounds. Manipulating sounds within words as well as producing rhymes.Lonigan, Farver, Phillips, and Clancy-Menchetti (2011); McIntosh, Crosbie, Holm, and Dodd (2007)Make explicit references to the form and meaning of print and encourage children to engage in writingMcGinty, Justice, Piasta, Kaderavek, and Fan (2011); Whitehurst et al. (1999)Use a variety of open-ended questions that stimulate inferential thinking and support story comprehension.De Rivera, Girolametto, Greenberg, and Weitzman (2005); Zucker, Justice, Piasta, and Kaderavek (2010)Provide direct verbal supports to facilitate peer-to-peer interactions.Girolametto, Weitzman, and Greenberg (2004); Schuele, Rice, and Wilcox (1995)Involve and communicate with families about how to extend on language and literacy in the home and community context.McLain and Heaston (1993); Melhuish et al. (2008); S é n é chal and LeFevre (2002)  123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120  The ICL skills audit    3 & Taggart, 2004). In a case study analysis of 12 childcare centres identified in the middle and upper range of effectiveness, the researchers found that centres where ECEs shared educational aims with parents obtained related positive outcomes for chil-dren ’ s language, reading, and other areas of academic development. Including the specific skills ECEs use to communicate and involve families to engage in language and literacy development could prove to be a useful component within a measurement tool. Second, some tools evaluate only global factors or the combination of global factors and individual skills rather than focusing on the individual skills of the ECE. Global measures are comprehensive assess-ments that include aspects of adult and child interac-tion but also measure a range of other attributes including the physical environment of the childcare centre, personal care, and safety routines. One exam-ple of a global measure is the Early Childhood Envi-ronmental Rating Scale-Revised (ECERS-R, Harms, Clifford, & Cryer, 2005). Mashburn, Pianta, Hamre, Downer, Barbarin, Bryant, et al. (2008) caution that using global measures may allow for an increased score on the measurement tool without any actual change to the quality of the ECE ’ s interaction with children merely by improving the quality and quan-tity of classroom materials. Other assessment tools measure specific, individual skills. For example, the Teacher Interaction and Language Rating Scale (TIRLS, Girolametto, Weitzman, & Greenberg, 2000) measures specific aspects of ECE ’ s interac-tions that support language development. O ’ Toole and Kirkpatrick (2007) and Girolametto, Weitzman, and Greenberg (2003) used the TIRLS to measure changes in ECEs ’ interactional skills following their participation in the Learning Language and Loving It Program (LLLI: Weitzman & Greenberg, 2002). However, in both studies the training effects of other elements of the LLLI program such as ECE ’ s skills to support literacy or peer  – peer interactions could not be measured via the TIRLS and other more intensive observational coding procedures were applied. What remains to be seen is whether one tool can sufficiently encompass ECEs ’ instruction and interactional skills relating to multiple aspects of both language and emergent literacy. Such a tool would allow for the measurement of training efficacy for individual ECEs. Third, there is a need to develop a tool that can be used within a collaborative framework between ECEs and a professional development trainer, one that is independent of a specific training program and transparent to all parties in its use within a pro-fessional development partnership. Speech-language pathologists work in collaboration with early child-hood programs to provide professional development to ECEs (e.g., Mroz, 2006; Paradice et al., 2007). Following such collaborations, positive changes to child outcomes have been documented (Hartas, 2004; Wright & Kersner, 1999; Zaslow et al., 2010); however, more research is needed to understand the impact of professional development on the specific aspects of the verbal interactions and instructional skills practiced by ECEs leading to these outcomes (Paradice et al., 2007). Ideally, such tools would be freely available, highly reliable, and have high ecological validity. The current study This paper presents a new tool; The Interaction, Communication and Literacy (ICL) Skills Audit, that can be used by both ECEs and SLPs to measure the interactional and instructional skills of ECEs known to facilitate children ’ s language and literacy development. The tool was designed to be used by both professions synchronously, with the ECE completing a self-assessment and the SLP observing the ECE in situ  . Both professionals complete a com-plementary form to allow for direct comparison of perceived ECE skills. The resultant dual rating allows for shared goal-setting within a partnership and coaching professional development approach (Mroz, 2006; Oldroyd & Hall, 1991). We report on the development of the ICL Skills Audit and present two pilot studies in which we aimed to explore whether: The ICL Skills Audit can be used reliably to (1) measure the skills of ECEs. The ICL Skills Audit has well-defined face (2) and content validity from both SLP and ECE perspectives. Method The University of Sydney Human Research Ethics Committee approved the research ((Ref. 11470, 13088). The development of the ICL Skills Audit is summarized below. Item development of the ICL skills audit In our recent systematic review, El-Choueifati et al. (2012) reported four broad areas of ECE behaviour which are documented to improve the language and literacy outcomes of children in early childhood set-tings. These include: “ (1) Developing high-quality adult  – child interactions, (2) Explicit literacy instruc-tion, (3) Developing storytelling skills, (4) Support-ing peer-to-peer interactions ” (p. 8). These skills are common to both speech-language pathology and ECE practice and are often the focus of collabora-tion between the professions. In addition, to trian-gulate these skills against the perceptions of ECEs about what was important, two ECE focus groups were conducted (El-Chouefaiti, McCabe, Munro, & Galea, 2009). Following these focus groups two additional skill areas were identified; working with small groups and developing family involvement.  1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191204  N. El-Choueifati et al. This completes the set of six skills that should be examined in professional development collabora-tions with ECEs, as shown in Table II. For each key skill area identified in Table II, sub-skills were then developed by breaking down the key skill area into its components. Next, examples of behaviours were developed for each sub-skill. These were developed by combining examples from the literature review and clinical experience. This was to provide practical, visible, and explicit definitions of the sub-skills of the ICL Skills Audit. ICL rating measures The ICL Skills Audit uses two scales to rate the lan-guage and literacy promoting behaviours of ECEs. The first scale reports the frequency of how often ECEs believe they use the identified language and literacy behaviours with a 5-point Likert scale from “ never ” to “ all the time ” (beliefs scale), while the sec- ond scale measures how confident ECEs feel about those behaviours (confidence scale). From the focus groups (El-Chouefaiti et al., 2009) we learnt that ECEs may feel uncertain in using particular skills, and this could be reflected in the frequency with which they used the particular skills. Measuring ECE confi-dence can, therefore, be used to enhance the reflection process and allow implicit understandings affecting practice to become clearer (Sheridan, 2001). It was important to acknowledge ECE ’ s perceptions about their own skills as, when ECEs undertake a process of guided self reflection, they are then involved in the process of learning and taking ownership for goal-setting (Nolan, Raban, & Waniganayake, 2005). Self-reported understanding of confidence for each identified sub-skill was determined on a 5-point scale from “ very confident ” to “ not at all confident ” . The ICL Skills Audit also required ECEs to provide exam-ples of how they demonstrate the listed skills and behaviours in their everyday interactions with children. This allowed ECEs to report their practical knowledge and understanding of the skill area and clarified any potential differences in terminology between them and the SLP (Hartas, 2004).   Administering the ICL skills audit Each ECE completed the ICL Skills Audit individu-ally, with observations of ECEs by an experienced SLP observer situated within the ECE ’ s workplace over the course of several days. The ECE ’ s verbal and non-verbal turns were observed and orthographi-cally transcribed by the observer. The transcriptions provided a record that could be used to support reli-ability and agreement of the ECE ’ s and SLP observ-er ’ s ratings. A minimum of three interactions were observed for each ECE. The length of an observation varied in duration from 5  – 20 minutes. No individual children were targeted for inclusion in observation, and no control was made over who the ECEs inter-acted with, as this tool is designed for use in uncon-trolled, naturalistic childcare settings. Following the assessment, the SLP observer met with each ECE individually to highlight their strengths and acknow-ledge the skills of the ECE. The meeting also pro-vided an opportunity to discuss any discrepancies between self-reported and observed skills as well as negotiate goals for professional development. Test phases of the ICL skills audit Test phases of the ICL Skills Audit were conducted in a cyclic approach within a participatory action research framework (Greenwood et al., 1993). This allowed for critical reflection within each cycle and provided opportunities to correct errors. Three cycles of development occurred involving a total of seven childcare centres and 26 ECEs. At the end of Cycle 1, suggested examples of ECE behaviours located on the ICL Skills Audit form were removed because this tended to restrict ECEs in reporting their own exam-ples. In addition, the 6th key skill area of family involvement was included. Cycle 2 resulted in the addition of skills areas supporting stages of bilingua-lism, while Cycle 3 concluded with changes to belief and confidence scales and the removal of double-barrelled questions. Following the development and test phases of the ICL Skills Audit, two studies were conducted: Study 1: Reliability of the ICL Skills Audit and Study 2: Validity of the ICL Skills Audit. Study 1: Testing the reliability of the ICL skills audit As our primary context was an SLP observing an ECE, we examined whether SLPs using the ICL Skills Audit had adequate intra- and inter-rater reli-ability. Thus, inter-judge and intra-judge agreement was examined. To do this, video samples of real ECE interactions with children in a childcare setting were collected and rated by SLPs on two occasions.  Method Participants.  Three SLPs were recruited as the raters. All had previous experience working collaboratively with ECEs ranging from 1  – 6 years of post-qualification experience in non-government early childhood Table II. Six key skill areas of the Interaction, Communication and Literacy (ICL) Skills Audit. Skill area1. Developing positive and responsive adult and child interactions2. Explicit literacy instruction3. Developing storytelling4. Involving all children in a small group5. Developing peer-to-peer interactions6. Developing responsive family involvement  123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120  The ICL skills audit    5intervention organizations, private practice, and community health services. One rater was also a PhD student. All raters were blinded to the research ques-tions and none had previously used the tool or been involved in its development. Collection and preparation of research video sample  . One ECE and 17 children (aged 2  – 5 years) were recruited for the research video. The ECE was recorded in three different interactions with small groups of chil-dren. Informed consent was obtained for all partici-pants. Segments of authentic ECE  – child interactions were recorded using a digital video camera located in a quiet space within the childcare centre. All the interactions between the ECE and the children occurred spontaneously and were not scripted. The contexts recorded included transition time, music time, book-reading, meal times and playing with puzzles. The footage was combined and resulted in a 15-minute research video sample. Rating training and procedure.  The SLP raters each attended two 60 minute rating sessions separated by 2 weeks. The first session included 15 minutes of training. All SLP raters were provided with three short training segments (each between 30 seconds to 1 minute) prior to rating the research video sample. These training segments illustrated a range of behav-iours identified in the ICL Skills Audit and were selected from publicly available video samples from the Early learning resource project website (DEEWR, 2009). The SLP raters were provided with the ICL Skills Audit and the task verbally explained. Any questions regarding the rating tool or task were recorded and answered. The SLP raters then watched the training samples one segment at a time, rated observed behaviours, and compared their scores with the trainer ’ s score. Once all three training segments were completed, the raters also had an opportunity to discuss and clarify any difficulties with the rating tool or rating task. Rating the research video sample. Following the train-ing, each rater viewed the 15-minute research video sample and rated the ECE behaviours using the ICL Skills Audit. The raters viewed the entire sample without rating and then rated the sample during the second viewing. Raters were then given 10 minutes to individually complete their ICL Skills Audit rat-ings. The SLP raters rated five of the six ICL skill areas. The sixth skill area considering “ developing responsive family involvement ” was excluded from the reliability study as the video sample did not cap-ture any interactions between the ECEs and families. Raters considered all five skill areas simultaneously and did not confer with one another about their rat-ings. Raters were asked not to discuss their ratings with each other until the re-rating session had been completed; however, no check was made to ensure that the raters complied. A second rating session was conducted 2 weeks later. The three raters were pro-vided with a new copy of the ICL Skills Audit and did not have access to their previous ratings. Again, each rater watched the same 15-minute research video sample of the ECE ’ s interactions. As in the first session the raters viewed the footage twice, with 10 minutes provided to rate after the second viewing. Results of study 1  Analysis.  The intra- and inter-rater agreement of the ICL Skills Audit was assessed by calculating the per-centage close agreement (PCA) (Belsky, Burchinal, McCartney, Lowe Vandell, Clarke-Stewart, & Tresch Owen, 2007), which allows a margin of   1 scale point difference across two rating occasions. PCA was chosen as it is the most common method used to measure agreement across two rating occasions (Rey, Plapp, Stewart, Richards, & Bashir, 1987; Sheard, Adams, & Davis, 1991). In addition, PCA was chosen over Cohen ’ s Kappa as the Kappa sta-tistics make an unnecessarily large adjustment for chance agreement. The use of Kappa statistics assumes that any agreement that could be explained by chance is explained by chance. While PCA does not make an overt adjustment for chance, the likeli-hood of chance agreement is low because of the pur-poseful participant recruitment of expert SLPs. Instead Fisher ’ s 95% confidence intervals (CI) were calculated for each PCA score to obtain the range of scores rated by a population. Fisher ’ s CI was calcu-lated using WINPEPI ( It is difficult to establish minimum levels of PCA, or indeed any other reliability measure (Portney & Watkins, 2009, p. 82  – 3), and, therefore, we have not provided guidelines for interpretation of the results that follow. Intra-rater agreement.  The intra-rater reliability for each rater of the rated ICL Skills Audit is shown in Table III. The PCA ranged from 91  – 94, with a mean of 92 agreement. This indicates that the rated items Table III. Summary of overall intra-rater agreement for SLP raters. Rater 1Rater 2Rater 3Mean percentage agreementOverall percentage close agreement on all items rated on the TSA *  91949192Fisher ’ s 95% CI for overall percentage close agreement81  – 96%85  – 98%81  – 96%83  – 97%   *  The number of items rated on the ICL Skills Audit was 65. Reliability scores; Poor      40%, Fair  – Good   40  – 79%, Excellent      80%.
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks