Studies in Educational Evaluation.
Vol. 12, pp. 225233, 1986 0191491X/86 $0.00 + .50 Printed in Great Britain. All rights reserved. Copyright © 1986 Pergamon Journals Ltd.
A NATURALISTIC METHOD FOR ASSESSING THE LEARNING OF ARITHMETIC FROM COMPUTERAIDED PRACTICE Nira Hativa
Tel Aviv University, Tel Aviv, Israel
In the last few years, computers have substantially penetrated schools. In elementary schools in the U.S.A. they are used primarily for drill and practice (Becket, 1984). This type of computer application has been shown to produce significant achievement gains in mathematics and, to a lesser extent, in reading and language arts (e.g., reviews by Kulik, Kulik and BangertDrowns, 1985; Burns and Bozeman, 1981 and Hartley, 1977). All the tens of studies summarized in these reviews used rationalistic methods of inquiry with statistics done on a large number of students. But statistical methods provide only general informa tion and often overlook problems of some individual students working with com puters. Guba (1981) suggests using the naturalistic method of study as comple mentary to the rationalistic method. Following Guba's suggestions, the author conducted a study using the naturalistic method of inquiry in order to investi gate the CAI (ComputerAssisted Instruction) contribution to students' perfor mance in arithmetic and to identify possible problems in this context. The study was aimed at understanding the holistic environment of students' individu alized drill in arithmetic with the computer. The holistic approach assumes that a description and understanding of a program's context is essential for understanding the program (Patton, 1980, p. 40). The naturalistic paradigm rests on the assumption that there are multiple realities, that inquiry will diverge rather than converge as more and more is known, and that all "parts" of reality are interrelated so that the study of any one part necessarily influences all other parts. (Guba, 1981, p. 77). To comply with naturalistic methods of inquiry, a variety of sources were used for data collection. These were: observations; interviews with students, teachers, parents and siblings; questionnaires for teachers; weekly reports of students' performance with their computer work; and paperandpencil tests. Of several types of methods for doing naturalistic study, the writer chose the multisite multisubjects maximumvarlety method (Bogdan and Biklen, 1982, p. 65). This approach is oriented toward developing theory and attempts to maxi mize the variation in site and subject selection. By attempting to increase the diversity of variation in the sample, the evaluator will have more confidence in those patterns that emerge as common among cases since it is possible to more accurately describe the variation in the program and to understand variations in experiences (Patton, 1980, p. 102). The schools, classes, and students for this
225
226
N. Ha va
study were chosen so as to satisfy the maximum variation requirement: 18 students were chosen for observations from five schools. The schools were from low to upper SES populations; the classes were from 2nd through 6th grade  all grade levels that used the CAI system; and the students were boys and girls of all range of ability level. Each student was observed from 24 through 42 ten minute consecutive computer sessions. THE CAI SYSTEM AND THE METHOD OF OBSERVATION The CAI system observed in the study is widely used in elementary schools in Israel and in several other countries, e.g., Spain, South Africa and the U.S.A. It was adapted in 1975 from an American system marketed in the U.S.A. by the Computer Curriculum Corporation (CCC) of Palo Alto, CA. The Israeli system is designed for all 2nd through 6th grade students in a school. The software consists of a large body of timed exercises encompassing the arithmetic curri culum and grouped into 15 strands. Each strand concentrates on a single arith metic topic, e.g., vertical addition. The exercises within each strand are arranged in an hierarchical sequence of difficulty. Each type of exercise is assigned a number which indicates the gradeequivalent of the exercise, e.g., exercise of level 34 is designed to be dealt with in the fourth month of the third grade. Each student gets randomly mixed exercises from all strands that are "active" for him/her. Students' work with the CAI system is fully managed by the computer with almost no teacher intervention. The software diagnoses each students' initial levels across strands; makes decisions as to the student's going up or down a level as the result of respectively satisfaction or dissatisfaction of specific mastery requirements; and provides feedback to students and teachers. A more detailed description of the CAI system is pro vided by Osin (1984). The method of observations was as follows: The observer sat next to the student in the computer lab in all computer sessions throughout the observation period. The observer copied from the screen each exercise, the student's answer and the computer response. When needed, the students were briefly interviewed at the end of a session. They were asked about specific questionable answers or errors that they made in the session. A few longer interviews were conducted outside the computer lab and at times included paperandpencil tests. Several of the classroom arithmetic lessons were also observed. The following analysis is based on data collected through all these procedures. THE UNIQUE CONTRIBUTATION OF THE NATURALISTIC METHOD FOR ASSESSING STUDENTS' LEARNING OF ARITHMETIC VIA THE COMPUTER The use of naturalistic methods as described above enabled the writer to identify and investigate several phenomena and processes involved in the lear ning of arithmetic through CAI drill and practice. These processes were not identifiable through regular rationalistic methods of inquiry. The main contri bution of the naturalistic approach of inquiry to the findings of the present study may be sorted into four categories: (a) investigating the continuous process of the learning of arithmetic with the CAI system and identifying students' problems in this process; (b) investigating students' strategies for learning in this special context; (c) explaining processes and problems in this context; (d) identifying misconceptions in arithmetic and processes of "learning without understanding". Next is a discussion of each of these categories.
LEVEL
. f I I
4
#~:
LONG
DIVISION
S2
51 4S
#8 : COMPU'r,~'I'I ON 8Y ME,~,P.'I
r
A t. ~'JE~
Sl~% 0 s ME~SU~EMEN'I"
I
~
.......... ~ ~
r
481,
4m ~  I
1 5
47
#9
:
Erou~,'rl Or.s
' ' ~
So
..........

4S 44 ) I
33 35 ~T ~HI 41 4,1 45 47
41
IS SS Sq ST SO
L~'SSON NO.
4I #I I s FRACI ] ONS
ssL #1 4 : WOt~.O P~O~LEMS ~ol
~ ) ~
_ _
. r
4Nr.~, ,, ~,.o..~.., ~ ~ T
' f
i '
~3 14 3 7 3.@ 41 1.3 4S 47
49
$1 53 55 5'7 59
LE:SSON NO.
~I'Jl Sl BI L.I'FY RU~ES, ~'A CI" O R ] N G ~NO
F> Ol,,.I I= I~. S ,..,. ~
Graph based on the observations ...... Graph based on the computer reports FIGURE i: One Student's Performance Across All Strands Active for HerGraphs Based on Observations and on Computer Reports
228
IV
Hativa
(a) Investigating the Continuous Process of the Learning of Arithmetic with the CAI System and Identifying Students' Problems in this Process Each class teacher gets a computergenerated class report which shows for each student the performance per strand and the mean performance across strands. These class reports are designed to inform the teacher on the advance ment of each student in the learning material. This should serve as a diagnosis for students' problems and also for evaluations of students' learning of arith metic through the CAI system. But each of these reports provides information on an instantaneous situation  the standings of the students at the end of the last computer session  whereas our continuous observations provide a full picture for each student's performance. To illustrate, Figure I presents two graphs of performance of one of our observed students across all strands that were active for her during the observation period. The solid lines in Figure I are based on the levies of performance recorded in the observations. The seg mented lines simulate information provided by her class computer reports. This latter graph has been produced by combining the levels at the end of every fifth session. In practice, the reports were produced once in 12 weeks, i.e., once in 48 computer sessions. Eyeballing both types of graphs in Table i, it is evident that the solid graph presenting the observations provides much more information on students' performance than the segmented graph presenting the computer reports. The observationbased graph shows problematic features in the student's learning such as going down one level (e.g., Strands nos. I0 & 14) or two consecutive levels (Strands nos. 6 & 7) or oscillating lengthily between two consecutive levels (Strands nos. g & 15). The graph based on the computer reports does not reflect appropriately such problems. Several of the other observed students showed even greater problems of going down more than two consecutive levels. Figure 2 presents the performance of two other students, each on one strand, that showed this type of problem.
LEUEL #1 4 a I..JORD PROBLEMS
34~' I LJI 3~ I
T
),

,o ~
ll__l
,,
I .I
28 I I LESSOt~I I',~10 .
i i i I iJj I i __ uJ~ i i I i . ~ i l 1 ~ 1 ~ i iJ I , * I lj J ~_
l} ,$ ,I I If I) ~ II I I I I~ $~' $I 41 II IS II 4 I 
13 15 20 30 40 50 53
I ,,J3, l~I 3 : SI GLqED NUIvlBERS
5,, __JJLLL
52
,,
 I L_/7 I L E S S ON
" ii it Ii I~ II H M P fl II ~ ~ H II Ii tl M l II II II
NO.
16 18 20 30 40 50 56
FIGURE 2: Decrease in More Than Two Levels in Students' Performance
Naturalistic Method
229
The feature of going down two or more consecutive levels means that students who have proved a one time mastery of exercises of a certain level prove nonmastery of the same type of exercises when encountered again. This points on superficial learning or on learning without understanding at the previous stages so that the knowledge acquired has not been "digested" well enough to be remembered for a long duration of time. The continuous observations enabled the writer not only to identify prob lems expressed in decrease in levels of performance but also to identify the causes underlying those problems. The following is a description of some of these causes. Memory Problems. The mode of work with the CAI system observed in this study does not allow to use paperandpencil for computations and students have to make all computations by heart. The exercises are planned so that the averageability student does not face too much difficulty in making the required computations mentally, though young or belowaverage performing students face problems in having to calculate more than one step by heart. For example, many of the errors observed with these students in vertical operations with carry over digits resulted from the requirement to remember the carryover digit by heart rather than writing it down. To further examine this point, all observers gave paperandpencil tests to the observed students. Results of these tests support the latter claim. All our mediumtolow achieving students performed on paperandpencil with writing the carryover digit much better than with the computer, on the same type of exercise. Good memory is needed also for remembering the exact order required for entering the answer for each kind of exercise. For example, an addition or subtraction exercise is to be typed lefttoright when it is in horizontal form and righttoleft when in vertical form, see Table i (the question mark appears in place of the digit to be typed). TABLE i: The order of entering an answer in horizontal and vertical addition Horizontal Addition Vertical Addition Given: 25+80 = ? Step i: 25+80 = i? Step 2: 25+80 = i0? Step 3: 25+80 = 105 (First type i, then O, then 5) Given Step 1 Step 2 Step 3 25 25 25 25 +80 +80 +80 +80 ? ?75 ?05 105 (First type 5, then 0, then i) Many students do switch the required order as presented in Table i. The computer, checking the first typed digit, counts that digit as an error which, in turn, results in recording the whole solution as incorrect. A second example is illustrated by the order required for entering the answer in vertical multi plication when it is easy to compute the answer mentally. Table 2 illustrates the required order for writing the answer to the multiplication of 35 by 20.