Technology Reading #5 — Does it Compute (NAEP Result Analysis)

Wenglinsky, H. (1998). Does it compute? The relationship between educational technology and student achievement in mathematics. Princeton, NJ: Educational Testing Service. Retrieved March 6, 2002, from ftp://ftp.ets.org/pub/res/technolog.pdf.

Preface: 

In this age of technological imperative, we do it simply because it can be done. Massive efforts are underway to convert traditional teaching to something that can be delivered via computer.  Measuring success has been a simple matter: count the number of computers, divide that by the number of students, and report how the ratio of computers to students has advanced-and it is always advancing. Then close the report lamenting that we don’t know much about the software being used on these computers, we don’t know how many are behind locked doors, we don’t know how many are broken, and we don’t know how many teachers really know how to use them (there are no assessments of teacher capability here). 

The Policy Information Center reported in May of l997 what we did know at that time, in Computers and Classrooms. We said then that the available data, and the research performed on it, did not tell us whether computer delivered instruction actually improved performance.  This report takes at least the first step in determining whether computer use is making a difference in mathematics, and what kind of computer use has what kind of effect, on which groups of students. 

This study uses a national database, the 1996 National Assessment of Educational Progress, and advanced analysis techniques, to isolate the effects of the computer from the myriad other factors involved in student achievement. The study was suggested to us by Education Week, and its author, Harold Wenglinsky, of the ETS Policy Information Center, has collaborated with the staff of Education Week throughout the analysis and writing phases of the report.

In addition to telling us what he found, Harold Wenglinsky also tells us what further research must be conducted to learn more, and give the nation specific guidance in its efforts to raise educational achievement through greater use of technology.

RQs

1. Does computer use make a difference in mathematics?
2. What kind of computer use has what kind of effect on which groups of students?

This report presented findings from a national study of the relationships between different uses of educational technology and various educational outcomes.  Data was drawn from the 1996 National Asssessment of Educational Progress (NAEP) in mathematics consisting of national samples of 6,227 4th graders and 7,146 8th graders.  The NAEP consists of tests in various subject areas administered to national samples of students, usually every 2 years.  It is used to measure trends in student performance overtime and between subgroups.  It also includes questionairres given to students, their teachers, and principals to put test scores into their educational context.  In 1996, it was the first time NAEP included questions on technology.  NAEP includes information on: frequency of computer use for math in school, access to computers and frequency of computer use in the home, PD of math teachers in computer use, and kinds of instructional uses of computers by mathematics teachers and their students(higher-order v. lower order).  Those four categories serve as the indicators for the organization of technology use and are related to 2 educational outcomes (academic achievement in math and social environment), along with each other.

The study used advanced analysis techniques and specifically examined the subgroups of ethnicity, gender, region, public v. private, economic status, and community status (rural, urban, etc.).  The study found that essentially technology could matter, but it depends on how the technology is used.  It also found that there is great inequity in how computers are used.  Specifically, the study found that (for 8th graders):

• Black students were less likely to be exposed to higher-order use of computers and more likely to be exposed to lower-order uses than White students.  Similarly, so were poor, urban, and rural students as opposed to non-poor, suburban students.
• Urban and rural teachers were less likely to have mathematics teachers who had received PD in technology over the last 5 years than suburban students.  Similarly, the same goes for poor vs. non-poor.
• Black students were less likely to have access to a home computer than White students, and this was similar for poor and urban students as well.

Specifically regarding academic achievement, the study found that:

• Teacher’s PD in technology and the use of computers to teach higher-order thinking skills were positively related to academic achievement.
• The frequency of home computer use is positively related to academic achievement.
• The use of computers to teach lower-order thinking skills was negatively related to academic achievement.
• The frequency of computer use was negatively related to acedemic achievement.  “DON’T JUST USE TECH FOR TECH SAKE!”

It is important to note some methods of this study.  First, “frequency of computer use” is based on student reports and ranges from “never of hradly ever’ to “almost everday”, with the option of “there is no computer at home” for the home use questions.  PD is based upon asking teachers whether they have received such development in technology, specifically computers, in the last 5 years.  For 8th graders, high-order use is measured as “simulations an applications” and lower-order uses are measured as ”drill and practice”.  Three characterisitcs of students and schools are taken into account: SES, class size, and teacher characteristics.   This means that all relationships between technology and educational outcomes reported represent the value added by technology for comparable groups.

It is apparent from this study that there are clearly equity issues within our educational system that include the use of educational technology.  It appears that there exist somewhat of a socioeconomic digital divide.  Disadvantaged groups seem to lag behind in the aspects of technology that do affect educational outcomes.  The answer starts with education at the top.  Students with teachers who have PD show higher levels of achievement.  Policymakers in all sectors involved with education should focus on adminstering educational technology in an equitable fashion.  This includes not just the equipment itself, but educating teachers in order to educate students.  Teachers need good PD (including applicable and ability to attend) in the areas of higher-order thinking and technology use and thus incorporating it into the classroom. 

Some limitations of this study are:

• Data was collected at a single point in time, and thus certain subgroups were not followed over a period of time to track results.  Additionally, aspects of technology studied here occurred at the ssame time as the educational outcomes of interest.  And so it may be that high-acieving students are more liely to use technology in certain ways rather than that these uses of technology promote high levels of academic achievement.
• The study does not take into account the tendency of teachers to teach in certain ways or detailed measures of teacher practices.  Thus it may be that computers are more of a medium conducive to high academic achievement. 
• The study does not take into account information about state technology policies, which could have a bearing on effectiveness.  It has been noted that states on left on their own in how to implement educational technology.