Brookings Institution: Never judge a book by its cover—use student achievement instead

In February and March of last year, as teachers were preparing for the first administration of the end-of-year assessments from the Partnership for Assessment of Readiness for College and Career (PARCC) and the Smarter Balanced Assessment Consortia (SBAC), the Center for Education Policy Research at Harvard surveyed a representative sample of 1,500 teachers and 142 principals across five states.[i] Our report focused on the instructional changes teachers were making and the effectiveness of the training and support they were receiving. However, we also asked about the textbooks they were using.

We matched each teacher to the students they were teaching and assembled data on students’ demographic characteristics, performance on prior state tests, and the averages of such characteristics for the peers in their classroom. We also estimated each teacher’s impact on student performance in the prior school year (2013-14) to use as a control. (We wanted to account for the fact that more effective teachers may choose to use particular textbooks.) After controlling for the measures of student, peer, and teacher influences above, we estimated the variance in student outcomes on the new assessments associated with the textbook used.[ii]

The textbook effects were substantial, especially in math. In 4^th and 5^th grade math classrooms, we estimated that a standard deviation in textbook effectiveness was equivalent to .10 standard deviations in achievement at the student level.[iii] That means that if all schools could be persuaded to switch to one of the top quartile textbooks, student achievement would rise overall by roughly .127 student-level standard deviations or an average of 3.6 percentile points. Although it might sound small, such a boost in the average teacher’s effectiveness would be larger than the improvement the typical teacher experiences in their first three years on the job, as they are just learning to teach.

As advocated by Russ Whitehurst and Matt Chingos, the search for more effective curriculum materials can yield outsized bang-for-the-buck, because schools are already buying textbooks and better textbooks do not cost more on average than less effective ones. We estimate that such a study would need to collect data from roughly 1,800 schools to have the statistical power necessary to detect a .10 impact for any textbook representing at least 10 percent of the market. Some states, such as Indiana and Florida, already collect data on textbook adoptions, but even if it were necessary to collect data from a new sample of schools, the study would likely cost less than $2 million annually.[iv] Across the PARCC and SBAC states, there are approximately 3 million 4^th and 5^th grade students. Assuming the more effective textbooks have a similar cost to the textbooks they replace, the incremental cost of the 3.6 percentile point gain in achievement would be the cost of the study itself—roughly 67 cents per student.

[ii] We dropped the 20 percent of students whose teachers reported they did not use a textbook, an additional 30 percent whose teachers did not use their textbook as their primary curriculum, and limited the sample to textbooks that at least two teachers reported using.