reading responsibly: nancy kaplan on the NEA's data distortion 11.30.2007, 5:21 PM
posted by ben vershbow
Analyzing long term trends in Americans' reading habits, To Read or Not to Read argues that three important trends produce a worrying prognosis for the future of reading, and indeed the future of civic life:
1. a historical decline in voluntary reading rates among teenagers and young adults;
2. a gradual worsening of reading skills among older teens;
3. and declining proficiency in adult readers.
These three trends provide the foundations for the central argument of the report, which can be summarized fairly succinctly:
Reading books in one's leisure time strongly correlates with reading proficiency so that when rates of voluntary book-reading decline and we also see a decline in reading proficiency, it is reasonable to suppose that people become less proficient readers because they are spending less time reading books for pleasure.
The entire argument, in short, depends on the ability to demonstrate both that reading proficiency is declining and that the number of people who choose to read books in their leisure time is also declining. From those two trends, the NEA draws some inferences about what declines in reading books and declines in reading proficiency mean for the nation as a whole.
Much of the data used to support the core claims derives from statistics gathered and analyzed by the National Center for Education Statistics (NCES). In particular, data on reading proficiency at three ages -? 9, 13, and 17 -? come from the National Assessment of Educational Progress (NAEP) report on long term trends, a "nationally representative and continuing assessment of what America's students know and can do in various subject areas. Assessments are conducted periodically [since 1971] in mathematics, reading, science, writing, the arts, civics, economics, geography, and U.S. history" (http://nces.ed.gov/nationsreportcard/about/). In addition to assessing reading proficiency across three contexts for reading ("reading for literary experience, reading for information, and reading to perform a task"), the assessment also "asked students to report contextual variables such as time spent on homework, the number of pages read for school and homework, and the amount of time spent reading for fun" (TRONTR, p. 27). Data demonstrating the decline in reading proficiency among adult readers come from a separate NCES study, the National Assessments of Adult Literacy (NAAL), which has been conducted periodically since 1992.
Despite the numerous charts, graphs and tables in To Read or Not ot Read, a careful and responsible reading of the complete data provided by the NAEP and the NAAL undermine the conclusions the NEA draws. Two examples of problematic uses of primary data sets will illustrate the issues.
The graph below shows the original data as it is displayed on the NCES web site (http://nces.ed.gov/nationsreportcard/images/ltt/ss01.gif) with the NEA's truncated and reformatted version superimposed above the original:
Although the data represented in the NEA version are strictly speaking accurate, they nevertheless seriously distort the data set from which they were derived in two key ways: by truncating the data set and by representing irregular time intervals with regularized spatial intervals. The first distortion creates a trend where none exists. The second distortion magnifies the effect of the decline in scores by making the slope of the line between the scores in 1999 and the scores in 2004 steeper than it should be. The steeper slope, then, suggests a more rapid and deeper decline than the underlying data support.
Note that the NEA graph begins with the year 1984 while the data set from NCES begins in 1971. Note too that the average scale score for 17 year olds in 2004 -? 285 -? is exactly the same as the average scale score for that age group in 1971. In other words, over the whole period for which data are available, there has been no change in reading proficiency among 17 year olds, although there was evidently a brief period of significant improvement between 1984 and 1992 (the asterisks mark statistically significant differences with the 2004 score). In short, there is no downward trend in reading proficiency over the whole period for which we have data. The downturn that did occur after scores peaked from 1988 through 1992 is statistically significant but it is on the whole not very steep nor particularly precipitous. In fact the magnitude and duration of the decline mirror the statistically significant uptick in scores over the four year period from 1980 to 1984.
A second graph produced by the NEA and used both in the executive summary and in Chapter 5 of the report highlights and magnifies both kinds of distortions. The graph uses a truncated set of the NCES data for two age groups -? 17 year old students and 9 year old students -? to exaggerate the difference in trends between the two groups. The underlying data are represented on the NCES graph at the top of the illustration. The green dots and line represent the relative changes in scores for 17 year olds from 1988 to 2004 and the blue line represents the relative changes for 9 year olds. The horizontal red line is the base 1988 score for each group. The 9 year old groups' score is 7 points higher in 2004 than it was in 1988 while the 17 year old groups' score is 5 points lower. In the NEA graph for these same data, the scale has been altered to create the impression that score differences are much greater than they actually are. The dramatic improvement in the scores of 9 year old readers is statistically significant, as is the 5 point decline in scores for 17 year olds over the period shown in the graphs. Yet in the NEA revision of the NCES graph, the data points for each test year are spaced evenly across the x-axis even though the tests occurred at uneven intervals. As a result, the changes in scores for both groups are made to seem much more dramatic and sudden than they really are.
Misleading graphs based on manipulated data are not the only fudge factor the NEA employs. In addition to relying on two sources of NAEP data, To Read or Not to Read also uses data from the National Assessment of Adult Literacy whose latest report, Literacy in Everyday Life, provides a nuanced and detailed analysis of adult literacy in America over 11 years (1992-2003). Despite the complexity of its subject and the detailed data it provides, the NAAL report begins with a clear and simple assertion: "between 1992 and 2003, there were no statistically significant changes in average prose ... literacy for the total population ages 16 and older..." (p. iv). Even though the overall picture did not change, the NAAL did report some data that, when removed from the larger picture, appear to suggest declines in adult proficiency. As the NEA notes, proficiency among adults with bachelor's degrees and post-graduate degrees declined in statistically significant measures. But the NAAL report explains the apparent contradiction by noting that underlying demographic changes account for the fact that overall proficiency did not decline while levels of proficiency among the more highly educated appear to have decreased:
The fact that average prose literacy decreased or remained the same for all levels of highest educational attainment raises an interesting questiion. How could prose literacy scores decrease at every level of education beyond high school without a decrease in the overall score? This pattern is called Simpson's Paradox. The answer is that the relative size of the groups changed. From 1992 to 2003, the percentage of adults with postsecondary education increased and the percentage of adults who did not complete high school decreased. The increase in the percentage of adults with postsecondary education, who, on average, had higher prose scores than adults who did not complete high school, offsets the fact that average prose literacy scores declined at every level of educational attainment beyond high school. (p. 37).
Among other things, the NAAL report finds that various demographic factors, especially an adult's first language and the age at which that person learned English, have significant effects on proficiency with literacy in English. A quick look at changes in population over the same period provides a reasonable hypothesis to explain the NAAL data. Over the period measured in the latest report, the US has experienced large increases in immigrant populations. For example, in 1990, 7.9% of the total US population were foreign born; in 2000, the 11.1% of the population were immigrants. The overwhelming majority of foreign born residents of the US (97% of naturlized citizens and 84% of all other foreign born residents) are adults. Such changes in the make-up of the population might have important effects on the data.
Data may be one kind of thing but polemic is another. The heart of the NEA's case appears not in the report proper but in preface provided by the Endowment's Chairman, Dana Gioia. There he explicitly asserts that reading books, preferrably every day, produces more prosperous and more virtuous citizens:
Strictly understood, the data in this report do not necessarily show cause and effect. The statistics merely indicate correlations. The habit of daily reading, for instance, overwhelmingly correlates with better reading skills and higher academic achievement. On the other hand, poor reading skills correlate with lower levels of financial and job success. At the risk of being criticized by social scientists, I suggest that since all the data demonstrate consistent and mostly linear relationships between reading and these positive results - ?and between poor reading and negative results -? reading has played a decisive factor. Whether or not people read, and indeed how much and how often they read, affects their lives in crucial ways.
All of the data suggest how powerfully reading transforms the lives of individuals - ?whatever their social circumstances. Regular reading not only boosts the likelihood of an individual's academic and economic success - ?facts that are not especially surprising - ?but it also seems to awaken a person's social and civic sense. Reading correlates with almost every measurement of positive personal and social behavior surveyed. It is reassuring, though hardly amazing, that readers attend more concerts and theater than non-readers, but it is surprising that they exercise more and play more sports - ?no matter what their educational level. The cold statistics confirm something that most readers know but have mostly been reluctant to declare as fact - ? books change lives for the better.
There is little doubt that modern information economies require many more proficient readers than older industrial economies did. Because of changes in the nature and conditions of work, declining proficiency in reading among American adults might cause some concern if not alarm. It is surely also the case that educational institutions at every level can and should do a better job. Yet there is little evidence of an actual decline in literacy rates or proficiency. As a result, the NEA's core argument breaks down. Even if we assume that high school seniors in 1971 spent more of their leisure time reading books than today's high school seniors do (although there is no data going back far enough to support the case one way or the other), there simply is no evidence that today's youngsters don't read as well as Mr. Gioia's peers did at a comparable age. From the information available, we simply cannot construct any relationship, let alone a causal one, between voluntary reading of books and reading proficiency.
Reading well, doing well, and doing good may exhibit strong correlations but the underlying dynamics producing each of the three effects may have little to do with what Americans choose to do in their leisure time. Read responsibly, the data underlying the NEA's latest report simply do not support Mr. Gioia's assertions.
Like many other federal agencies under our current political regime, the National Endowment for the Arts seems to have fixed the data to fit its desired conclusions.
Matt K. on November 30, 2007 11:05 PM:
This is very important, and needs to be widely read and circulated.
Dale Lipschultz on December 3, 2007 10:08 AM:
This is an excellent, well researched, and clearly articluated analysis of the issue of declining literacy rates, the use of primary data, and the impact of limited reading for pleasure. As ALA's Literacy Officer and former president of the National Coalition for Literacy, I applaud your efforts and will share your comments with my colleagues throughout the literacy field.
Andrew W on December 3, 2007 11:30 AM:
Ha, the ironic thing is that for the report's data to be believed, the reader would have to have bad reading comprehension skills.
ken Dowlin on December 5, 2007 9:57 PM:
I was concerned about the conclusions from the report since there seems to be no consideration for the fact that use of a computer or other communications devices such as text messaging requires the person to be able to read. I suspect that the traditional measurements are totally based on reading traditional materials. We don't know how to factor in the new media.
ReHeated on December 6, 2007 10:34 AM:
Really good post - interesting from a UK perspective given the recent flap over the report of the International Literacy Study.
Slightly off track, but I thought you might be interested in my post on reading and the brain. A few speculative thoughts...
Matt K. on December 6, 2007 9:13 PM:
The report repeatedly asserts that it takes "screen reading" into account; the exact nature of the methodology is not always clear. See my piece in the Chronicle (link a few posts further down) for some more discussion of this.
Stephens Mitchell on December 10, 2007 2:54 PM:
The ability to read, comprehend, and recall long passages is a self-evident measure of reading proficiency. The printed book takes this measure to its logical endpoint. Onscreen reading takes it precisely nowhere. Onscreen reading is for the most part an undisciplined, easily distracted process whereby the brain is assaulted with materials produced under sub-optimal editorial oversight, accompanied by the hoots of a puerile peanut gallery, and having very little in the way of enduring value. Witness if:book. That the outcome could be anything other a disaster is an interesting hypothesis.
bowerbird on December 10, 2007 5:21 PM:
ok, well i guess that sums it up! :+)
sebastian mary on December 11, 2007 11:55 AM:
Talking of 'puerile peanut gallery':
"Now thousand tongues are heard in one loud din:
The Monkey-mimicks rush discordant in.
'Twas chatt'ring, grinning, mouthing, jabb'ring all,
And Noise, and Norton, Brangling, and Breval,
Dennis and Dissonance; and captious Art,
And Snip-snap short, and Interruption smart.
'Hold (cry'd the Queen) A Catcall each shall win,
Equal your merits! equal is your din!
But that this well-disputed game may end,
Sound forth, my Brayers, and the welkin rend.'
Pope, Dunciad Variorum (1729), II.228-36
Perhaps it's time for another Essay on Criticism...?
Stephens Mitchell on December 11, 2007 8:14 PM:
Having catalogued them all, Pope provides a handy bludgeon for any sort of critic. Peanut gallery also is a palpable hit. For puerile, allow me to substitute dyspeptic.
The point remains that we are all becoming idiots, more enamored of the latest medium, than of the quest for truth. The notion -- and you will grok the allusion -- that a sitcom/advert could have anything to do with a Renaissance is preposterous. It's on a par with the idea that reading proficiency will rise in proportion to the number of text messages we ingest, rather than the reverse.
sebastian mary on December 12, 2007 5:55 AM:
It's no more preposterous than the invocation of some idealised past populated by earnest Leonard Basts reading Chaucer and Dickens over their bread-and-dripping. The kind of long-form literacy under discussion here has always been an minority activity. The only things that have changed are 1) that this kind of cultural production is no longer privileged as it once was (about which I have very mixed feelings), and 2) in tandem with this change we've seen a rise in mass, short-form quasi-literacy.
Just because you don't like the form a culture is taking doesn't mean it won't go ahead without you.
Stephens Mitchell on December 12, 2007 11:48 AM:
No such thing was invoked. In a real past, a larger fraction of people than today read long works, as NEA puts it, "voluntarily".
As evidence, you may disregard the NEA, or for that matter, the NCES. The year 1971 has never been mistaken for a high water mark in American literacy.
Since you mention Dickens, consider his ability to make a living, and more than a living, writing serial novels. A mass public paid to read his long works. These works were composed of sentences, which themselves were so long, as to require a Dickensian number of commas. Yet people read and understood them, and clamored for more. (Whether this occurred over bread and drippings, is not for me to say.)
If a comparable market existed for "networked novels", the authors of Flight Paths would not be still be casting about for ways to "monetise" their "exciting new project". It's not as if e-commerce or the subscription business model were in their respective infancies.
That the future will come, with or without me, is hardly worth stating. What the rise of a kleptomaniac, short-form, quasi-literacy has to do with either an artistic Renaissance, or the future of the book, is a mystery. Perhaps you could elaborate.