Diane
Ravitch confirms my interpretation of NY's NAEP performance in Fed
test scores show state grade inflation is real.
Of
course, there are multiple interpretations in addition to the one
discussed below. One is that teaching to the test may dramatically
improve scores on one test but not on another. If that's the case,
one must ask whether students are any better educated now than in
the past.
NY's performance on national math and reading tests is not nearly
as impressive as on the state's own tests. In general, half
as many students pass the national tests and the rate of
improvement is half that reported by the state Education
Department.
In a long series of commentaries on NY's exams,
culminating in "Nine
Commentaries on NY K-12 Exams: Testing to the Results," I
have warned that NY's academic performance was not increasing as
rapidly as claimed in press releases. Changes in exam
content, cutoff scores and scale scores, among others, created the
appearance of improvement, which I estimated to be only about half
the amount advertised. NY's NAEP results substantially
confirm this.
Although the Education
Commissioner, Richard P. Mills, has championed the standards
movement which has somewhat, though far from greatly, improved
academics, he has let his responsibility to lead and motivate
public schools trump his duty to be completely honest and
forthcoming about the causes for rising test scores.
While always declaring "we have a long way to go," he
generally attributes rising scores to the hard work of teachers
knowing that other factors are in play, which, if publicly known,
would likely undermine confidence in the results reported.
This must stop. (See, e.g., Test
scoring to change for Del. schools). When changes are made to exams, cutoff
scores or scale scores, these changes must be explained in the
press releases that report exam results. The people who pay
the bills, students and educators deserve to be told the whole
truth and nothing but the truth.
In this first
graph, the average pass rate (levels 3 & 4) on the state
math and English exams of 4th- and 8th-graders is compared to the
average pass rate (proficient and advanced) of the same grades for
New York students on
the national math and reading tests. The graph confirms that
student performance on these exams has improved over time,
although NY's average pass rate on the national tests showed no
improvement over 2003 scores despite the increased average
pass-rate to 61.6% from 55.3% (during 2002 to 2004) on state
exams.
The
graph also shows two important differences between the state tests
and the national tests. The most obvious is that about twice
as many students pass the state tests as pass the national
tests. An average of 64.7% passed state exams in 2005
compared to 33.5% on the national exams. To some extent,
this reflects a difference of opinion on what students need to
know--or at least on how they should be tested and graded. More
troubling, however, is the difference in the rate of
improvement. State scores are increasing 2.2 times faster
than national scores, as shown by the slopes of the lines (0.0245
on the state trendline versus 0.112 on the national
trendline). Given the strong evidence of scoring changes on
state exams and the acclaimed "gold standard" status of
national exams (without any evidence of scoring manipulation, to
my knowledge), it seems reasonable to conclude that student
performance on state exams is not improving nearly as quickly as
exam results indicate. It's true that I estimated all but
two of the data points on the national exam, but this was simply
done by assuming a constant rate of improvement during the years
when either the math or reading exams weren't given. If
anything, this would unduly improve the R2
of the national trendline (93%), which is less than the R2
of the state trendline (97%). Both suggest high confidence
that the observed improvement is real, which isn't being
questioned here.
The next graphic is for 4th
grade math. It's much the same story as for the overall
pass rates. State performance is more than twice as high as
national performance by New York students, with the rate of
increase on the state exam being 70% faster than that of the
national exam.
The
graphic for 8th grade math is even
worse than for 4th grade. Although the state pass rate is a
little less than twice the national pass rate, the rate of
improvement on the state exam is 3 times that of New York students
on the national exam. This strongly suggests that state
changes to this exam and its scoring have been largely responsible
for the increase in scores on the state exam.
The
4th grade English exam is
even worse than the 8th grade math exam. State performance
is twice that of national performance for New York students, and
improvements are 3.6 times faster on the state exam than on the
national exam. The national exam results show that 4th-graders
haven't made any improvement since 2002.
The
most reliable of all the state's exams (pertaining to both pass
and improvement rates) is the last--the 8th
grade English exam. Both sets of tests show a very small
amount of improvement, and the low R2 values reflect a
lack of confidence about whether any real improvement has
occurred. On these tests, the state pass rate comes closest to
matching the pass rate reported on the national test, though it's
still several percentage points above the national pass
rate.
It's
true that I have taken particular issue with this exam and its
scoring, but the thrust of that criticism has been that scoring
changes have made it easier for students to escape Level 1 (below
basic) and enter Level 2 (basic). The above graphic deals
only with Level 3s and 4s (proficient and advanced), so most of
the scoring changes made by the state would not show up in this
graphic. However, I note that the percentage of students passing
has risen from 44% in 2002 to 48% in 2005 and even this small
increase could be related to changes in the exam and its scoring.
Conclusion
New
Yorkers should be especially cautious in relying on state
Education Department data to assess the strength of our students'
academic abilities and the rates at which they are increasing--if
at all. Unless the state's press releases specifically
explain exam changes, or disclaim making any changes, the amount
of any reported improvement or decrease in exam scores should be
taken with a grain or two of salt. Under NCLB, the state has
strong incentives to show annual yearly progress to avoid federal
sanctions and bad publicity for public schools. It is way
too easy to make a statistical change here and a scoring change
there to create the appearance of progress, especially since no
one can see it happening and the state has no incentive to tell you
about it. If past practice is a guide to future conduct,
only about half of improvements on state exams may be coming from
improved learning in classrooms. The other half will be
coming from what I call institutionalized cheating. It's as
unethical and as manipulative as it sounds.
