Can New York Clean Up the Testing Mess?
Two top education officials tackle rampant score inflation.
As New York States executive and legislative branches sank into a swamp of corruption and political paralysis this winter, something brave, honest, and totally unexpected took place in one Albany office. Pounding the table and refusing to accept any more excuses, the new chancellor of the Board of Regents, Merryl Tisch, forced the states notoriously dysfunctional department of education to submit to an outside audit of the reading and math tests, mandated by the federal No Child Left Behind (NCLB) act, that it has administered to all students in grades three through eight. A memorandum of understanding between the department and Harvard professor Daniel Koretzone of the nations top testing expertsgives Koretz access to the data that hell need to determine whether New Yorks test scores have been inflated. And once he does, its likely that the states claims of spectacular student progress will be revealed as an illusion.
Such a development would be healthy for education reform nationally, because all states need to come clean about test-score inflation. Reliable tests of student achievement are as essential for improving education as accurate monitoring of blood-sugar levels was to advancing the treatment of diabetes. Unfortunately, when NCLB became law, it left the door wide open to massive test inflation by stipulating that all American students will be proficient by the year 2014and imposing a series of increasingly onerous sanctions on districts and schools not moving toward that goalyet allowing each state to develop its own tests and set its own standard for proficiency. Since men are not angels, it was inevitable that some state education authorities would lower the proficiency bar to make themselves look good to the feds.
Now the Obama administration has launched the most expansive (and expensive) federal school-reform initiative in American history. Like NCLB, the initiative judges teachers, schools, and states by improvements in students test scores. But testing could be the Achilles heel of Obamas reform agenda unless states like New York, where sudden student improvements strain all credulity, shape up.
For more than two decades, mainstream American social science has recognized that accountability schemes like NCLB can lead to fraud and distortion. The principle even has a name: Campbells Law, after Donald Campbell, one of the greatest American social scientists of the twentieth century. In one study, Campbell observed various companies attempts to improve employees performance indicators by giving them incentives. He came up with this general formulation: The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
Campbell even extended the law to the realm of education testing, writing that when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways. In plain English: when school districts, prompted by systems like NCLB, offer teachers and administrators substantial incentives to raise students test scores, the teachers and administrators will be tempted to find extraordinary meansbut not always ethical onesto get the test scores up.
Thats exactly what seems to have happened. The best evidence of test-score inflation is the growing gap between the number of students that states deem proficient on their own teststhose they administer under the terms of NCLBand the number deemed proficient by the National Assessment of Educational Progress (NAEP), often referred to as the nations report card. One reason the federal NAEP tests are the gold standard in student assessment is that they cant be gamed by teachers or administrators. Every two years, NAEP math and reading tests are given to a statistically valid sample of all fourth- and eighth-grade students in each state; teachers arent able to teach to the test, and school districts cant offer students practice tests because no one knows ahead of time which students will be tested. And in nearly every state in the union, the NAEP exams deem far fewer students proficient than the states own exams do.
An even more disturbing phenomenon: states reporting huge gains on their own tests while their NAEP results dont budge. One glaring instance happened recently in New York, where the percentage of eighth-graders reaching proficiency on the states math test rose from 58.8 percent in 2007 to a stunning 80.2 percent in 2009, while over the same period, the NAEP math scores for the eighth-graders remained flat. On the states fourth-grade reading exams, the proficiency rate went up from 68 percent in 2007 to 76.9 percent in 2009, while the NAEP test again showed no gain. On the states eighth-grade reading test, the proficiency rate went from 57 percent to 68.5 percent, while the NAEP tests showed a 1 percentile-point gain. Its hard to escape the conclusion that states are gaming their testsnot only to satisfy NCLB but also to celebrate the education miracles that theyve supposedly worked.
Not every state has succumbed to the temptation, thankfully. Massachusetts made an honorable effort to maintain high academic content standards and then carefully aligned its state tests to those standards. Not surprisingly, it is one of the few states without significant gaps between student performance on state tests and on the NAEP.
But Massachusetts is an exception. For a depressing instance of the rule, turn to New York, whose education department, even before the NCLB era, was repeatedly cited in official audits for failing to uphold its statutory responsibility to oversee local school districts. A 1991 report by state comptroller Ned Regan, for example, revealed that 10 percent of high school Regents exams had been improperly graded, leading to passing grades for many students who should have failed. But state education authorities did little to establish control over districts grading practices and didnt even consider revisiting the astonishing policy of letting teachers grade their own students tests. It therefore shouldnt have come as a surprise last year when the states current comptroller, Thomas DiNapoli, audited the grading of Regents exams and discovered that many students who should have failed had again passed.
But the problem worsened considerably in 2002, when NCLB was enacted and New Yorks testing agency was about to become the education departments most important bureau, in charge of supervising the NCLB-mandated reading and math exams for grades three through eight. Two years later, the position of state testing director became vacant in New York, giving the Board of Regentswhich governs the New York State Education Departmenta golden opportunity to hire a highly qualified director committed to creating an honest and transparent testing system. Instead, the job went to David Abrams, a high school English teacher who had spent ten years as an administrator in an Albany-area school district. Abramss curriculum vitaeapparently written hastily after I asked the education department for itshows that he lacks professional credentials in education testing, statistics, and psychometrics. He has bachelors and masters degrees in English, plus an administration M.A., which is required of every assistant principal in the state. One member of the Regents tells me that Abrams has no qualifications for the job, and hes responsible for many of our blunders on the tests.
Perhaps Abramss most serious blunder was ignoring a warning about those tests in 2008. On September 12, he received an urgent memo from Daniel Koretz and Howard Everson, a professor of education testing at the CUNY Graduate Center and chair of the state education departments Technical Advisory Group (TAG). The memorandum cited growing skepticism by the public and press about the reported score gains on the 2008 math and reading tests for grades three through eight. None of the reporters found the large increases credible, Koretz and Everson wrote. While we both emphatically assured them that there is no malfeasance, we had no basis for reassuring them with respect to test design and score inflation. They requested the education departments support for a program of validation studies to be initiated this academic year, proposing to study score inflation and the undesirable instructional activities that produce it as well as the ways that score inflation can create an illusion the achievement gap is closing, or that it is closing much faster than it actually is. Abrams never responded.
Though Koretz and Everson graciously absolved the education department of malfeasance as an explanation for score inflation, there was something very close to that in the way the department continued to defend itself whenever a reporter asked questions about the 2008 tests integrity. Education commissioner Richard Mills reiterated that New York was protected against test corruption because of the highly regarded testing experts who served on the TAG and certified the testing process. The same defense appeared on the departments website. All this despite the TAGs chairman having already written the memo pointing out that, absent an independent study, there was no basis for reassuring [the public] with respect to test design and score inflation!
Rather than finally doing something to restore the integrity of the tests, the education department released more wildly implausible results the following year. On the states 2009 math tests, for example, 87 percent of the seventh-graders achieved proficiency, up from 55 percent in just three years. In many districts, the number of students in all grades scoring above the proficiency bar was nearly 100 percent. In City Journal and the New York Daily News, Ive called results like this the Lake Wobegon Effect, after Garrison Keillors tales about a town where all the children are above average. More bluntly, the New York Posts Michael Goodwin called the test results one of the most destructive frauds in many years.
To see how New Yorks tests became corrupted, its necessary to understand how the scoring process works. The state hires a private education publisher, CTB/McGraw-Hill, to design its tests. A standard-setting committee of CTB/McGraw-Hill psychometricians then converts raw test scoresthat is, the total number of questions a student gets rightto a wider scale, much as raw points on the SAT get converted to a scale of 800. From these scale scores, the committee then recommends the cut points that establish the minimum bar for students to reach one of the four designated achievement levelsbelow basic, basic, proficient, and advanced. If the cut points are set too high, student achievement will appear to fall; if they are set too low, achievement will appear to rise. Even in the best of circumstances, according to Everson, setting the cut scores is more art than science.
Arguments about who set the cut scores for the troubling 2009 testsand for what purposeset off an unprecedented tumult at the Board of Regents last year. Shortly after her election as chancellor in April 2009, Merryl Tisch confided to several colleagues that she suspected Commissioner Mills of tampering with test scores to boost his own legacy. Another member of the Regents, former New York City schools superintendent Betty Rosa, went further, claiming that she was told by a high-ranking department official that Mills and Abrams had lowered the cut scores on the 2009 math test. Rosa even lobbied unsuccessfully to delay the release of the scores until there was an independent investigation of possible test corruption.
When I wrote Abrams to ask whether he and Mills had played any role in setting the cut scores for the 2009 tests, he didnt respond, but the education departments press secretary, Tom Dunne, assured me that the cut scores are set by CTB/McGraw-Hill. And Everson confirmed that protocol calls for CTB/McGraw-Hill to recommend to the commissioner what the final cut scores should be. But the commissioner doesnt have to follow the recommendation and always has the last word, Everson added. In a phone conversation, I asked Mills whether hed had anything to do with setting the 2009 cut scores. He didnt say either that he did or that he didnt. I did what was appropriate, and the record speaks for itself, he noted in our brief exchange.
Whoever set the cut scores, something untoward seems to have happened at the education department last year. Several researchers analyzed the 2009 tests and demonstrated that the standards dropped precipitously and arbitrarily. Diana Senechal, a Yale Ph.D. and former New York City teacher, established that in some grades it was possible for a student to attain the basic level just by guessing on every multiple-choice question, even while totally disregarding the section of the test that required longer written answers. Senechals study is buttressed by the fact that practically no students score below basic any more. For example, only 0.1 percent of all sixth-graders were designated at the lowest level on the 2009 reading test, down from 7.3 percent in 2006. Clearly, the downward trend was driven by the steady reduction in the percentage of raw points needed to reach basic status. Thus the percentage of raw points that sixth-graders needed to reach basic in reading dropped from 41 percent in 2006 to 17.9 percent in 2009.
The June 2 press conference announcing the 2009 math results was one of the strangest in the education departments history. Mills sounded triumphant as he described the spectacular double-digit gains by students all over the state and in every grade. But Tisch sat on the same dais, and it was obvious that she didnt believe a word of it. Just because scores have gone up dramatically does not mean that our youngsters are ready to go to college, she said. No one should interpret this as an enormous victory.
James Williams, the African-American superintendent of Buffalo schools, promptly complained that Tisch was disparaging the performance of his districts black children just as they were finally making academic gains. And in New York City, Mayor Michael Bloomberg and Schools Chancellor Joel Klein reportedly werent happy that Tisch was raining on their parade. They certainly had reason to be surprised: nothing in her background might have predicted that she would take on the states education bureaucracy. Shes a member of one of Gothams wealthiest families, part of the citys ruling class, according to a New York Times profile. She travels in the mayors social circle and counts both Bloomberg and Klein as friends.
However, Tisch had already pushed successfully for Millss ouster, and to succeed him, she eventually supported the appointment of David Steiner, the reform-minded dean of the Hunter College School of Education. As a Boston University political-science professor in the early 1990s, Steiner had played a role in advancing the Massachusetts education-reform law that mandated a knowledge-based core curriculum. Raised in England, Steiner had never even seen a multiple-choice test until he moved to the United States for graduate studies. He was shocked to discover that New York allowed teachers to grade their own students Regents exams, and he took office in October believing that the states testing system needed a major overhaul.
Tischs other major movesupporting the audit by Koretzunleashed a backlash by the education departments old guard, who feared (reasonably) that an honest evaluation of the states testing system would turn out to be extremely embarrassing. One high-level staffer even warned Steiner that Koretz was known in the field as a test bashera fabrication. But the reformers have stuck to their guns. Koretzs team at Harvard is already at work, mapping student responses to individual test items over the past four years.
It isnt only NCLB-mandated tests that suffer from score inflation, by the way. The same pattern of lowered standards has infected New Yorks high school Regents exams. Former CUNY education school dean Alfred Posamentier, a member of the states Mathematics Standards Commission, tells me that the Regents exam in algebratypically taken by students in the ninth gradehas been drastically dumbed down. According to Posamentier, the algebra Regents is no longer a very challenging instrument and, amazingly, requires a student to answer only about 35 percent of the questions correctly to earn a passing grade. A passing grade could then send a student on to the next coursetypically, geometryknowing only about one-third of the material from the previous mathematics course. This student is truly doomed to failure.
For Mayor Bloomberg, test-score inflation has been the gift from Albany that keeps on giving, an elevated platform from which to rejoice overand take credit forNew York Citys newly successful schools. Bloomberg and Chancellor Klein have also found ways to build even higher monuments to their success on top of that platform. Starting in 2005, for instance, the citys Department of Education (DOE) dissembled to reporters about the magnitude of score increases, appropriating to its column gains that had actually been achieved under the previous education administration. Also, many of the citys checks against cheating have disappeared in recent years. Under the old, pre-Bloomberg Board of Education, the safeguards included sending district administrators to every school on testing days to oversee test security procedures. The board also identified schools that had unusually high score increases from the previous year and subjected them to closer scrutiny in checking test papers. These procedures are nonexistent in todays DOE.
Moreover, beginning in 2005, the administration offered principals (and teachers, a few years later) a variety of inducements, including cash payments, for pushing test scores up, but didnt bother to ask too many questions about how the deed was done. Principals received cash bonuses of up to $25,000 for each year that scores went up substantially, and thousands of teachers got schoolwide bonusesa powerful enticement to inflate scores. If scores didnt go up, teachers and principals faced sanctions, such as having their schools closed. Finally, CTB/McGraw-Hill now provides yet another gimmick to help city teachers get the scores up: the Predictive Assessment, essentially a test-prepping device disguised as a mini-test that students take once a year and that closely reflects the blueprint and structure of the state tests.
The paradigmatic case of test-score inflation happened in the middle of the 2005 mayoral campaign at P.S. 33, an elementary school in the South Bronx. The percentage of the schools fourth-graders passing the reading examthat is, scoring at or above proficiencyhad more than doubled in a single year, skyrocketing from 35 percent in 2004 to 83 percent in 2005. Instead of ordering an investigation of this astonishing and literally unbelievable rise, Mayor Bloomberg held a press conference at the school to proclaim another miracle improvement on his watch. The schools principal, Elba Lopez, then quickly retired with a $15,000 bonus for her great work, which boosted her annual pension by about $8,000 for life. The next year, the same cohort of students, now fifth-graders, fell back to a pass rate of only 47 percent, and the pass rate for the new crop of fourth-graders was just 41 percent.
After I reported this in City Journal and Andy Wolf did the same in the New York Sun, the presumption that somebody had tampered with the students 2005 exams became so strong that the DOE was forced to take a look. But the department waited almost two years to start an investigation, by which time the suspicious test papers had been destroyed. In a report issued almost four years after the tests were taken, the departments investigator found no wrongdoing, yet neglected to interview Lopez. When I asked DOE counsel Michael Best how the most likely suspect could be cleared without even an interview, he responded that the departments investigator hadnt been able to locate her. A few months later, New York Post reporter Yoav Gonen found Lopez at her apartment in the Bronx, exactly where she had always been. Lopez assured the Post that there was no cheating; the reason that the students didnt maintain their spectacularly high scores for more than one test cycle, she explained, was that the school had a new, inexperienced principal. By its negligent handling of this well-publicized case, the DOE sent a clear signal to all adults working in the schools: feel free to tamper with students tests, so long as the miracle scores continue to make city hall happy.
In undermining the integrity of the tests, the Bloomberg administration had many powerful enablers, including the United Federation of Teachers (UFT). At the celebratory press conferences each year, union president Randi Weingarten appeared beside the mayor, nodding in approval as he detailed amazing stories of student improvement. Weingarten told confidants that the test scores were too good to be true. But in public, she maintained the fiction: Didnt the rising scores prove, after all, that teachers had earned their unprecedented raises of 43 percent since Bloomberg took over the schools?
Now that Bloomberg has informed the UFT that the days of wine and roses are over and negotiations for a new teachers contract are at an impasse, Weingartens handpicked successor, Michael Mulgrew, has suddenly reversed course on the testing issue. Should we accept these miracles on faith? the new UFT president asked in a satiric editorial for the union newspaper. Or do they warrant closer scrutiny? Reason alone tells us that when . . . 82 percent of students in grades 3 to 8 meet standards in math, the bar is too low. Only four months earlier, when the union still expected a 4 percent yearly raise from the city, Mulgrew had happily mounted the dais with Bloomberg and Weingarten to share praise for that very 82 percent proficiency rate.
Chancellor Tisch and Commissioner Steiner have a unique opportunity to offer parents an honest assessment of how their children and their schools are doing, even before the results of the Koretz audit come in. For starters, they should finally conduct a national search for a highly qualified testing director committed to transparency and an honest assessment of student academic achievement. The $148,000 salary that the education department already allocates for the position should be enough to attract someone with excellent professional credentials and a commitment to reform.
Tisch and Steiner could take a significant step toward solving blatant cheating by recommending legislation that would make it a crime for adults to tamper with students test sheets. They should also end the practice of allowing teachers to grade their own students Regents exams. School districts could easily create procedures in which exam papers are distributed randomly for grading by teachers from other schools.
Tisch and Steiner should also press for a moratorium on all bonus schemes based on students test scores until the Koretz audit is completed and there is greater confidence that the test-inflation problem is being solved. And instead of echoing NCLBs impossible goalall students reaching proficiency by some arbitrary dateTisch and Steiner should declare a much more realistic target: narrowing the gap between students performance on the state tests and on the NAEP tests. When the NAEP results are released, the education department should issue a report on how much progress (or lack thereof) has been made toward achieving that goal.
We have to stop lying to children, education secretary Arne Duncan said recently at a meeting of the National Governors Association (NGA). We have to look them in the eye and tell them the truth at every stage of their educational trajectory. Duncan has offered seed money to states to develop tests aligned to the Common Core State Standards project initiated by the NGA and endorsed by the administration. But Tisch and Steiner need not wait for the feds to fund better tests. By putting their own house in order, they can make New York a model for the kind of political courage and educational honesty that are desperately needed all over the nation.
Sol Stern is a contributing editor of City Journal and a senior fellow at the Manhattan Institute.