A quarterly magazine of urban affairs, published by the Manhattan Institute, edited by Brian C. Anderson.
• • • • • • • • • • • • • • •
Compstat and Its Enemies
A study questioning the NYPDs statistics is irredeemably flawed.
17 February 2010
The crime analysis and accountability system known as Compstat, developed by the New York Police Department in 1994, is the most revolutionary public-sector achievement of the last quarter-century. Since its inception, Compstat has driven crime in New York down an astounding 77 percent; veterans of the Compstat-era NYPD who have gone on to run police departments elsewhere have replicated its successes. Other government agencies, both in New York and nationally, have applied the Compstat model to their own operations, using minutely analyzed data to hold managers accountable for everything from improvements in public health to decreases in welfare dependency to road repairs.
Now, however, a survey of retired NYPD commanders by two criminologists purports to cast doubt on the wisdom of Compstat. Compstat has undermined the reliability of the NYPDs crime-reporting system, say former John Jay College of Criminal Justice professor Eli Silverman and Molloy College professor John Eterno. Weekly meetings in which top brass grill precinct commanders about crime on their watch place too much pressure on the commanders, resulting in data manipulation, the professors charge. Those people in the Compstat era . . . felt less pressure to maintain the integrity of the crime statistics, Eterno told the New York Times. The system provides an incentive for pushing the envelope, Silverman added. Silverman and Eterno anonymously asked several hundred retired NYPD captains if they were aware of instances of commanders changing crime data. About half of the survey respondents (157 of 309) said that they were aware of changes to crime reports. A follow-up question asked whether the changes were ethically appropriate. Of the 160 respondents who answered, 22.5 percent thought that the changes were ethical, while 53.8 percent believed that the changes were highly unethical.
Critics of the NYPD have seized gleefully on the study, which has not yet been publicly released. The New York Times obligingly wrote it up on its front page and followed up with several articles on the topic.
NYPD foes can put away their party hats. Nothing in the survey discredits Compstat or its crime-fighting accomplishments. Eternos claim about a decreased emphasis on crime-data integrity in the Compstat era is demonstrably false. It is ludicrous to suggest that a department where the top brass did not even get crime data until six months after the crime and then did nothing with themas was the case in the pre-Compstat eracared more about the accuracy of crime statistics than one in which every deployment decision is made based on the minute-by-minute reality of crime on the streets. Nor does the study, which has several design flaws, cast any doubt on the citys record-breaking crime drop. Given the enormous efforts that the NYPD makes to ensure the validity of its statistics, the study ultimately comes down to a dangerous argument against accountability systems per se.
The message from the powers-that-be is clear and unequivocalDO NOT COOK THE BOOKS. So responded a retired NYPD captain to my e-mailed questions about the Eterno-Silverman study. My correspondent is no friend of the department; he still bears the animosity of the rank and file against headquarters. He readily conceded, however, the all-consuming emphasis that the Compstat-era department places on data accuracy. Other commanders echo this view. The biggest sin you could commit in the NYPD was fudging, says Steve McAllister, a former borough chief in the transit bureau who now serves as crime-control strategy specialist in the Newark Police Department. The consequences of fudging were worse than the consequences of a crime increase. You would lose your command and be vilified. It was a career-ender.
Silverman and Eternos claim that Compstat has led to a weakening of pressure to maintain the integrity of the crime statistics ignores both the actual history of the Compstat era and the philosophy behind it. NYPD resources devoted to data integrity have steadily increased during the Compstat years. The department did not even begin routine audits until 1999. In standard auditing practice, the size of the audited sample mirrors changes in the volume of data; in the NYPD, however, even though the number of reported crimes has dropped sharply, the number of audited crime reports has grown. In 2009, 47,510 crime reports were audited, compared with 29,277 in 1999, despite budget cuts over the last nine years that have forced the attrition of 5,000 cops. The crime misclassification rateoften the product of innocent errorhas dropped from 4.4 percent in 2000 to 1.5 percent in 2009.
The NYPD goes beyond accepted auditing conventions in other ways as well, as an independent assessment of the departments data-quality controls in 2005 observed. When the New York State Comptroller reviewed the integrity of the NYPDs crime reporting in 2000, it used a typical sampling practice: it selected 25 precincts for auditing and cut off its audit early after finding no significant evidence of misreporting in the first 17 precincts it analyzed. The NYPD, however, does not merely sample among its reporting units; it audits each command twice a year and samples every crime category within that command, observed New York University professor Dennis Smith and SUNY Albany professor Robert Purtell in their 2005 study.
If a precinct is suspected of manipulating dataif it shows a sharp crime drop when surrounding precincts show an increase, sayinvestigators will scrutinize up to 5,000 crime reports from that precinct. The chance of hiding misbehavior under such scrutiny is almost nil. When the Quality Assurance Division [the auditors] comes in, they turn everything inside out and peruse every statement in the record, says Steve McAllister. If Im trying to change burglaries to criminal trespass, I cant create a system that would rewrite every victim statement in my file. As the commanding officer of the 66th Precinct in Brooklyn, McAllister experienced one of the departments most intense data audits. A malcontent from the community had falsely accused McAllister of lying about crime. My borough commander called me to task, McAllister recalls. I said: Audit me. We had the burden of discrediting everything [the malcontent] said. The departments Quality Assurance Division pulled thousands of crime reports from McAllisters watch and subjected them to individual scrutiny, including reinterviewing victims and witnesses. None gave evidence of cheating.
Smith and Purtell compared the NYPDs auditing processes with those of eight other large departments. No other department has systems in place that come close to matching the set of high level, overlapping, systematic audits of crime reporting now employed by NYPD, they concluded. The NYPDs integrity apparatus encompasses both a Data Integrity Unit that focuses on coding accuracy within computer reporting systems and the Quality Assurance Division that audits the processes that lead up to computerized crime reports. Whereas in other departments, underreporting scandals have been discovered by outsiders, the NYPD itself discovers instances of questionable reporting, observed Smith and Purtell. Confirmed cases have resulted in serious sanctions. The 11 confirmed instances of intentional misreporting since 2002 have led to four commanding officers being stripped of their commands and a range of other discipline against lower-level officers.
The NYPDs intense dedication to reliable data grows out of the Compstat philosophy itself. One of Compstats most powerful accomplishments was to yoke crime data and analysis intimately to deployment decisions. Compstat can accomplish its crime-lowering goal only if the crime data that commanders scrutinize so obsessively is accurate. Jack Maple [Compstats flamboyant co-originator] stressed the necessity for accurate and timely intelligence, recalls Edmund Hartnett, a former executive officer of the NYPDs Narcotics Division and now commissioner of the Yonkers police department. Everything that we do is based on crime data; if the data is not accurate or you are missing dots on the map, youre not deploying properly. A pattern of felony thefts ignored because of data manipulation will spread and worsen, a contagion that no police commissioner should be confident can be manipulated out of public awareness. In the Compstat era, commissioners live and die by the reality of crime. It is overwhelmingly in their self-interest to ensure faithful crime reporting since, without it, they will misallocate their scarce crime-fighting resources.
The Silverman-Eterno survey offers no evidence that New Yorks reported crime drop is inaccurate. The magnitude of that drop is simply too great to be affected by a few instances of commanders reclassifying a theft from a felony ($1,000 of stolen goods) to a misdemeanor (theft under $1,000) by cross-checking on eBay a victims assessment of the value of the stolen goods. In order to design a system that would significantly hide crime, a retired officer says, you would need multi-level coordination from a precincts patrol officers through its desk officers and crime-analysis unit and finally to its executive and commanding officers. People cant agree on lunch orders. Do you really think that a conspiracy of this intricacy could operate without discovery? he asks.
The most authoritative independent check on a departments reported crime trends backs New Yorks crime data. The Justice Departments National Crime Victimization Survey showed an even stronger correlation with the NYPDs crime data after Compstat began, two federal Bureau of Justice Statistics researchers concluded in 2003. No one has ever suggested that homicides can be reclassified as mere assaults or hidden altogether. The NYPDs 77 percent homicide drop has led most other crime categories. If felonies were being systematically reclassified as related misdemeanors, the misdemeanor rate should rise. But misdemeanors have dropped along with felonies.
Moreover, the public has confirmed the NYPDs crime reporting with its feet. Few of the new residents and business owners who have flocked to the citys once violence-saturated neighborhoods since crime began dropping have heard of Compstat, much less checked Compstat data on the Internet. Rather, they have based their relocation decisions on word of mouth and their own lived experience of an area. Every marker in the citys revitalized neighborhoodsfrom the absence of the once ubiquitous No radio signs in cars, to the opening of national chain stores, to the appearance of children riding their bikes on the sidewalkbuttresses the reality of the crime drop.
There are several reasons, however, to question the significance of Silverman and Eternos findings. The professors received responses from 323 former NYPD captains and above who retired sometime after 1995, when Compstat was in full force, out of 1,200 surveys that were mailed out. We dont know when they retired; it could have been in 1996 or 2006. If the former, their knowledge of the post-Compstat department is limited. We do not know how many different incidents of report revision are being referred to by the 87 respondents who said they were aware of changes to crime reports and who found the changes highly unethical. It could be 87; it could be 17. The survey did not ask if the respondents had personally witnessed data manipulation or had just heard about a case. Nor did it ask if the manipulation was detected and what the consequences were. If retired editors at the New York Times were asked if they were aware of reporters making up quotes, the positive response rate would likely be 100 percent, since every editor would be aware of Jayson Blair, the Times reporter forced to resign in 2003 for fabricating sources and plagiarizing articles.
Nearly 34 percent of the respondents who mailed back their survey (166 individuals) had retired before 1995. Silverman and Eterno do not disclose what percentage of these pre-Compstat-era captains said that they were aware of changes to crime data, a figure that would provide some benchmark for their thesis. Even if the percentage is lower than in the post-Compstat era, however, that difference could simply demonstrate the greater publicity attendant on data-changing charges in the Compstat era. The only finding in the professors survey that backs up their claim that the department is less concerned about data integrity since Compstat is the slightly lower rating that post-1995 retirees give to the departments demand for integrity in crime statistics compared with pre-1995 retirees: post-1995 retirees rate the demand for data integrity at a mean of 6.52 out of ten, compared with a mean of 7.18 for pre-1995 retirees. This is meager evidence, though. There is no way to know whether the pre- and post-1995 retirees interpret the surveys one-to-ten rating scale identically. A more reliable indicator would have been to ask the post-1995 retirees to compare the departments data accuracy pre- and post-Compstat, but this the authors apparently failed to do. In any case, the slight difference in the ratings given to the demand for data integrity does not come close to overriding the significance of the massive auditing apparatus that the department has put in place since Compstat began.
Silverman and Eterno provide selected anonymous comments from survey respondents at the end of their summary. It doesnt paint a pretty picture, though not necessarily for the reason that Silverman and Eterno intend. Many of the respondents display the petty resentments toward higher-ups that are depressingly on view at the unofficial website for NYPD cops, NYPD Rant. They charge favoritism and unfair treatment by managers. To what extent the survey answers were driven by this resentmentand to what extent the favoritism charges are true and not just sour grapesis unknowable. But a New York Times profile, meant to buttress the Silverman-Eterno thesis, unwittingly shows how that resentment and self-pity undercut the significance of the surveys already paltry findings. An anonymous retired captain who took part in the survey tells the Times that he was himself subjected to investigation after crime in his precinct dropped far more than in neighboring precincts:
Eventually, [the captain] was called in before the departments Internal Affairs Bureau, and accused of improper classification of some of those crime reports.
The commander insisted Sunday he had done nothing wrong. . . . But ultimately, he said, he decided to retire after 20 years on the job because he said the department threatened to punish him for the offenses it believed he had committed. Wrongdoing was never proved in the matter, but he was also never able to prove that he acted properly, he said.
It eats its own, the commander said of the department. Thats what it does. But I was doing the right thing for the department. The moral of the story is, Dont try to be perfect; dont try to do too much.
Amazingly, the Times thinks that this lachrymose tale supports the thesis that the department is lax about crime accuracy. The paradox of a department punishing people for the data improprieties that it is said to be indifferent to is an irony that no one pushing the anti-Compstat story seems troubled by.
Compstat indisputably created hard feelings in the department. It was a radical change to the status quo that upset longstanding hierarchies. It demanded measurable output from officials who had long been accustomed to coasting. Especially in its early years, Compstat could be a brutal experience; whether it still is is unknowable, since Commissioner Ray Kelly has closed it off to outside observers. It is no coincidence that the Captains Union funded the Silverman-Eterno study; the radical fringe in all the departments unions wants Compstat thrown out. The crap rolls downhill, as they say, observes an NYPD veteran and police commissioner at another department. When a CO is yelled at at Compstat [meetings], he comes back to the precinct and yells at his cops to start taking proactive action against the weed-smokers and guys drinking on the corner. The cops think that if we got rid of Compstat, they could ride around the sector like they used to 16 years ago waiting for victims to call in crimes.
The effect of Silverman and Eternos poll, whether intentionally or not, is to cast doubt on all accountability systems. Without doubt, once an institution starts scrupulously measuring results and holding managers responsible for those results, some people will try to game the system. The school accountability movement has produced numerous efforts to manipulate test scores, some more deceitful than others. The question is: what is the alternative? Short of getting rid of accountability altogether, the best that a system can do is to work tirelessly to uncover fraud. That the NYPD clearly does. Would it be better to lighten up on the departments relentless focus on outcomes? Compstat produced a sense of urgency about fighting crime that had never existed before. And unlike many police agencies that also experienced a crime drop in the 1990s, the NYPD has never lost that sense of urgency. The results speak for themselves: whereas the crime drop has leveled off or reversed itself in several cities that were considered competitors to New York in the 1990s, the NYPD has kept its crime rout going, allowing New York to hold on to its title as the safest large city in the country for a decade.
Perhaps there are additional measures that the department could take to beef up its already unparalleled integrity systems, such as making deputy inspectors in the borough commands responsible for quality control, or auditing the big precincts more frequently. But Compstat has accomplished what no other police reform ever has: driving down crime to record-breaking lows and focusing the department relentlessly on crime reduction. New Yorkers would be foolish to call for a watering down of the intense pressure it puts on commanders to perform.
Heather Mac Donald is a contributing editor of City Journal and the John M. Olin Fellow at the Manhattan Institute.