NYC Comptroller John Liu: High School Progress Reports Don’t Measure Progress

******************************************

Audit Finds that Department of Education’s Revisions to its High School

Grading System Leave Educators, Students Chasing a Moving Target

******************************************

New York City Comptroller John C. Liu announced that an audit of the Department of Education’s (DOE) High School Progress reports raised questions about the usefulness of the reports in comparing the yearly progress of schools.

“It’s troubling that a system that is used to decide school closings leaves teachers and students confused about what they need to do to improve,” Comptroller Liu said. “The Department of Education should not leave parents, educators or students in the dark when it’s deciding their fates.”

High School Progress Reports are a DOE accountability tool that assigns schools an annual grade of A through F. The Report grades play a significant role in the DOE’s decisions to reward high performing schools, perhaps with added funds, and restructure or close low-performing schools.

According to the audit, the DOE has revised the complex formula behind the grades every year. The frequent changes the agency has made to its grading and other formulas — without determining the impact of those changes — makes it difficult, if not impossible, to get a true picture of a school’s progress by comparing its grade from one year to the next. As a result, the High School Progress Reports paint an unreliable and confusing picture of a school’s progress or failure over time. Auditors recorded complaints from schools that the DOE’s lack of consistency made it difficult to set goals for students.

The audit focused on 10 high schools representing the five boroughs. It included three schools (Jamaica, Metropolitan Corporate Academy, and Norman Thomas high schools) that the DOE selected in January 2010 for closing.

CHIEF AMONG THE FINDINGS:

1. Inaccurate Picture of Year-to-Year Progress

The DOE’s changes to the formula behind the Progress Report grades make it difficult for parents and educators to measure a school’s performance from one year to the next.

The DOE says the Progress Report grades are meant as “a one-year snapshot” comparing one school against another in a given year, and not as a measure an individual school’s progress over time. Yet, the agency itself uses the formula to track achievement from one year to the next. For example, a school that receives a “C” three years in a row may be targeted for corrective action.

Example:

One school, Metropolitan Corporate Academy High School, which is set to close in 2014, improved its score every year from 2006 to 2010, but because of the DOE’s changing formula Metropolitan never rose above a “C.” In fact, the school fell to a “D” in the 2008-2009 school year even though its numeric score would have earned it a “B” under the 2006-2007 grading formula. It is impossible to tell to what extent Metropolitan’s scores reflect changes in its own performance or DOE’s changes to the grading formula. (See Table)

“The Comptroller’s audit of the High School Progress Reports demonstrates the difficulty of comparing a school’s letter grade over time when a school’s peer group composition and the cut scores for the grades change from one year to the next,” said Professor Aaron Pallas of Teachers College, Columbia University. “The recommendation that the DOE report high school progress report grades using both the old and new criteria would enable stakeholders both inside and outside of the schools to understand trends in school performance more completely.”

Action:

Since the audit, the DOE has posted an advisory on its website regarding year-to-year comparisons of High School Progress Report grades.

2.Lack of Communication

The audit determined that, while DOE met with school principals and others about changes, auditors found no evidence that it actually integrated feedback from them into the Progress Report. In fact, some educators told auditors that they felt as if they were chasing a moving target as they attempted to understand the changes that the DOE made to the grading formula each school year and to prepare students. The audit also found that the DOE did not do enough to inform schools what effect the changes to the grading system were expected to have on Progress Report grades.

“Taken alone, Progress Reports are an unreliable index for determining school closures or related high-stakes decisions,” said Professor David C. Bloomfield, chairman of the education department at the College of Staten Island. “Greater feedback by stakeholders, as recommended by the audit, might help to improve Progress Reports’ utility in this process.”

Action:

The DOE has since published materials summarizing and responding to feedback from educators and others involved in the 2010-2011 review process.

3. Data Reliability

The audit found that the data — student grades, Regents exam scores, and other information — that the DOE used to calculate each year’s Progress Report grades was representative of student data recorded in the DOE’s computer systems and verifiable. However, while the data in a given year was accurately recorded, it was not useful as a measure of an individual school’s progress over time.

RECOMMENDATIONS:

The DOE generally agreed with nine of the audit’s 10 recommendations and has begun to implement a number of them. However, the audit notes that “DOE inappropriately misinterpreted and even exaggerated, many of the audits ‘positive’ conclusions as an endorsement for the progress reports,” while simultaneously discounting its weaknesses.

SCOPE:

The audit was launched in March 2010 using data from the high schools’ 2008-2009 progress reports, the most recent data available at that time. The audit was expanded to include progress reports for the 2006-2007, 2007-2008, and 2009-2010 school years. In addition, auditors interviewed staff at the high schools in April and May 2010.

BACKGROUND:

High School Progress Reports are a DOE accountability tool that assigns schools an annual grade (A through F) based on a variety of factors, including student performance, student progress, school environment, and comparisons between schools with similar populations. The letter grades were introduced in the 2006-2007 school year.

Comptroller Liu credited Deputy Comptroller for Audit H. Tina Kim and the Audit Bureau for presenting the findings. The full report is available at http://comptroller.nyc.gov/audits.

NYS Comptroller Tom DiNapoli Audit Identifies Discrepancies in Dropout Rate Reported by NYC Department of Education

The dropout rate among New York City public school students is higher than claims made by the city Department of Education (DoE), according to an audit released by State Comptroller Thomas P. DiNapoli. DiNapoli’s auditors found that for the 2004 through 2008 school years, the dropout rate may have been as high as 16.5 percent, rather than the 13 percent cited by DoE. As a result, the graduation rate may have been as low as 62.9 percent, rather than the 65.5 percent reported by DoE.

“The city school system needs to sharpen its pencils when it comes to knowing which kids are dropping out and which kids are transferring to another school,” said DiNapoli. “DoE should be doing its homework and making sure the right papers are turned in to back up the reasons why students are leaving school.”

High school graduation and dropout rates are regarded as important indicators of a school’s effectiveness.


While the audit considered reported rates within 5 percent of audited rates to be generally accurate, the difference means that the graduation rate and discharge rate include thousands of students who actually dropped out.

DiNapoli’s auditors attribute the discrepancy to DoE’s erroneous classification of dropout students as having been “discharged” from high school. Discharged students should only be categorized as such when they transfer to another school or another educational program, leave the country, or are deceased.

DiNapoli’s auditors examined DoE’s discharge records for its 2004-08 general education cohort (the group of students who entered ninth grade in 2004 and were expected to graduate four years later), and found that in a random sample of 500 “discharged” students, 74 (14.8 percent) didn’t have the required documentation. As a result, all 74 should have been classified as dropouts.

Projecting the results of the sample to the entire cohort, DiNapoli’s auditors found that the correct graduation rate for the cohort was between 62.9 and 63.6 percent, rather than the 65.5 percent reported by DoE, and the correct dropout rate was between 15.5 and 16.5 percent, rather than the 13.0 percent reported by DoE. At some individual high schools, the correct graduation rates could be lower, and the correct dropout rates higher, than DoE reported.

According to DoE, the city’s 2004-08 general education cohort had a total of 88,612 students, of whom 46,896 graduated, 15,368 were still enrolled after four years, 17,025 were discharged, and 9,323 dropped out.

DiNapoli’s auditors also examined DoE’s discharge classifications for its 2004-08 special education cohort and identified similar errors. Auditors estimated that the correct graduation rate for this cohort was between 8.9 and 9.3 percent, rather than the 9.7 percent reported by DoE, and the correct dropout rate was between 20.6 and 23.8 percent, rather than the 17.2 percent reported by DoE. Even with the identified error rate, the NYC graduation rate is trending upwards as reported by DoE.

DiNapoli recommended that DoE officials:

• Ensure that DoE discharge guidelines fully align with New York State Education Department (SED) regulations;
• Instruct all schools to adhere to the SED regulations for discharge classifications, and provide training in the regulations for school staff who administer discharges; and
• Conduct periodic reviews of discharge classifications to determine whether they are being made and documented in accordance with SED regulations.

DoE officials generally agreed with DiNapoli’s recommendations and indicated they have taken action or will be taking action to implement them. Most notably, DoE’s guidelines were amended before the 2009-10 school year to better align with SED’s guidelines on required documentation to support a discharge classification. Click here for a copy of the report.

Progress Report: The MTA Capital Security Program - New York State Comptroller Tom DiNapoli

Improvements Made Despite Delays and Cost Overruns

Despite significant delays and unplanned costs, the MTA has greatly improved its security through capital improvements, operational initiatives and enhanced cooperation with other security agencies, according to a report released today by New York State Comptroller Thomas P. DiNapoli.

“The capital security program the MTA has implemented since 9-11 has made New Yorkers more secure,” said DiNapoli. “The MTA has made progress, particularly in the last two years. But the mass transit system is still inherently vulnerable. Individual projects in this program are months if not years behind schedule and well over budget, and additional capital improvements are needed. My office will continue to track MTA management of this program.”

The projects in Phase 1 of the MTA’s capital security program target the system’s most vulnerable and heavily used assets, including stations, transit hubs, bridges and tunnels. Each project involves one or more facilities and security improvements to elements such as electronic security and surveillance, fire, life and safety and evacuation enhancements, perimeter protection and structural hardening. This phase, originally scheduled for completion by September 2008, will not be completed until June 2012.

After more than nine years, the MTA has completed 11 of the original 16 security projects as well as elements of the five remaining projects. The MTA has hardened all 14 facilities planned for Phase 1; improved lighting, communication systems, and smoke and fire detection equipment in 15 facilities; installed perimeter protection around four facilities; and despite significant setbacks, the electronic security program.

As of December 2010, the MTA had completed 31 of 38 planned construction tasks and the remaining seven tasks were all in the process of construction, though more than 60 percent of the 38 tasks were behind their established schedules, including 11 that were behind by more than one year (five tasks were more than 30 months late). The cost of Phase 1 (including two facilities that were deferred from Phase 1 to Phase 2) has grown from $591 million to $851 million, an increase of 44 percent.

Electronic security projects in particular have encountered significant problems. The cost of this program has nearly doubled, growing from $265 million to $515 million. The MTA has continued work on the electronic security program by hiring other contractors, but full implementation is not scheduled to be completed until June 2011. A lawsuit between the MTA and Lockheed Martin, the original contractor responsible for the electronic security program, is still ongoing.

The MTA has also implemented a number of operational initiatives that have improved the system’s safety: security personnel have been increased, coordination with other security agencies has been enhanced and the MTA has implemented a public relations campaign to encourage the public to report suspicious activity.

The Comptroller noted that additional capital improvements are needed to improve overall security. Phase 2 will fund some of the remaining 33 security projects of the original 57 that were identified through a risk-based assessment. However, the MTA will not begin work on the remaining 16 projects until the scope is determined and funding becomes available.

Comptroller DiNapoli maintains a comprehensive approach to oversight of the MTA. Since 2007, DiNapoli has issued 27 reports on the MTA, including 14 audit reports and 13 reports examining MTA finances and operations. There are currently eight audits underway in various stages of completion, including a forensic audit of MTA use of overtime and one audit that is being performed jointly with the New York City Comptroller’s Office.

Click here for the report: http://www.osc.state.ny.us/press/releases/mar11/MTASecurityRpt12-2011.pdf

###

Follow NYS Comptroller on Twitter: @NYSComptroller