Report: Human error broke Fla. county 911 system, made fixing it harder
Hillsborough County’s 911 system failed for just over 22 hours in September, leaving 768 calls for help unanswered while officials struggled to fix the problem
Tampa Bay Times, St. Petersburg, Fla.
TAMPA — Hillsborough County’s 911 system failed for just over 22 hours in September, leaving 768 calls for help unanswered while officials struggled to fix the problem.
The real problem, though, wasn’t just technical. It was human error, according to reports obtained by the Tampa Bay Times.
Lack of oversight prevented those wireless callers from ever reaching a 911 operator, the reports said, and lack of urgency prevented crucial 911 services from being restored sooner.
That was not the only breakdown of the county’s 911 system last year, but the September blackout was the longest and most serious occurrence. The investigation into that incident, completed in December, revealed “a serious lack of supervision and direction during the recent failure.”
That led county leaders to dismiss Hillsborough’s top 911 administrator and embark on a complete overhaul of its sprawling 911 Agency — an undertaking Deputy County Administrator Greg Horwedel hopes to complete in the next four to six months.
“It’s really critical for us that this never happens again and that everything is done the way it’s supposed to be done — that’s our obligation as a county government,” Horwedel said. “We’re committed to making sure we do everything we can to assure our citizens that they can call 911 and they will get a response, that we have a system that is operational and ready in any emergency.”
The December audit delivered to County Administrator Mike Merrill detailed “an apparent lack of urgency by County technical staff” during the outage, which blocked some wireless callers from reaching 911 from 1 p.m. on Sept. 6 until 11:30 a.m. on Sept. 7. The breakdown took place on two weekdays, from Thursday to Friday.
“There was no one failure here, no one thing that went wrong, but a total lack of situational awareness,” said Bruce Moeller, a consultant with Fitch and Associates who was hired by the county to spearhead the inquiry. “All the safeguards and backup systems were in place, but there was no oversight or communication to prevent the problem form happening.
“And when it did, problem solving was impacted because people didn’t have a strong understanding of the system.”
The technical fault lay with two Frontier Communications “trunk lines” — one pair out of about 100 fiber network cables that carry wireless 911 calls to the county’s operation centers — that were taken offline for maintenance.
The cables had been rebuilt but never received the proper testing before another county vendor, Motorola Solutions, cleared the hardware to go online. And because there was no documentation of the procedure, and no oversight or direction from the county officials responsible for overseeing it, no one noticed that 911 calls picked up by those two lines weren’t getting connected to operators.
Instead, the callers routed to those lines were getting busy signals — or being dropped completely.
Three hours passed before the Tampa Police Department’s 911 center grew alarmed at the low number of calls and alerted Motorola about the problem.
Horwedel, Merrill, Hillsborough Fire Rescue Chief Dennis Jones and a host of other county officials got the call about the breakdown just after 11:30 p.m. the night of Sept. 6. Officials worked to solve the snowballing crisis until 4 a.m. the next morning, Horwedel said.
The county’s 911 system was designed with an identical, fully-functioning backup that could have prevented the calls from being dropped — but the report said no one switched it on.
In a Dec. 18 memo outlining the audit’s findings, county staff told Merrill they agreed with recommendations to immediately replace 911 Administrator Ira Pyles with a “qualified in-house manager” who would work alongside new technical support staff dedicated to fixing a similar crisis in the future.
Pyles in particular was singled out in the audit for failing to work directly with the vendors responsible for keeping the 911 system working.
“Collaboration between Motorola and others involved in the county’s 911 system was generally uncoordinated and left to occur organically,” the memo said. “While this type of laissez-faire approach to troubleshooting can and does develop over time between staff and trusted vendors, it’s an unacceptable practice to allow for a mission-critical emergency system like 911.”
Pyles was placed on administrative leave on Jan. 3, said county spokeswoman Liana Lopez. He has until Monday to decide whether to apply for another job within the county or retire from the agency. Pyles did not return calls for comment.
That September incident followed a series of blackouts that plagued the county's 911 system at the start of 2018. But those were just 2-3 hour outages compared to the 22-hour breakdown, and there were multiple sources of blame. A statewide network outage suffered by Frontier led to intermittent outages in January; and a road-widening project that cut service cables in February.
Officials have approved a plan for a system-wide review of the county’s 911 operations and infrastructure, provide staff with additional training and retooling risk mitigation practices. The cost of that project will be about $193,000. The first priority will be getting the system vendors and all nine of the county’s 911 operations centers on the same page, operating under the same protocols with clear lines of communication and oversight, Moeller said.
His team is also working with the county to establish long-term goals for 911 service, including upgrades to existing infrastructure and building in Text-to-911 capabilities for users who are deaf, hard of hearing, or cannot safely speak to an operator. Pinellas County rolled out such a system in December.
Hillsborough’s emergency response system will grow from the mistake, Horwedel said. But he fears the toll of that September breakdown may never be known.
“Thank God I never heard about a major catastrophe from the outage,” he said. “But I can’t begin to speculate on what those calls might have been about.
“My hope is that they kept calling, or called another agency and we were still able to help the situation, whatever it might have been.”
©2019 the Tampa Bay Times (St. Petersburg, Fla.)