By Brittany Wallman
The Sun Sentinel
FORT LAUDERDALE, Fla. — Arthur Buckley was startled awake by screaming and yelling inside the Coral Ridge home he was visiting in Fort Lauderdale.
He scrambled through the house to find a gory scene: his friend on the floor next to his bed, blood from a head gash streaming across the marble floor.
For eight to 10 minutes, he tried to get through to 911, first on the land line, then his cell.
“It rang and rang, and nothing happened,” Buckley recounted.
The son of a surgeon, he knew the stakes were high. He feared his friend would bleed to death.
Across Broward County in Pembroke Pines, Henry Foulon dialed 911 in the early morning of Jan. 14, too. His 79-year-old father-in-law had fallen and cut his forehead. No one answered. “Am I dialing the right number? What is going on here?’' Foulon remembered thinking.
All told, 426 people called 911 between 4:05 a.m. and 8:35 a.m. that morning. Sixty-five percent of them didn’t get through, according to county records.
A report detailing the disruption says a mixture of human and technological error during a software upgrade rendered the system unstable for several hours. Some callers got through, others got endless ringing.
“It was a very scary few hours,’' Broward Commissioner Lois Wexler said at a meeting of fire and police officials earlier this year.
The consequences of the phone outage remain partly a mystery. Records about the unanswered calls show that many of the 911 callers couldn’t be reached when dispatch personnel later tried to call them back.
The countywide 911 system has been under intense scrutiny since it went live in October 2014, straining to properly dispatch calls all over the county. The system, run by the Broward Sheriff’s Office, fields up to 3 million calls a year.
Much of the focus from frustrated fire and police personnel has been on the 911 call-takers and dispatchers.
But the system’s technological challenges also are weighty. Internal records show a continued fight against faulty equipment: Computers freeze, 911 headsets fail, calls are inexplicably dropped, radios used to communicate with first responders fill with static, the dispatch system suddenly fails. Other complaints are more routine, the records show, such as a keyboard that won’t type the digit “2.”
Software for the Viper phone system was being updated that morning by vendor West Corp., formerly Intrado Government Solutions Division. But a West Corp. technician missed the first alert that a server was down, and was slow to do anything about it when the disturbance came to light, the report says. The phones were fully restored hours later, after part of the system was taken offline.
Records about the phone outage show that:
• 278 callers couldn’t get through.
• 85 callers didn’t answer when the dispatch center tried calling them back.
• The phone outage affected callers all over Broward County. Calls were marked as “abandoned’’ from Fort Lauderdale, Hollywood, Oakland Park, Tamarac, Coconut Creek, Cooper City, Coral Springs, Pembroke Pines, Dania Beach, Pompano Beach, Deerfield Beach, Davie, Sunrise, Hallandale Beach, North Lauderdale, Miramar, Parkland, Lauderdale Lakes, Weston, Margate, Wilton Manors, Lauderhill, Plantation, Southwest Ranches, and there were several misdirected calls from Miami-Dade County.
According to a detailed report of the incident by West Corp.:
The upgrade began at 3:31 a.m. About a half hour later, the servers started going down but the local technician missed it. At 4:28 a.m., both servers on the primary phone system went down. By then, the technician noticed a spike in callers hanging up without getting through.
At 4:34 a.m., a half hour after the troubles began, the technician asked the help desk to assist. But the help desk and local technician decided to “allow the upgrade process to continue’’ before doing anything further to fix the problems, according to the West Corp. report.
The upgrade was complete at 5:01 a.m. Yet the phone system problems continued.
Just before 7 a.m., one of the servers was rebooted “in an attempt to stabilize the situation.’' At 8:12 a.m., another server was rebooted. At 8:35 a.m. the problematic part of the phone system was taken offline, and call processing returned to normal.
At 12:21 p.m., more phone troubles prevented call-takers from logging in to the system. The troubles persist until 8:30 p.m.
Officials at Broward Sheriff’s Office, which operates the system under a contract with the county, said they know technology is imperfect, and they’re pleased the county is working to replace critical elements of the system.
In 2017, an $18 million, modernized Motorola Solutions Inc. computer-aided dispatch system is expected to replace the existing software. And the county in May agreed to negotiate with two firms for a $40 million replacement of the radios and radio system used by police and fire personnel and emergency dispatchers.
The county raised property taxes to pay for the system. It serves all cities except Coral Springs and Plantation, which opted out but whose residents still pay for it.
“We’re dealing with technology and equipment, and nothing works perfectly 100 percent of the time,’' BSO executive director of community services Robert Pusins said. “When we do have issues, we certainly report it to [the county] for them to handle. We just operate their equipment.’'
Brett Bayag, interim 911 director, told police and fire personnel that the software upgrade had been completed successfully in “at least a couple dozen’’ other counties, and similar upgrades had been done in Broward “over a couple dozen’’ times.
In its report to the county, West Corp. promised additional employee training to improve decision-making, and altered the software upgrade process to help prevent such an outage from occurring again.
In a Feb. 11 memo to county commissioners, County Administrator Bertha Henry said she was confident the necessary changes had been made.
In Foulon’s case, he had the non-emergency number to the Pembroke Pines Police Department handy, and called them instead. His father-in-law’s head wound was stapled closed.
Buckley’s 911 call eventually got through, after he ran around frantically, calling from different phones in the house and from his Massachusetts-based cell phone. Fort Lauderdale paramedics arrived quickly, he said, and he was relieved. His buddy was taken to Holy Cross Hospital.
Amidst the clamor of complaint, the county hired Missouri-based consultant Fitch and Associates to analyze the system. Officials in Fort Lauderdale and Pembroke Pines are awaiting the report expected by summer’s end before deciding whether to act on threats to withdraw.
Copyright 2016 the Sun Sentinel