Florida Hospital performs recovery tests twice a year, recovering more than 100 applications. The healthcare organization needed a recovery strategy that improved communication and efficiency, managed a complex change control process, and that would keep staff at home during a disaster.
With electronic medical records, healthcare reforms, and an explosion in data volumes, healthcare organizations have become much more complex, and far more reliant on IT. These factors are leading many healthcare IT departments to take a more comprehensive approach to test management programs and recovery strategies.
Florida Hospital is the largest healthcare provider by number of patients seen every year in a Medicare facility. The organization comprises seven distinct hospitals in the central Florida area, the largest of which is in downtown Orlando.
The hospital had selected Sungard Availability Services to provide disaster recovery services in 2003, when, according to Carl Minear, data center director at Florida Hospital, “It was simply a matter of shuttling tapes back and forth and spending days doing a recovery process.”
“We were what you might call a ‘tapes and trucks’ kind of recovery program,” says Robert Goodman, disaster recovery coordinator at Florida Hospital.
A Disaster Recovery wake-up call
In 2004, a series of three hurricanes signaled to hospital administration that a stronger, more effective recovery program was critical.
“When the hurricanes hit, luckily we did not have to declare,” says Minear. “But it was a wake-up call.”
Goodman adds, “We found out we didn’t have enough people. We didn’t have enough time. The logistics were a nightmare.”
As a result, the recovery program couldn’t meet the recovery time objectives and recovery point objectives the hospital needed.
“That was the point when the hospital realized we have a problem that needs to be fixed,“ says Minear.
Florida Hospital moved first to Sungard AS’ Advanced Recovery services, utilizing asynchronous replication between Orlando and a Sungard AS recovery data center in Philadelphia. Then, most recently, the healthcare organization migrated to the Managed Recovery Program (MRP). MRP combines the set of essential Sungard AS recovery services into one fully managed program, so organizations like Florida Hospital don’t miss a single step in recovery procedure development, implementation, or execution.
The benefits of MRP to Florida Hospital became clear almost immediately.
“The Managed Recovery Program has just been a phenomenal success for the last two tests,” says Goodman.
Turning over responsibility for the management of a full recovery program requires identifying a service provider with a proven track record, and the hospital found that in Sungard AS. Goodman puts it this way: “What does Sungard AS do? Hundreds of recoveries. That’s who you want: Someone who’s done it so that we can draw on those strengths. It’s almost like having a set of recovery consultants.” Says Minear, “When we were lacking in expertise, Sungard AS brought it to the table.”
Sungard AS assigned a crossdiscipline team to the Florida Hospital recovery program, with application expertise specific to the hospital’s application environment.
Additionally, Sungard AS assigns a single point of contact—the Service Delivery Manager—to act as liaison with Goodman and his recovery team. This streamlined approach keeps communication errors to a minimum and gives the hospital one number to call to coordinate the recovery program.
“While he’s working the problems from a Sungard AS standpoint, I’m working them from a Florida Hospital standpoint,” says Goodman. “As a result, things that would have taken an hour or two now are done in minutes.”
Among the most frequently cited causes of a problematic recovery is a lack of adequate change management—keeping the configurations in a recovery environment current with the configurations in the production environment. Because of the high volume of changes in Florida Hospital’s production environment, one of the most visible benefits of MRP is that Sungard AS manages the change control process on the recovery side.
“It used to be twice a year we would update our recovery scripts, and quite frankly, that’s good,” says Goodman. “That’s probably as good as I could get. But bring on the Sungard AS team, and they’re virtually coming into our change control meetings, looking at the systems, and then noting things that have to be changed in recovery scripts, new systems, retired systems, and version changes. They drive that process.”
With the Managed Recovery Program, Goodman says that what was once a twice-yearly change management process now happens on a weekly basis. In addition to keeping the recovery environment up-to-date, the process also alerts Florida Hospital to changes that need to be made on the production side.
“If our versions get out of sync or need to be upgraded, Sungard AS keeps us aware of that,” says Goodman. “This has been a tremendous offload of a lot of little things that you have to keep track of. With Sungard AS driving that effort, our technicians can stay focused on the new projects, and do DR as needed.”
Keeping staff where they’re needed
“When you have a very large disaster recovery strategy, the number of people required to perform the recovery is through the roof,” says Minear. “We would have people lined up waiting to fly to [the Sungard AS recovery facility in] Philadelphia to perform the recovery test. Then when you’re all through, you go back.”
The problem, says Minear, is that during an actual disaster declaration, many of those people are needed back in Orlando. With the Managed Recovery Program, Sungard AS assumes responsibility for the recovery of Florida Hospital’s critical systems and assigns staff as necessary to scale to the hospital’s needs. Where the hospital had been sending as many as 19 staff members to the recovery facility in Philadelphia, it now sends about three.
Goodman and Minear say they are “thrilled” with the results of the Managed Recovery Program, both as a way to drive efficiency and as a way to deliver a more effective and reliable recovery program.
“It’s no longer a customer/vendor relationship. It became a partnership. Sungard AS responded to our questions and our needs, and at the next test, everything we needed— and then some—were delivered."