CSG International processes 1 billion cable statements annually relying on Managed Recovery to ensure resiliency
Imagine watching the college football playoffs or your favorite Walking Dead or Game of Thrones episode, when suddenly your cable picture goes dark. What could have happened? It may be something as simple as your cable provider not being able to recover customer data following a power outage. In today’s always-on world, this is more than just a frustration.
CSG International (CSG) provides business critical software and service solutions for managing customer data, analyzing data, statement production, and customer service, primarily for the telecommunications industry. It is the second-largest cable-billing provider in the world, processing 1 billion cable statements worldwide each year. That’s a lot of data – and it’s evergrowing. A “lost” data center could prove a catastrophe.
That’s why CSG relies on Sungard Availability Services. CSG deemed it essential to add a managed recovery program, or MRP, to its businesscontinuity and disaster-recovery capabilities. It views this addition as an evolution, reflecting the reality that both the complexity of the recovery and the volume of servers that need to be recovered could surpass CSG’s ability to hit contractual recovery time objectives (RTOs).
Partnering for business solutions
Sungard AS possessed the ability to connect required servers to its infrastructure rapidly and also to quickly build operating systems. In addition, Sungard AS could add more people to ensure any recovery occurred quickly, say Dave Mucia and Cindy Jenkins, CSG’s Director of Business Continuity, and Business Continuity Lead, respectively. “Sungard AS is one of only two disaster-recovery providers that have the equipment and capability we require to recover,” explains Mucia. “Our main recovery challenge is hitting our RTOs with a growing environment of complex and interdependent operating systems and product offerings, but still having finite personnel resources.”
“Our documentation was in some cases non-standardized, and overly complex. Sungard AS helped us better structure our technical recovery documentation, standardize it and ensure it is written for the least common denominator in terms of who executes it.”
The CSG-Sungard AS collaboration is certainly key. Mucia says the Sungard AS team of BC/DR specialists has partnered with CSG’s team to help develop and employ new techniques to ensure that the DR strategy is more effective.
CSG also invests in data-replication appliances and its proprietary and confidential network connectivity between its production data centers and Sungard AS’ Philadelphia data center. This replication is used for data, operating system images for UNIX and virtual machine images for Windows, among other uses. The team also realized that for many legacy environments – such as Sun Solaris, mainframes, along with the x86 and virtual platforms – they had six OS types with over 400 OS instances to manage varying stakeholder targets of MARC (Minimal Acceptable Recovery Configurations). No other vendor in the industry had experience with such a heterogeneous and hybrid environment.
With MRP from Sungard AS, Mucia says that CSG technicians no longer have to build servers in the first few hours after a disaster; Sungard AS professionals take that on. CSG IT professionals can handle connectivity and other essential duties. “It’s good to have Sungard AS in our back pocket,” he says.
He also recalls that when CSG first contracted for MRP services, his team realized the number of OS images was growing rapidly, leaving CSG in a position of needing to expand resources devoted to server builds. He says the Sungard AS team was aware of and understood the CSG environment. That meant Mucia could rely on his team and two-tothree-times that number of Sungard AS professionals to handle any operating system build to recover data.
CSG’s team is in frequent communication with the Sungard AS account members – even though it has never had a disaster. Jenkins has a weekly meeting with the Sungard AS team to review upcoming recovery exercises, which are critical to being prepared for an actual disaster. In addition to traditional recovery activities, CSG uses these exercises to test proof of concepts for new products entering production. For an upcoming exercise for one data center, she already has five proof of concepts planned in addition to the usual data-recovery exercise.
“CSG commonly receives two or three requests a month to demonstrate its DR strategy and test results. With its Sungard AS partnership, CSG can review the test objectives and the tools and products used for the testing and follow-up.” DAVE MUCIA Director of Business Continuity CSG International
While Mucia doesn’t possess a “hard number for ROI”, he cites several different ways CSG values the services he gets from his DR partnership with Sungard AS.
When he first came to CSG a decade ago, it was rare for clients to ask for specific information in terms of DR, either in contracts or in an audit requiring them to prove they could handle a disaster. Now, CSG commonly receives two or three requests a month to demonstrate its DR strategy and test results.
With its Sungard AS partnership, CSG can review the test objectives and the tools and products used for the testing and follow-up. Furthermore, large customers want 45-days’ notice before DR testing including a synopsis of what will be tested, the products used and test expectations. Increased compliance requirements reflect many of these requests.
Previously when conducting exercises, Mucia notes, CSG reserved at least one person in each platform discipline to address any production issue that arose at the same time. That person was expected to handle everything should something major occur during the testing. Today, CSG has its Sungard AS team dedicated to the exercises, to allow production CSG personnel to concentrate on any production issue that comes up during the test. “We’re not nearly as worn out as we used to be,” he says.
Looking to the future, Mucia sees more collaboration with Sungard AS as CSG continues to grow and develop new products and services its clients demand within the current RTOs. This relationship enables CSG to develop and deliver more innovative product offerings which assist its clients in delivering services faster and more effectively.
“CSG doesn’t stand still, and Sungard AS knows you don’t leave us alone, even for a couple of days,” Mucia concludes.
“Our documentation was in some cases non-standardized, and overly complex. Sungard AS helped us better structure our technical recovery documentation, standardize it and ensure it is written for the least common denominator in terms of who executes it.” DAVE MUCIA Director of Business Continuity CSG International