Sorry, the language specified is not available for this page

    // Blog

    Disaster recovery lessons learned: 5 takeaways from 4,000+ disasters

    March 29, 2021 | By Sungard AS |

    We’ve helped businesses recover from more than 4,000 disasters in the last 40 years.

    We’ve seen hurricanes, cyberattacks, floods, fires and plenty of power outages. We’ve also seen some unprepared companies nearly wiped out by natural disasters, terrorist attacks, cyberattacks and more.

    What separates the companies that successfully navigate disasters from those that never recover?

    Here are four disaster recovery (DR) lessons we’ve learned from thousands of disasters that can help you prepare for the worst.

    1. Plan for displacement and data loss

    These are the two most common challenges after a disaster.

    Traditionally, the primary threat of a disaster has been displacement. Hurricanes, wildfires, flooding, power outages or even a car through the front of the building can displace your employees if they shut down your facilities.

    In this case, employees need an off-site recovery center where they have power, connectivity and access to a replication of your network and data to get back to work. As we learned during the COVID-19 pandemic, sometimes a recovery center is not the best option if employees can’t gather in enclosed spaces, so it’s important to have other options for employees to get online.

    One thing to consider is concentration risk. You don’t want all employees with the same knowledge and skill sets in one building or region where they can all be simultaneously impacted by the same disaster. By ensuring you have similar talent spread across multiple geographic areas, you’ll be better able to keep your business running even if you experience a hurricane during a pandemic, as many did in 2020. 

    In addition to displacement, data loss has become a major threat in the aftermath of a cyberattack or disaster. Cyberattacks often happen weeks or months before anyone notices, which means malware can spread and potentially corrupt your backups. Without a segmented network or multiple copies, your data is at risk.  

    This is where a compromised data recovery program will come in handy. Even if you do lose data following a disaster, you may still be able to recreate the data you weren’t able to recover

    2. Changes in production derail DR plans

    You’re ready for a disaster. You have your DR plan fine-tuned. But when’s the last time you made changes to your environment? Often companies upgrade their compute, memory or storage, or make other changes in production without corresponding adjustments in their DR plan.

    You also must account for different systems. Some organizations have one type of system, and some have five, seven or more. Integrating those multiple systems in production creates daily tasks to ensure they stay integrated. When you go to recover, those same considerations must be brought into recovery.

    This is why it’s so important to conduct a business impact analysis (BIA) and regularly test your DR plan

    Evaluating changes in production to see how they impact your recovery configuration is an easy way to avoid a slower, more complicated, or flat-out impossible recovery when the worst-case scenario hits.

    Whether you’re handling DR in house and need to be more vigilant, or you use a partner and need to shift your contract to ensure your new configuration is covered, make sure you account for every change in production.

    3. The cloud increases your resilience

    Years ago, one of the best options for storing data to recover offsite was tape backup. Once you copied data to the tapes, you’d physically move them to an offsite storage location. If the data center went down, you would drive out to get the tapes and recover from them.

    While tape backup may still be a viable option for some businesses, now we have the cloud. It’s much more seamless and faster than tapes, and can reduce the cost of downtime, which averages more than $4,300 per minute. Using the cloud for storage replication across a dedicated network ensures site-to-site recovery and prevents data loss. It’s also more cost-effective, since you only pay for what you use, and it scales with your data.  

    However, you shouldn’t rush into the cloud without performing your due diligence first. Whether it’s a public, private or hybrid cloud solution, make sure you choose a platform – or platforms –according to what’s best for your business needs.

    4. RTO is relative

    The key question in disaster recovery is, “how long can you be down before you have to be back up and running?” The answer to that question varies widely by company and industry.

    If your business is 100% online, you’ll have a low tolerance for any downtime in your ordering system. If you’re a manufacturing company, however, that same downtime might be less of a disruption if you already have production orders lined up to work on for several days. Regulatory requirements for financial services and other industries might dictate RTO as well.

    Understand that not all applications and data are the same, so it’s important that you identify which are most critical to successfully conduct your business. By tiering your applications, you can bring back the most important ones first, minimizing the most costly downtime. For example, sales, marketing and manufacturing applications might be considered Tier 1 because they’re vital to revenue generation, while human resources applications and data may fall under Tier 2.

    However, as your business evolves, your mission-critical applications may evolve as well. So, make sure you’re keeping up with – and factoring in – these changes and how they impact your DR plan.

    Maintaining a proactive approach to resilience

    Disasters can happen when you least expect it. You might not be able to prevent the storm, but you can prepare your company to handle the situation by taking the necessary precautions to keep your business up and running with minimal disruptions.

    Paying attention to these four lessons is a good place to start. This is what the most resilient companies do, and it’s why they’ve stayed afloat while others have not.