Hurricane Irene Offers LessonsInsights on Disaster Recovery Planning
Healthcare organizations must make sure they plan for "high-impact/low-probability" scenarios when building their disaster recovery and business continuity plans, says Linda Reissman, the hospital's director of emergency management.
As the hurricane approached, the hospital wound up customizing its disaster plan to deal with the potential for heavy flooding, she notes. The hospital sandbagged high-risk areas, including the IT department; brought in additional IT staff; and stockpiled extra food, water, bedding and battery-operated lighting so staff could spend the night at the facility.
Reissman stresses the need to involve all departments in business continuity planning to avoid what she calls "assumption planning ... where everyone assumes that someone else is going to do something for them in a disaster."
In an interview with HealthcareInfoSecurity's Howard Anderson (transcript below), Reissman explains how the hospital:
- Accommodated the transfer of 13 patients evacuated from a nearby facility;
- Made arrangements in case it needed to relocate its telecommunications, pharmacy or other departments due to flooding;
- Took steps to ensure it had redundant ways to communicate with certain staff, including satellite phones, multiple cell phone providers and a radio system. This approach was based on lessons learned during the 9/11 attacks, when communications links were cut off, she says. "We also made sure we had a redundant e-mail system as well."
Reissman has 30 years of experience in the municipal, private and governmental sectors. In her role as director of emergency management for Memorial Sloan Kettering Cancer Center, she is responsible for emergency preparedness and business continuity for the 450-bed hospital and 30 regional clinical and business sites. Reissman has also served with the New York City Mayor's Office of Emergency Management, where she spearheaded a $500,000 federal terrorism grant and assisted in the development of the city's bio-terrorism plan and family assistance plan.
HOWARD ANDERSON: Please describe your role as director of emergency management and how you work with others on disaster recovery and business continuity issues?
LINDA REISSMAN: As director of emergency management, I'm responsible for the hospital's overall emergency preparedness. I coordinate with our business continuity planning group here, and of course, we are very heavily involved with our IT department and their disaster recovery processes.
Short-Term PlanningANDERSON: I understand your hospital accepted about 13 patients from another hospital that was evacuated because of the hurricane. What challenges did that create for those involved in the planning effort, and did you have access to those patient's electronic records?
REISSMAN: We received a phone call from one of the nearby hospitals to receive their cancer and bone marrow patients because our hospital is a cancer specialty hospital. Obviously, since we are not normally a 911 receiving hospital, we don't normally get a volume of patients in at one time at such a surge, but it did create certain challenges .... We were able to surge as we needed to. As far as electronic medical records go, there was not a transfer of electronic medical records. Face sheets were transferred via fax. The hospital that was evacuating had over 450 patients ... to evacuate and .. since they had to evacuate within 24 hours, there was no real time to try to coordinate an electronic medical record process. I think it went very well. Patients were pre-registered and went through the pre-admissions process since we had some advanced warning, and within 45 minutes to an hour, 13 patients were delivered to our hospital and were in their rooms without incident.
Customized Disaster PlanANDERSON: Since you had some warning about the approaching storm, I understand you customized your disaster plan for the specific situation that you faced. When it comes to information technology, what specific challenges did the storm pose and how did you address them? And how did you address staffing issues?
REISSMAN: ... We [customized] the plan for such a severe water event; we were worried about flooding. It might have been a category one hurricane in terms of wind, but from a water standpoint or flooding standpoint, it was the equivalent of a category three. With information technology, obviously, you worried about the stability ... of the areas that house your IT infrastructure. We were worried about flooding to those areas and to our backup centers, because it was such a regional storm. Some of the pre-mitigation things that we did were sandbagging and making sure that pumps were in place, making sure that our facilities, and our IT facilities specifically, were identified for the local emergency management groups as a critical infrastructure that in the event flooding occurred, it would be high on their priority list.
As for staffing issues, I know our IT department absolutely had additional staff. We actually developed the plan for them where folks could sleep there if they had to, and that includes bedding, food, extra water, emergency battery-operated lighting, large-scale lighting ... so that we absolutely would be able to have some business continuity in those areas in the event power and other things were lost. Our backup center does have a generator and we just wanted to make sure obviously that was going to remain stable.
... We reorganized our hospital incident command system to actually have an IT chief [who] delegated roles to folks both in the data centers and throughout the hospital. I think that involving them in that level of planning and in that high level of incident command is a critical component for any hospital ... I recommend everyone really involve these folks overall in your incident management planning throughout your hospital, not just from a disaster-recovery standpoint.
ANDERSON: Did you ever lose power during the storm?
REISSMAN: We never lost complete power during the storm. I know that there were some power losses in the area, but we did prepare for power loss. We anticipated losing phone ... service. We made sure that our ... POTS lines or plain-old telephone lines or emergency red phones were all operational. Our IT department went out and assessed areas for emergency power, making sure that critical IT components were within the vicinity of emergency power and could reach it. Phone lists were sent out for the plain-old telephone lines so that in case voice-over-Internet failed, we had that communication capability. We did a lot of mitigation in terms of looking at where we might have to relocate certain areas, even our telecom areas. We had pre-designated areas around the hospital where we [could] move our telephone operators because of flooding or power loss. And we were assured that we had areas both for pharmacy and for other critical areas of the hospital to relocate to. IT did a pre-assessment of those areas to ensure that we were able to transfer that equipment if necessary to another area of the hospital.
Communication RedundanciesANDERSON: Did you have redundant ways to communicate with the IT staff during the emergency?
REISSMAN: Yes we did. Aside from having satellite-phone capability, we have air cards, multiple cell phone providers, an e-notify messaging system, Vocera, Nextel, and radios ... over UHF and VHF. We made sure way back when that we had that capability. In addition to that ... [based on] our experiences post-9/11 of losing communications, one other thing that we did I thought was very important was ... making sure that we had a redundant e-mail system as well and that we could fall over if necessary into another e-mail system and having a back-up in our Outlook system as well. The redundancies are over and over and over again, and that was just a great lesson learned from 9/11, that if a tower goes down, especially in a wind and rain situation like this, you just don't know what's going to be up. We have ... a back-up data center and we were just really sure that we had enough in place so that we would not lose communications.
Lessons LearnedANDERSON: What lessons did you learn from your experience, and what advice would you give to other hospitals about disaster recovery plans based on the hurricane?
REISSMAN: I think the fact that we did business continuity planning with a number of our critical infrastructure departments here gave us a lot of insights into things that I call "assumption planning." Everyone assumes that someone else is going to do something for them in a disaster, and when you start to drill down and really look at what people's strategies are, you realize a lot of times they are not valid or that they just made certain assumptions that things would be taken care of for them.
In any type of business continuity or disaster recovery type planning from an IT perspective, you just want to make sure that the upstreams and downstreams of your impacts are taken into consideration in your planning, and that you drill down really, really deep to see if the strategies that other people are putting in place are indeed valid and the assumptions that they make are indeed valid.
Overall, business continuity planning is very separate and distinct in one way from emergency planning in terms of the types of strategies and drill downs that we do, because it's not incident-specific. It's really class-specific, but yet we have to work very closely both with our DR people and our business continuity people or else emergency management in and of itself won't work. You need to have a much broader perspective.
[The use of] technology now is exponentially increased throughout our industry and to not involve your IT folks at a level of emergency planning and business continuity overall is a grave mistake. And we learned that it's a collective process here at the hospital. It's never stopping. We're constantly reevaluating.
And one more thing is that when you do construction or relocate areas, you really need to do an assessment to make sure that your emergency power is there, that your electrical outlets are there, that your IT equipment is backed up with that capability. A lot of times, with the rapid construction that's going on, sometimes some of these things are overlooked. Those are my recommendations.
ANDERSON: Have you made any changes in your whole approach as a result of your experience?
REISSMAN: We actually just debriefed everyone through an online survey and we had over 100 people in our incident command process answer us in the survey, and we're collating that data now. What you really have to do is listen to what folks say because sometimes you may look at things from one perspective, but you really need to go down to the folks that actually do this on a day-to-day basis and not discount their ideas ... listen really hard about what they think would be a better process. The more information that you have in planning, the better. If you think your theories are good that's one thing; but you need to validate everything that you do.
What I learned from this planning is that no matter how hard you plan, there's always a better way to do stuff and there's always a better way to approach it. ... This [hurricane experience] taught us that what you have in place could always be improved and there's always a challenge that will arise, so you have to really think about the "what if's." A lot of people don't like to do that. They think it's wasteful because it will never happen - the high-impact, low-probability [incidents]. But ... don't discount the high-impact, low-probability events. Good strategies come out of that for all of your planning.