Network upgrades -- especially on core components like switches and routers -- can be disruptive to your client's...
business and can cripple your reputation. Most disruptions are brief, but they can reduce productivity and cause revenue loss for clients. What's more, unforeseen problems can exacerbate disruptions, causing significant downtime and a possible rollback to prior devices. Fortunately, there are some tactics that solution providers can use to minimize the impact of infrastructure upgrades.
Plan and test upgrades in advance
Plan and test network upgrades well in advance of the actual job. That means, for instance, getting familiar with a new switch or router in the lab environment beforehand.
"A solution provider who tests a switch or router prior to installation, simulating the client's environment, can significantly minimize downtime and disruptions by working out any issues prior to installation," said Dave Sobel, CEO of Evolve Technologies, a solution provider headquartered in Fairfax, Va. Many solution providers will acquire the new switch or router and actually perform simulated upgrades in a lab that mimics the customer's site. This may also include pre-configuring the new device in advance to reduce setup time during the actual deployment.
Even with planning and testing, always have a rollback plan in place. Upgrades are often completed without a hitch, but if trouble does occur, it may be necessary to roll back a problematic deployment while the trouble is identified and a solution is found. In that case, the deployment can usually be rescheduled to a more suitable time.
Keep the client informed
Clients hate surprises, and experts say that keeping a client involved in the upgrade plan right from the beginning is often the best way to ease pressures and stay flexible as problems arise. "Let them know what the plan is and when you intend to execute it," said Karl W. Palachuk, CEO of KPEnterprises Business Consulting Inc., a small business IT consulting firm located in Sacramento, Calif. "Tell them which systems will be affected and what they can expect."
Schedule upgrades for the least disruptive time
Remember that downtimes and disruptions are not necessarily the same thing. A client may schedule regular daily, weekly, or monthly downtime periods (sometimes called "planned outages") for maintenance and support purposes. Coordinate deployments with regular client downtimes to minimize disruption. If the upgrade can be completed within the scheduled downtime, there may be no disruption at all from a business standpoint. If not, the additional downtime will be minimized. If there are no suitable downtime windows, plan new upgrade cycles for evenings or weekends (or other periods when user traffic is lowest).
Back up the current configuration
Create a copy of the current switch/router configuration before taking the device out of service. The goal is to create a backup of the old configuration that can serve as a foundation for new device configurations, as well as protecting the old configuration if it's needed for a rollback. "This is key to making sure you can get back to where you were, in case something goes wrong," Palachuk said. "Upgrades almost never go wrong. But when they do, you need a quick step back to the previous state."
Have vendor support contacts handy
When trouble strikes, it may be necessary to contact vendor support services for the new device (or even the old device). "After all, if something goes wrong in the middle of your upgrade, you're going to need to call someone. Set that process in place before you start," Palachuk said. Gather up all of the contact information necessary to reach the vendor's support resources, such as product makes, models, part numbers, serial numbers, MAC numbers, firmware version numbers and so on. Also, be sure to have service contracts or warranty identification details, otherwise you or your customer may be billed for support calls.
In fact, you can even pre-empt the initial delay of support calls by opening a help desk ticket with the vendor before getting started -- especially for mission-critical upgrades. "The first call can sometimes take an hour to get going. We'd rather do that before we take the systems down," Palachuk said. "In this way, if something goes wrong, we have a ticket number and we can get quick support."
Develop the client's network to avoid single points of failure
Switches and routers are often single points of failure in many network designs, so upgrading the switch or router will usually result in some amount of disruption for the client. However, clients that need to minimize network disruptions or downtime should consider using high-resiliency network infrastructures utilizing multiple switches/routers in a failover configuration. "They'll have multiple paths -- no single points of failure," said Steve Reese, director of solutions marketing at Nexus Information Systems, a Valencia, Calif.-based solution provider. "I can failover to my secondary while I perform the primary upgrade," he said, "and once the primary is upgraded, I can fail back and I can upgrade the secondary … with zero outage."
Document changes for the customer
Make it a priority to document any changes resulting from the switch or router upgrades in accordance with the client's process or change control policies. While documentation can't minimize disruptions for the work that's been completed, it can help to prevent problems or oversights on future projects. Many solution providers include documentation work as a line-item cost in the overall project proposal, while other providers offer documentation services as a separate billable task.