News Stay informed about the latest enterprise technology news and product updates.

Communication failure makes a tough network project even tougher

Sprint gear turned out not to be up to snuff; but crossing that chasm was simpler than changing the whole project plan to keep up with surprises from top management.

The first part of Navigant's networking saga sets the stage for the article that follows.

Navigant's technical savvy made for quicker progress once the project details were settled, but also made for some strong up-front debate with Berbee. Pointing to industry standard practices, Navigant's people questioned Berbee's projections about the number and positioning of Cisco client-manager clusters, for example.

Staffers also debated issues such as whether to tie user data and permissions together using Microsoft Corp.'s Active Directory -- designs and approaches Berbee's team had refined and quantified in a list of documented best practices – or the directory functions in Cisco's CallManager call-processing software. Ultimately Navigant picked CallManager because it was a better fit with the design and usage plan for the network, Dongvillo said.

Navigant part I: How to succeed when the IT is the least of your problems.

Managed-care company nails rush-migration to VoIP

VoIP flows over optimized pipes -- A case study 

VoIP services: A clarification

"It was smart people talking to smart people," said Mike Gelinas, Berbee's account manager. "But as we walked them through our approach, we were able to come to consensus."

The partners split off into multiple project teams comprised of blended talent from Berbee, Navigant and Cisco people. But one move by Berbee proved prescient: It assigned one engineer to serve as project manager with over-arching responsibility for advance site planning.

"The project manager did a phenomenal job of keeping things on track as we approached crunch time on each implementation," says Joe Kennedy, of Navigant's network operations and service desk. "He built a standard 'needs' and scheduling template for every site, and evolved it over time to reflect our experiences with each deployment. With each new site, we were more prepared and more efficient."

Berbee also hosted a collaborative Microsoft SharePoint environment, so the combined teams could share knowledge more quickly.

But the teams' best-laid plans left out one major disrupting factor: Navigant had no plans to slow acquisitions during the monumental makeover. It acquired five more companies between the start of the project in April of 2005 and June 2006, forcing the project team to rebuild the timeline and extend the plan to include the new properties.

Apparently, Navigant's staff took the issue for granted, while Berbee's thought the project was defined without those acquisitions. Oops.

Despite the dueling assumptions, both teams had to adapt. The acquisitions were ongoing, like it or not, and had to be dealt with.

Rolling out the infrastructure while rolling in new businesses -- all in real time -- made a daunting project all the more challenging. And just when the team thought nothing else could go wrong they realized the IP virtual private network (VPN) Sprint provided as wide-area network backbone wasn't doing the job.

"We had started implementing, and gotten about 14 sites in," Kennedy said. "We were having all kinds of voicemail issues, and bad call quality. And after some fighting back and forth, we came to find out that the WAN we were on with [Sprint] just wasn't up to supporting the QOS we needed."

Sprint acknowledged the problem and switched out the IP VPN service for a higher-performing MultiProtocol Label Switching (MPLS) VPN service at no cost, Dongvillo said.

"It has worked -- knock on wood—solid as a rock, for the most part," Kennedy said. "But [the experience] was a forest fire." One that delayed the project five months.

Lessons learned

Dongvillo said the team didn't do enough load testing and call simulation on the Sprint WAN backbone to make sure the system would work. "It was a group thing," he said. "They said it would work, we didn't do enough testing, and we both ended up getting bit on that piece."

Kennedy agrees, but suggested alternative approach that can save time: get the vendors to provide reference accounts from companies with parallel implementations.

"That would have offered a way, quicker than testing, to find out if the configurations we were planning on deploying had any undocumented patterns we should know about," he said.

The infrastructure phase of Navigant's company-wide roll-out finished June of '05, and the company is now moving on to adding advanced services. While the team didn't estimate the project's benefits in advance, they characterize the operational savings as in the "mid-seven figures," according to Kennedy.

Among the quiver of other benefits are:

  • A stable, up-to-date network.

  • Insourced telephony management that saves cost and reduces complexity.

  • Lower support costs for telephony equipment (Cisco is less expensive, because PBX systems traditionally charge per port).

  • Lower or even nonexistent long-distance costs, because the internal network carries a significant proportion of the traffic.

  • Diminished costs for moving users from office to office, because their personal preferences and voice-mailbox are centralized.

  • A unified company phone directory.

  • Mobility features that allow users to take direct calls wherever they are inside the company-wide system.

  • Simpler team-building across offices.

  • And what Navigant officials call "reachability;" making it easy for Navigant's customers to get through to any staff member, wherever they might be located worldwide.

The current phase has Berbee installing IP Contact Center into Navigant's help desk operation. Berbee's Gelinas said it will help the consulting firm better serve clients by linking help desk agents directly with their back-end database, providing immediate access to precise customer experience information.

When that's done, Meeting Place for Conferencing, an IP-based virtual room for interactive meetings and presentations, will be added. Navigant is also planning to use the telephone system as a back-up communications system if e-mail goes down.

"If there's a problem with our e-mail system and we're trying to communicate with users about that, obviously we can't send an email message," Kennedy says. "We can use the voicemail system to distribute messages to the entire staff to explain what's going on, and what they should do or expect."

Berbee's Scheckel is pleased. "Overall it was a fairly large implementation, very complex, rolled out at an aggressive schedule, with constant moving parts with acquisitions," he says. "In the end, it was, without a doubt, very successful. We got it done on time, on budget, and we have an extremely happy customer."

One reason for Navigant's happiness with the implementation, despite the many challenges: Berbee staffed their part of the team without a lot of turnover—an amazing feat, considering the project's death-defying work hours and intensive travel. That consistency paid off, because knowledge gathered from previous office conversions wasn't lost, nor did it have to be transferred to new people.

Within in months after the project was completed, most of the project team had left Navigant, Dongvillo said.

"I don't think there's any magic formula for a successful project like this," Kennedy said. "It's a combination of in-house experience, leaning hard on Cisco and Berbee and forging a team with the dedication to see it through to the very end."

Dongvillo agrees: "There were rough times. There were times when it became like a little family. You have arguments with your family. So it wasn't all smiles and hugs and kisses. To be honest, when you've been up for however many hours, and its 5 a.m., and you're on one of the large sites, people can get a little 'chippy'. But we kept it all together."

Click here for the first part of Navigant's networking saga.

Let us know what you think about the story; e-mail: Kevin Fogarty, News Director

Dig Deeper on Campus area networks and services

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.