The Prototype Phase
Depending on the design that was decided on by the organization, the prototype phase varies greatly in complexity and duration. It is still critical to perform a prototype, even for the simplest environments, to validate the design, test the mailbox migration process, and ensure that there won't be any surprises during the actual upgrade. The prototype lab should be isolated from the production network via a virtual LAN (VLAN) or physical separation to avoid interfering with the lives of users.
The prototype phase also gives the project team a chance to get acquainted with the new features of Exchange Server 2010 and any new add-on applications that will be used and to configure the hardware in a low-stress environment. If an external company is assisting in this phase, informal or formal knowledge transfer should take place. Ideally, the prototype lab exactly mirrors the final messaging configuration so that training in this environment will be fully applicable to the administration and support skills needed after the upgrade.
Always take advantage of the unique opportunities granted to you in the prototype phase. Because the prototype is built as a replica of the planned production design, you can practice disaster recovery, server deployment, mailbox moves, and application integrations with no concerns about impacting users the way they would be in production.
What Is Needed for the Lab?
At a bare minimum, the lab should include a new Exchange Server 2010 server, one each of the standard desktop and laptop configurations, the tape drive that will be used to back up the public and private Information Stores, and application software as defined in the Design document. Connectivity to the Internet should be available for testing OWA and Windows Mobile access. You will also need at least one domain controller that is configured as a global catalog. The preferred method to deploy this domain controller is to promote a spare domain controller in production and after it has fully replicated, remove it from the network and move it to the lab network. After being isolated, seize the Flexible Single Master Operations (FSMO) roles on the lab domain controller. In production, use NTDSUTIL to perform a metadata cleanup to remove the references to the temporary domain controller. In this way, you have an accurate view of Active Directory for the prototype phase. This can be especially helpful because directory problems that would show up in a production migration will appear in the lab.
Existing data stores should be checked for integrity and then imported to Exchange Server 2010 to ensure that the process goes smoothly. Exchange Server 2010 comes with improved mailbox migration tools, which are more resistant to failure when corrupt mail-boxes are encountered and are multithreaded for better performance.
The recommended route for customers with Exchange Server 2007 or 2003 servers to get to Exchange Server 2010 is to install an Exchange Server 2010 server into the environment and move mailboxes. If hardware availability is limited, consider upgrading one location at a time and use the "replaced" server as the new Exchange Server 2010 server in the next site. This assumes the hardware is capable of running Exchange Server 2010 and is appropriately sized. This method is often referred to as a "leap frog" upgrade.
If site consolidation or server consolidation are goals of the project, the prototype lab can be used for these purposes. Multiforest connectivity can now be tested, but this requires a Microsoft Identity Integration Services server in one or more of the forests to enable directory synchronization.
Exchange Server 2010 also comes with a number of new tools to aid in the testing and migration process, which are covered in detail in Chapters 15 and 16. These include a prescriptive guide that walks through the deployment process, preparation tools that scan the topology and provide recommendations, and validation tools.
For more complex environments and larger companies, the lab should be kept in place even after the upgrade is completed. Although this requires the purchase of at least one additional Exchange server and related software, it provides a handy environment for testing patches and upgrades to the production environment, performing offline database maintenance, and in worst-case scenarios, a server to scavenge from in times of dire need.
Depending on the complexity of the Exchange Server environment, this long-term lab might potentially be run in a virtual environment. Deploying the lab via VMware or Microsoft's Hyper-V allows you to mimic the interactions of multiple servers and server roles on a single box. Both VMware and Microsoft's Hyper-V solutions support 64-bit guest operating systems and, therefore, are viable options for an Exchange Server 2010 lab environment.
After the lab is configured to match the end state documented in the Design document, representative users from different departments with different levels of experience and feature requirements should be brought in and given a chance to play with the desktop configurations and test new features and remote access. Input should be solicited to see whether any changes need to be made to the client configurations or features offered, and to help get a sense for the training and support requirements.
Disaster Recovery Testing
Another important testing process that can be performed prior to implementation of the new solution on the live network is business continuity or disaster recovery testing. Ideally, this was covered in the design process, and disaster recovery requirements were included in the design itself. Definitely take advantage of practicing your disaster recovery process in the prototype phase. This is likely your only opportunity to create and destroy servers without regard for impacting end users.
Documentation from the Prototype
During the prototype phase, a number of useful documents can be created that will be useful to the deployment team during the pilot and production upgrade phases, and to the administrators when the upgrade is complete.
As-built documents capture the key configuration information on the Exchange Server 2010 systems so that they can easily be replicated during the upgrade or rebuilt from scratch in case of catastrophic failure. Generally, the as-built documents include actual screenshots of key configuration screens to facilitate data entry. Having carefully prepared as-built documents allows you to go into production with a well-tested build process. Not unlike a disaster recovery situation, you want to simply follow your own instructions during the deployment; you don't want to have to learn as you go.
Assuming that disaster recovery requirements for the project were defined as suggested previously, this is a perfect time to summarize the testing that was performed in the lab and record the steps a knowledgeable administrator should take in the failure scenarios tested.
One last item of value to take out of the prototype phase is a well-documented list of any surprises that came up during the testing. If you tested the move mailbox process from an Exchange Server 2007 server that was restored from production and you had errors moving mailboxes, you can expect to have these exact same errors in the production move. If you were able to solve the issues in the lab, you should have well-documented notes on how to deal with the same error in production. Being prepared in this manner is the key to a smooth migration.
Final Validation of the Migration Document
When the testing is complete, the migration plan should be reviewed a final time to make sure that the testing process didn't reveal any showstoppers that will require a change in the way the upgrade will take place or in the components of the final messaging solution.
The end users who have had a chance to get their feet wet and play with the new Outlook 2007 client and learn about the new capabilities and enhanced performance of Exchange Server 2010 should be spreading the word by now, and the whole company should be excited about the upgrade!
The Pilot Phase: Deploying Services to a Limited Number of Users
With the testing completed, the Exchange Server 2010 upgrade team has all the tools needed for a successful upgrade, assuming the steps outlined so far in this chapter have been followed. The Design document is updated based on the prototype testing results so that the end state that the executives and decision makers are expecting has been conceptually proven. Unpleasant surprises or frantic midnight emails requesting more budget are nonexistent. The road map of how to get to the end state is created in detail, with the project schedule outlining the sequential steps to be taken and the Migration document providing the details of each step. Documentation on the exact server configurations and desktop configuration are created to assist the systems engineers who will be building and configuring the production hardware.
The project team has gained valuable experience in the safe lab environment, processes have been tested, and the team is brimming with confidence. End users representing the different departments, who tested and approved the proposed desktop configurations, are excited about the new features that will soon be available.
To be on the safe side, a rollback strategy should be clarified, in case unforeseen difficulties are encountered when the new servers are introduced to the network. Disaster recovery testing can also be done as part of the first pilot, so that the processes are tested with a small amount of data and a limited number of users.
The First Server in the Pilot
The pilot phase officially starts when the schema is extended and the first Exchange Server 2010 server is implemented in the production environment. The same testing and sign-off criteria that were used in the lab environment can be used to verify that the server is functioning properly and coexisting with the present Exchange servers. Surprises might be waiting that will require some troubleshooting because the production environment will add variables that weren't present in the lab, such as large quantities of data-consuming bandwidth, non-Windows servers, network management applications, and applications that have nothing to do with messaging but might interfere with Exchange Server 2010.
The migration of the first group of mailboxes is the next test of the thoroughness of the preparation process. Depending on the complexity of the complete design, it might make sense to limit the functionality offered by the first pilot phase to basic Exchange Server 2010 functionality, and make sure that the foundation is stable before adding on the higher-end features, such as voice mail integration, mobile messaging, and faxing. The first server should have virus-protection software and backup software installed. Remote access via OWA is an important item to test as soon as possible because there can be complexities involved with demilitarized zone (DMZ) configurations and firewalls.
Choosing the Pilot Group
The first group of users, preferably more than 10, represents a sampling of different types of users. If all members of the first pilot group are in the same department, the feedback won't be as thorough and revealing as it could be if different users from different departments with varied needs and expectations are chosen. It's generally a good idea to avoid managers and executives in the first round, no matter how eager they are, because they will be more likely to be the most demanding, be the least tolerant of interruptions to network functionality, and have the most complex needs.
Although a great deal of testing has taken place already, these initial pilot users should understand that there will most likely be some fine-tuning that needs to take place after their workstations are upgraded; they should allocate time from their workdays to test the upgrades carefully with the systems engineer performing the upgrade. This will correctly set the expectation for the pilot users, as well as allow the upgrade team to get the feed-back they need before moving into the full migration.
After the initial pilot group is successfully upgraded and functional, the number of users can be increased because the upgrade team will be more efficient and the processes fine-tuned to where they are 99% error free.
For a multisite messaging environment, the pilot process should be carefully constructed to include the additional offices. It might make sense to fully implement Exchange Server 2010 and the related messaging applications in the headquarters before any of the other locations, but issues related to WAN connectivity might crop up later, and then the impact is greater than if a small pilot group is rolled out at HQ and several of the other offices. It is important to plan where the project team and help desk resources will be, and they ideally should travel to the other offices during those pilots, especially if no one from the other office participated in the lab testing phase. Be sure to have sufficient coverage for issues that might arise if the pilot groups span multiple time zones.
The help desk should be ready to support standard user issues, and the impact can be judged for the first few sub-phases of the pilot. Issues encountered can be collected and tracked in a knowledge base, and the most common issues or questions can be posted on the company intranet or in public folders, or used to create general training for the user community.
Gauging the Success of the Pilot Phase
When the pilot phase is complete, a sampling of the participants should be asked for input on the process and the results. Few companies do this on a formal basis, but the results can be very surprising and educational. Employees should be informed of when the upgrade will take place, that no data will be lost, and that someone will be there to answer questions immediately after the upgrade. Little changes to the workstation environment, such as the loss of favorites or shortcuts, or a change in the network resources they have access to, can be very distressing and result in disgruntled pilot testers. Your goal is for your employees to be happy about the upgrade experience after it's been done. Their opin-ions will reach the rest of your users and they'll be a lot more cooperative if they aren't expecting to have problems.
A project team meeting should be organized to share learning points and review the final outcome of the project. The company executives must now make the go-no-go decision for the full migration, so they must be updated on the results of the pilot process.
The Production Migration/Upgrade
When the pilot phase is officially completed and any lingering problems have been resolved with the upgrade process, there will typically be 10%--20% of the total user community upgraded. The project team will have all the tools it needs to complete the remainder of the upgrade without serious issues. Small problems with individual workstations or laptops will probably still occur, but the help desk should be familiar with how to handle these issues by this point.
A key event at this point is the migration of large amounts of Exchange Server data. The public and private Information Stores should be analyzed with eseutil and isinteg, and complete backup copies should be made in case of serious problems. The project team should make sure that the entire user community is prepared for the migration and that training has been completed by the time a user's workstation is upgraded.
It is helpful to have a checklist for the tasks that need to be completed on the different types of workstations and laptops so that the same steps are taken for each unit, and any issues encountered can be recorded for follow-up if they aren't critical. Laptops will most likely be the most problematic because of the variation in models, features, and user requirements, and because the mobile employees often have unique needs when compared to workers who remain in the office. If home computers need to be upgraded with the Outlook 2007 client and if, for instance, the company VPN is being retired, these visits need to be coordinated.
As with the pilot phase, the satisfaction of the user community should be verified. New public folders or SharePoint discussions can be started, and supplemental training can be offered for users who might need some extra or repeat training.
If at all possible, get your users to clean up their mailboxes and clear their deleted items prior to the migration, as this can result in a very large time savings. Experience has shown that typically 50% of the data moved in an Exchange Server migration is Sent Items and Deleted Items. The time it takes to move a mailbox is more affected by the item count than the overall size of the mailbox in gigabytes.
Decommissioning the Old Exchange Server Environment
As mentioned previously, some upgrades require legacy Exchange servers to be kept online, if they are running applications that aren't ready or can't be upgraded right away to Exchange Server 2010. Even in environments where the Exchange Server 2003 or 2007 servers should be completely removed, this should not necessarily be done right away.
Supporting the New Exchange Server 2010 Environment
After the dust has settled and any lingering issues with users or functionality have been resolved, the project team can be officially disbanded and returned to their normal jobs. If they haven't been created already, Exchange Server Maintenance documents should be created to detail the daily, weekly, monthly, and quarterly steps to ensure that the environment is performing normally and the databases are healthy.
If the prototype lab is still in place, this is an ideal testing ground for these processes and for testing patches and new applications. By following the Exchange Server Maintenance documents and keeping up with regular maintenance tasks, you will be much less likely to have issues with your Exchange Server 2010 environment in the future.
Someone famous once said, "It's not the destination, it's the journey." In the case of an Exchange Server 2010 upgrade, or any project for that matter, it's both. This chapter has shown that the key to success in a major undertaking such as an Exchange Server 2010 upgrade is to follow a strong methodology that accounts for both the journey and the destination.
The use of a discovery phase allows the people who will be involved in the project to gather as much information as they can about the existing state of the environment, as well as the needs of the environment. This prepares them to make design decisions that will allow them to support the needs of the business without putting the existing environment at risk.
A design phase allows the group to work interactively to design a new end state that best provides for the needs of the company. A key concept to keep in mind during a design phase is that there is no "one perfect design" -- there is only a design that is most appropriate for you and your needs and limitations.
A prototype phase allows you to validate your design and your migration methodologies by testing them in a safe replica environment. This allows you to discover potential problems before they come up in a production migration. Always take advantage of the proto-type phase to try out the "what-if" questions that will result in you and your team having a stronger knowledge of how the new environment will work.
The pilot phase allows you to try your migration steps in the real world with reduced exposure to problems through a controlled membership of pilot users. Take this opportunity to get feedback from the pilot users to update or modify your steps to reduce impact on users or administrators. Remember, if you need to make major changes after the pilot, run a second pilot and keep running pilots until you feel your process is sufficient. This shouldn't take too much feedback if you took full advantage of the prototype phase.
The implementation phase allows you to push through the migration full force and get all the users migrated to the new environment.
Utilize a support and retirement phase to make sure you have time to retire old servers and to make sure you have a bit of extra time with the enhanced support and help desk to make sure everyone is happy after the migration (or at least happy about Exchange Server 2010).
By following this standard methodology, you will greatly increase the chances of having a smooth and uneventful migration. This will help build credibility for the IT organization and make it that much easier to get projects approved in the future.
The following are best practices from this chapter:
- An upgrade to Exchange Server 2010 should follow a process that keeps the project on schedule. Set up such a process with a four-phase approach, including initiation, planning, testing, and implementation.
- Documentation is important to keep track of plans, procedures, and schedules. Create some of the documentation that could be expected for an upgrade project, including a Statement of Work document, a Design document, a project schedule, and a Migration document.
- Key to the initiation phase is the definition of the scope of work. Create such a definition, identifying the key goals of the project.
- Make sure that the goals of the project are not just IT goals, but also include goals and objectives of the organization and business units of the organization. This ensures that business needs are tied to the migration initiative, which can later be quantified to determine cost savings or tangible business process improvements.
- Set milestones in a project that can ensure that key steps are being achieved and the project is progressing at an acceptable rate. Review any drastic variation in attaining milestone tasks and timelines to determine whether the project should be modified or changed, or the plans reviewed.
- Allocate skilled or qualified resources that can help the organization to better achieve technical success and keep it on schedule. Failure to include qualified personnel can have a drastic impact on the overall success of the project.
- Identify risks and assumptions in a project to provide the project manager with the ability to assess situations and proactive work and avoid actions that might cause project failures.
- Plan the design around what is best for the organization, and then create the migration process to take into account the existing configuration of the systems within the organization. Although understanding the existing environment is important to the success of the project, an implementation or migration project should not predetermine the actions of the organization based on the existing enterprise configuration.
- Ensure that key stakeholders are involved in the ultimate design of the Exchange Server 2010 implementation. Without stakeholder agreement on the design, the project might not be completed and approved.
- Document decisions made in the collaborative design sessions, as well as in the migration planning process, ensure that key decisions are agreed upon and accepted by the participants of the process. Anyone with questions on the decisions can ask for clarification before the project begins rather than stopping the project midstream.
- Test assumptions and validate procedures in the prototype phase. Rather than learn-ing for the first time in a production environment that a migration will fail because an Exchange Server database is corrupt or has inconsistencies, the entire process can be tested in a lab environment without impacting users.
- Test the process in a live production environment with a limited number of users in the prototype phase. Although key executives (such as the CIO or IT director) may want to be part of the initial pilot phase, it is usually not recommended to take such high-visibility users in the first phase. The pilot phase should be with users who will accept an incident of lost email or inability to send or receive messages for a couple of days while problems are worked out. In many cases, a prepilot phase could include the more tolerant users, with a formal pilot phase including insistent executives of the organization.
- Migrate, implement, or upgrade after all testing has been validated. The production process should be exactly that: a process that methodically follows procedures to implement or migrate mailboxes into the Exchange Server 2010 environment.
Planning, Prototyping, Migrating, and Deploying Exchange Server 2010
Initiating an Exchange 2010 upgrade: identifying goals
Using an Exchange Server 2010 design document for upgrades
Using a migration document for Exchange Server 2010
Exchange Server 2010 upgrade: Prototype, pilot phases
Printed with permission from Sams Publishing. Copyright 2009. Exchange Server 2010 Unleashed by Rand Morimoto, Michael Noel, Chris Amaris, Andrew Abbate and Mark Weinhardt. For more information about this title and other similar books, please visit Sams Publishing.
This was first published in April 2010