SAs [system administrators] continually find new opportunities to centralize processes and services. [Network] centralization does not innately improve efficiency. It brings about the opportunity to introduce new economies of scale to a process. What improves efficiency is standardization, which is usually a by-product of centralization. The two go hand in hand.
To the previous owners of the service being centralized, centralization is about giving up control. Divisions that previously provided their own service now have to rely on a centralized group for service. SAs who previously did tasks themselves, their own way, now have to make requests of someone else who has his or her own way to do things. The SAs will want to know whether the new service provider can do things better.
Before taking control away from a previous SA or customer, ask yourself what the customer's psychological response will be.Will there be attempts to sabotage the effort? How can you convince people that the new system will be better than the old system? How will damage control and rumor control be accomplished? What's the best way to make a good first impression?
The best way to succeed in a centralization program is to pick the right services for centralization. Here are some good candidates.
- Distributed Systems: Management of distributed systems. Historically, each department of an organization configured and ran its own web servers. As the technology got more sophisticated, less customization of each web server was required. Eventually, there was no reason not to have each web server configured exactly the same way, and the need for rapid updates of new binaries was becoming a security issue. The motivation was to save money by not requiring each department to have a high level of web server expertise. The problem being fixed was the lack of similar configurations on each server. A system was designed to maintain a central configuration repository that would update each of the servers in a controlled and secure manner. The customers affected were the departmental SAs, who were eager to give up a task that they didn't always understand. By centralizing web services, the organization could also afford to have one or more SAs become better-trained in that particular service, to provide better in-house customer support.
- Consolidation: Consolidate services onto fewer hosts. In the past, for reliability's sake, one service was put on each physical host. However, as technology progresses, it can be beneficial to have many services on one machine. The motivation is to decrease cost. The problem being fixed is that every host has overhead costs, such as power, cooling, administration, machine room space, and maintenance contracts. Usually, a single, a more powerful machine costs less to operate than several smaller hosts. As services are consolidated, care must be taken to group customers with similar needs.
Since the late 1990s, storage consolidation has been a big buzzword. By building one large storage-area network that each server accesses, there is less "stranded storage"—partially-full disks—on each server. Often, storage consolidation involves decommissioning older, slower, or soon-to-fail disks and moving the data onto the SAN, providing better performance and reliability.
Server virtualization, a more recent trend, involves using virtual hosts to save hardware and license costs. For example, financial institutions used to have expensive servers and multiple backup machines to run a calculation at a particular time of the day, such as making end-of-day transactions after the stock market closes. Instead, a virtual machine can be spun up shortly before the market closes; the machine runs its tasks, then spins down. Once it is done, the server is free to run other virtual machines that do other periodic tasks.
By using a global file system, such as a SAN, a virtualization cluster can be built. Since the virtual machine images—the data stored on disk that defines the state of a virtual machine—can be accessed from many hardware servers, advanced virtualization management software can migrate virtual machines between physical machines with almost unnoticable switch-over time. Many times, sites realize that they need many machines, each performing a particular function, none of which requires enough CPU horsepower to justify the cost of dedicated hardware. Instead, the virtual machines can share a farm, or cluster, of physical machines, as needed. Since virtual machines can migrate between different hardware nodes, workload can be rebalanced. Virtual machines can be moved off an overloaded physical machine. Maintenance becomes easier too. If one physical machine is showing signs of hardware problems, virtual machines can be migrated off it onto a spare machine with no loss of service; the physical machine can then be repaired or upgraded.
- Administration: System administration. When redesigning your organization (see Chapter 30), your motivation may be to reduce cost, improve speed, or provide services consistently throughout the enterprise. The problem may be the extra cost of having technical management for each team or that the distributed model resulted in some divisions' having poorer service than others. Centralizing the SA team can fix these problems.
To provide customization and the "warm fuzzies" of personal attention, subteams might focus on particular customer segments. An excellent example of this is a large hardware company's team of "CAD ambassadors," an SA group that specializes in cross-departmental support of CAD/CAM tools throughout the company. However, a common mistake is to take this to an extreme. We've seen at least one amazingly huge company that centralized to the point that "customer liaisons" were hired to maintain a relationship with the customer groups, and the customers hired liaisons to the centralized SA staff. Soon, these liaisons numbered more than 100. At that point, the savings in reduced overhead were surely diminished. A regular reminder and dedication to the original motivation may have prevented that problem.
- Specialization: Expertise. In decentralized organizations, a few of the groups are likely to have more expertise in particular areas than other groups do. This is fine if they maintain casual relationships and help one another. However, certain expertise can become critical to business, and therefore an informal arrangement becomes an unacceptable business risk. In that case, it may make sense to consolidate that expertise into one group. The motivation is to ensure that all divisions have access to a minimum level of expertise in one specific area or areas. The problem is that the lack of this expertise causes uneven service levels, for example, if one division had unreliable DNS but others didn't or if one division had superior Internet email service, whereas others were still using UUCP-style addresses. (If you are too young to remember UUCP-style addresses, just count your blessings.) That would be intolerable!
Establishing a centralized group for one particular service can bring uniformity and improve the average across the entire company. Some examples of this include such highly specialized skills as maintaining an Internet gateway, a software depot, various security issues—VPN service, intrusion detection, security-hole scanning, and so on—DNS, and email service. A common pattern at larger firms is to create a "Care Services" or "Infrastructure" team to consolidate expertise in these areas and provide infrastructure across the organization.
- Left Hand, Right Hand: Infrastructure decisions. The creation of infrastructure and platform standards can be done centrally. This is a subcase of centralizing expertise. The motivation at one company was that that infrastructure costs were high and interoperability between divisions was low. There were many specific problems to be solved. Every division had a team of people researching new technologies and making decisions independently. Each team's research duplicated the effort of the others. Volume-purchasing contracts could not be signed, because each individual division was too small to qualify. Repair costs were high because so many different spare parts had to be purchased. When divisions did make compatible purchasing decisions, multiple spare parts were still being purchased because there was no coordination or cooperation. The solution was to reduce the duplication in effort by having one standards committee for infrastructure and platform standards. Previously, new technology was often adopted in pockets around the company because some divisions were less averse to risk; these became the divisions that performed product trials or became early adopters of new technology.
This last example brings up another benefit of centralization. The increased purchasing power should mean that better equipment can be purchased for the same cost. Vendors may provide better service, as well as preferred pricing, when they deal with a centralized purchasing group that reflects the true volume of orders from that one source. Sometimes, money can be saved through centralization. Other times, it is better to use the savings to invest in better equipment.
- Commodity: If it has become a commodity, consider centralization. A good time to consider centralizing something is when the technology path it has taken has made it a commodity. Network printing, file service, email servers, and even workstation maintenance used to be unique, rare technologies. However, now these things are commodities and excellent candidates for centralization.
Case Study: Big, Honkin' File Servers
Tom's customers and even fellow SAs fought long and hard against the concept of large, centralized file servers. The customers complained about the loss of control and produced, in Tom's opinion, ill-conceived pricing models that demonstrated that the old UNIX-based file servers were the better way to go. What they were really fighting was the notion that network file service was no longer very special; it had become a commodity and therefore an excellent candidate for centralization. Eventually, an apples-to-apples comparison was done. This included a total cost-of-ownership model that included the SA time and energy to maintain the old-style systems. The value of some unique features of the dedicated file servers, such as file system snapshot, was difficult to quantify. However, even when the cost model showed the systems to cost about the same per gigabyte of usable storage, the dedicated file servers had an advantage over the old systems: consistency and support. The old systems were a mishmash of various manufacturers for the host; for the RAID controllers; and for the disk drives, cables, network interfaces, and, in some cases, even the racks they sat in! Each of these items usually required a level of expertise and training to maintain efficiently, and no single vendor would support these Frankenstein monsters. Usually, when the SA who purchased a particular RAID device left the group, the expertise left with the person. Standardizing on a particular product resulted in a higher level of service because the savings were used to purchase top-of-the line systems that had fewer problems than inexpensive competitors. Also, having a single phone number to call for support was a blessing.
Printing is another commodity service that has many opportunities for centralization, both in the design of the service itself and when purchasing supplies.
Network centralization and decentralization
Candidates for centralization
Candidates for decentralization
Reproduced from the Addison-Wesley Professional book The Practice of System and Network Administration, 2nd Edition, by Thomas A. Limoncelli, Christina J. Hogan and Strata R. Chalup. ISBN 978-0321492661. Copyright 2007, Addison-Wesley Professional. Reproduced by permission of Pearson Education Inc., 800 East 96th St., Indianapolis, IN 46240. Written permission from Pearson Education Inc. is required for all other uses.