Solution provider takeaway: Virtual machines can be created quickly and easily, creating a common problem called virtual machine sprawl. This chapter excerpt, from VMware ESX Essentials in the Virtual Data Center, provides tips on how to bring optimal performance back into a client's environment and how to avoid common misdiagnosis of performance issues.
Configuration of the Virtual Machine
When you create virtual machines for your environment, accepting default responses aren't necessarily the right choice. Will it work? Sure. Will it be optimized for performance? Perhaps not. There are things that you should be aware of and things that you can do with the configuration of your virtual machines to squeeze out additional performance.
Remove Unneeded Virtual Hardware
One of the nice things about working with a virtual machine is the ease with which you can add or remove hardware components. You don't need any tools, and you don't need to open the hood of the server. A quick and easy way to gain a small amount of performance back in your guest environment is to disconnect or remove any unused or unnecessary devices. If your virtual machine doesn't need a CD-ROM drive, Floppy drive, Network Adapter or COM and LPT ports, get rid of them. When you do need them, it is extremely easy to either enable them or add them back. This can free up IRQ resources and it eliminates IRQ sharing conflicts.
Power off Idle Virtual Machines
Because virtual machines can so quickly and easily be created, a common problem called virtual machine sprawl has erupted. If it isn't managed properly with processes and controls, unmanaged and forgotten virtual machines can spring up and consume datacenter resources. It is important to identify virtual machines that are no longer necessary or no longer being used, because these virtual machines should be powered off or suspended to keep them from wasting valuable resources. Even if a virtual machine is idle, it still uses memory and CPU that other virtual machines could be leveraging for their own performance.
Virtual Machine Processors
When a virtual machine's performance seems sluggish, many people automatically assume that it just needs more processing power. And since VMware has upped the ante on the number of virtual processors that can be assigned to any given virtual machine, why not just add more virtual CPUs to the virtual machine to increase performance? Unfortunately, that doesn't always work. In fact, in many cases, doing so may adversely affect the entire environment and further degrade performance.
Adding additional processors to a virtual machine by using VMware's Virtual SMP is not always going to solve the performance problem. Why? First off, your performance problem may not even be caused by a lack of processing power. Your bottleneck could be memory, disk I/O or network I/O. If the guest operating system or applications aren't being starved for CPU resources, adding more CPUs may only negatively affect the rest of the environment on that host server.
And second, not all operating systems and applications are able to take advantage of having multiple CPUs. If your application is single threaded and will not benefit from the additional VCPUs, you shouldn't use Virtual SMP. In this case, adding Virtual SMP may consume valuable physical processor resources without actually offering the virtual machine any added performance benefits, and as a result, take away resources from other virtual machines on the same physical host server. Virtual SMP should therefore be used sparingly and only after determining whether or not an application is indeed multi-threaded. Running a virtual machine with a single virtual CPU, in many cases, will outperform the same virtual machine with Virtual SMP turned on. This determination should be treated on a case by case scenario, and you should test each virtual machine environment to see if Virtual SMP will help or not.
Make sure to configure your virtual machines with the correct HAL or kernel. Single processor virtual machines should be configured with a UP HAL or kernel while a multi-processor virtual machine should be configured with an SMP HAL or kernel. SMP versions are required in order to fully utilize multiprocessor systems but can also be used on single-processor systems. Because of their synchronization code, SMP operating systems used on single-processor systems will operate slower than a UP operating system.
Another way to affect virtual machine performance is by altering the virtual machine's minimum and maximum CPU resource allocation percentages. If you want to avoid CPU starvation for a virtual machine, set its minimum percentage to something other than zero. Conversely, to keep low priority virtual machines from consuming too many CPU cycles, you can set its maximum percentage to something lower, like 50%, thereby effectively throttling down that virtual machine and allowing other more pertinent virtual machines to make use of those valuable CPU cycles.
In addition to setting thresholds with MIN and MAX settings, you also have the ability to control which physical processor or processors that each virtual machine can use. This control is called processor affinity. The default setting is to use no affinity, and this is usually the best choice for most situations. You should really only set a virtual machine's CPU affinity when absolutely necessary. As an example, if you have a very resource intensive virtual machine running on a host server, you might want to set its CPU affinity to isolate that virtual machine and to protect its performance. Doing so will also protect the performance of all other virtual machines running on that same host server by changing each of their affinity settings to a different processor from that of the resource intensive virtual machine.
Virtual Machine Memory
If you are trying to optimize the performance of your virtual machine and find that the performance degradation isn't being caused by the processor, examine the virtual machine's use of memory. If the guest operating system is paging or swapping memory too much, performance will suffer since writing to disk is much slower than writing to memory. To identify if your virtual machine is paging, use vmstat from the command line on Linux or the Performance tool found under Administrative Tools on Windows to check the value for pages/second. If the number is high, such as 1000 pages/second, increase the amount of memory assigned to the virtual machine to eliminate excessive paging.
Only allocate as much memory to the virtual machine that is needed to allow enough memory to hold the working set of applications being asked to operate in the guest environment. Some amount of testing is needed here since giving a virtual machine too much memory will reach diminishing returns and simply be wasteful. The wasted memory could have been used as an additional resource elsewhere on the host server.
If possible, configure Linux virtual machines to use less than 896MB of memory. Linux uses different techniques to map memory in the kernel if the amount of physical memory is greater than 896MB. Every physical page of memory up to 896MB is mapped directly into the kernel space. This memory section is faster and more efficient and can keep your guests running optimally. Any amount of memory over that is no longer permanently mapped but is instead temporarily mapped. These techniques add additional overhead on the virtual machine monitor and can lower performance.
It usually depends on whether you are talking to the VMware ESX administrator or the virtual machine end--user as to what the "proper" configuration amount of memory is for each virtual machine. The amount of memory a virtual machine actually needs and the amount of memory the end-user wants normally doesn't match up. Finding a proper balance is the key to a successful infrastructure implementation.
Virtual Machine Networking
If you are trying to create a networking connection between virtual machines that live on the same host server, connect the virtual machines to the same vSwitch. While it isn't mandatory that these virtual machines be connected to the same vSwitch, doing so will keep the networking traffic from going out and across the wire. At the same time, it also keeps CPU and network overhead down and increases the performance of the network communications that take place between those virtual machines.
When you create a new virtual machine, the default network adapter that is emulated on a 32-bit guest is the AMD PCnet32 device and it is configured with VMware's vlance driver. For performance reasons, in a GigE environment, you should change the emulated network adapter to use either VMware's vmxnet driver or e1000. The vmxnet driver passes through network traffic from the virtual machine to the physical network adapter with minimal overhead. And the latest version of VMware ESX provides a new version of the vmxnet virtual device called Enhanced VMXNET. It includes several new networking I/O performance improvements such as support for TCP/IP Segmentation Offload (TSO) and Jumbo Frames. It also includes 64-bit guest operating system support. The vmxnet driver comes with the VMware Tools.
VMware ESX Server: Performance optimization
Configuration of the host server
Host server processor and memory performance
Host server storage and network performance
Configuration of the virtual machine
Configuration of the guest operating system
About the book
VMware ESX Essentials in the Virtual Data Center details best practices for ESX and ESXi, guides you through performance optimization processes for installation and operation, uses diagrams to illustrate the architecture and background of ESX and covers the two most popular releases, 3.0 and 3.5.
This was first published in November 2008