VMware server: How to troubleshoot VMware ESX Server installations

A VMware server installation rarely goes without issue. This book excerpt by Chris Wolf explains how to troubleshoot VMware ESX Server installations.

IT reseller takeaway: VMware server installations aren't typically clean and simple tasks. Problems may arise. This book excerpt from Virtualization: From the Desktop to the Enterprise. will help value-added resellers (VARs), systems integrators and consultants know how to troubleshoot VMware ESX server installations.

Sometimes things don't go as planned during the ESX Server installation. If something goes awry, the System Configuration Wizard will present itself after logging into the MUI. The wizard generally launches for one of three reasons: the ESX Server has been installed without being configured, ESX Server has been upgraded from a previous version, or the VMkernel failed to load. If ESX Server proclaims, "Your system has not yet been configured, or the existing configuration is incomplete; the wizard will guide you through the system configuration process," hold onto your keyboard: ESX Server is throwing you a curveball. If you run into problems with a new installation, repeat the install to see if you can duplicate the configuration process. You don't want any anomalies related to software or hardware sneaking into production.

More virtualization resources
Server virtualization support

After connecting to your ESX Server through a Web browser, select Yes to the SSL security alert. If you want, you can view it and install the certificate. You'll next be presented with the VMware Management Interface login screen. For now, enter root as your username and the appropriate password. Check your Caps Lock key, and remember that everything is case sensitive. Once you've logged into ESX Server's MUI, you'll need to complete several post-install tasks if something went wrong during installation, which may include the following:

  • Accepting the license agreement
  • Configuring the startup profile
  • Setting up the storage configuration
  • Configuring the swap configuration
  • Setting up network connections and switches
  • Adjusting security settings

If your install fails to provide the initial configuration pop-up—it's a good thing! Keep reading, though, because we'll cover a few things here that aren't covered if you have an uneventful installation of ESX Server. If something went awry during the installation, the ESX Server Configuration screen will open in a new window.

How to accept VMware's license agreement

The first thing you must do is accept VMware's license agreement on the End User License Agreement screen. Be sure to check the box signifying your compliance and enter your serial numbers. If you didn't purchase Virtual SMP, leave the relative fields empty. If you don't have the proper licensing information, you won't be allowed to proceed with the server configuration. You can get a demo license by registering with the VMware Web site.

View your license rights, and confirm the correctness of the license expiration, number of supported host machines, number of supported host processors, and number of processors supported for each virtual machine. If all is okay, click Next to continue.

Startup profile

Before being presented with the Startup Profile screen, there will be a brief pause while the startup profile is loaded. You'll need to specify how much memory is to be reserved for the system console. This number will be based on how many VMs you intend to host on your server. Being that RAM is cheap and you want to err on the side of performance, under System Startup, pick a setting one greater than you need. For instance, if you plan to run eight virtual machines, use the settings for sixteen virtual machines.

Under Hardware Profile, you now need to assign the network adapters to the Service Console or to VMs. If you have only one NIC, you can assign it for VMs to use, but you'll lose the console's network connection. For ease of administration, you should install two NICs. If not, you'll be forced to configure everything from the CLI or be left to hack together a locally running copy of X Windows. Though running X Windows locally is possible, it consumes valuable resources and isn't officially supported; moreover, it's a good way to cause your VMs to quit functioning. After dedicating memory and NICs as necessary, click Next to continue your installation.

How to configure storage for VMware ESX Server

ESX Server may take a moment to save your startup profile and reboot. On reboot, log back into the MUI. You may receive an error message stating that the system isn't configured or it has an incomplete configuration. You have a choice to view the system logs, so take a moment to explore them by selecting the System Logs hyperlink. Cycle through the four tabs, and make note of any pertinent information. The logs can get quite long and be cryptic; if you're not a Linux guru, be prepared to break out your Linux books to decipher the contents. Whether or not you're required to troubleshoot the install process, you'll eventually see the Storage Management Disk and LUN configuration screen, assuming all the necessary partitions were created during the install. You can take the opportunity to repartition the system, or you can click Next to continue. If you have to create partitions, you'll need to know that guest VMs can be stored only on VMFS partitions or physical disks. You must have a least three partitions: /boot, swap, and /. The partitions created during the install that contain the Linux file system and the Service Console swap partition can't be changed.

How to configure swap files

On the Swap Configuration screen, select Create to make a memory swap file. You'll be presented with another swap file configuration screen. Generally, ESX Server does a good job of determining the filename, size, and activation policy. If you'd like to change the storage volume or filename, now is a good time to do it. Make sure the swap file is a minimum of 256 MB and no larger than twice the server's physical memory. If desired, you can direct ESX Server to allow you to manually start the swap file. If you choose this as the activation policy, you'll have to start it manually by entering vmkfstools –w at the CLI. Take a moment to activate your swap file now. If you do, you won't have to reboot.

How to configure virtual machine networks

After completing the Swap Configuration screens, you'll be presented with the Network Connections/Virtual Switches dialog boxes. If necessary, you'll need to configure network adapters that guest VMs will use. Before setting your speed and duplex, know how your switch is configured. If you decide to team network adapters, know the team bonding type of your connecting switch. As a rule of thumb, avoid autonegotiation and reconfigure your switch if necessary. You can experience high latency intervals while the NIC and switch attempt to hash out their respective settings.

If you have one NIC, you'll need to configure the adapter after completing the Configuration Wizard. You'll need to perform the following steps:

1. Use Vi to edit /etc/module.conf. Comment out the alias eth0 line by placing a pound, hash, square, gate, or number sign at the beginning of the line: # alias eth0 e100

2. Connect to ESX Server from a Web browser, http://<hostname>. Select Edit, and under Device Allocation, reassign the NIC to Virtual Machines. Reboot.

3. Log in as root at the console. Use Vi to edit /etc/rc.d/rc.local, and append the NIC's name. If you're unsure of your NIC's name, you can use the findnic command as an aid. insmod vmxnet_console devName="vmnic0" ifup eth0

4. Reboot, and the single NIC sharing is complete. You'll now have to perform all administration of ESX Server at the console.

Binding NICs together to form a bond is the same as teaming NICs. When creating bonds, be sure to use functionally similar physical NICs because ESX Server will support only what they have in common, such as VLAN tagging, TCP segmentation offloading, and checksum calculations. You're limited to bonding a total of ten physical adapters for any given VM. You may also be required to create and configure a virtual switch. You'll need to supply a name for the switch in the Network Label fields. Next, look at Outbound Adapters, and assign an available NIC from the list to the switch. Then click Create Switch.

Similar to a physical switch, a virtual switch has 32 ports; it can be used to load balance and team NICs and provide redundant links. You can connect one VM to each port. Virtual switches have labels and can be renamed only when no VMs are using them, so take precautions when doing this (double-check everything). If you rename a switch with a programmatically attached VM, it won't boot. If you're creating a network of host-only VMs, deselect all NICs from the configuration of the switch. In addition, you can create port groups, which are similar virtual local area networks (VLANs). VLANs logically group switch ports into a single network and allow for secure communications amongst the hosts. Before creating a port group, you must have an existing network created. You can have IDs between 1 and 4095. When you're done configuring the switch and NIC, click Next to continue.

How to secure VMware ESX Server

The ESX Server installation process will now present you with the Security Settings screen. By default, the system selects high security for you. If it's necessary to make changes, choose Custom to enable the minimum number of services you need. Though this should go without being written, the more you open up ESX Server, the more vulnerable it is to attacks. When you're done selecting your security settings, click Next to continue.

After completing the ESX Server Configuration screen, you should be presented with the VMware Management Interface Login screen. Should your system continue to fail the configuration process, double-check your hardware configuration, and make sure it meets the minimum requirements for ESX Server.

If you're still having problems, make sure you have at least one NIC and one SCSI hard drive available for guest VMs: both are minimum requirements. You'll also notice that the host name isn't properly configured. After triple-checking your system against the minimum requirements and VMware's HCL for ESX Server, give the installation another shot. You'll find that it really is seamless, and you can move onto what you should expect to see from a successful installation.

Caution: Installing X Windows on ESX Server for educational or test purposes is okay. Don't install X Windows on a production ESX Server system—the overhead will ravage resources and render a production server useless. Given the aforementioned warning, you can install X by installing it as well as its dependencies by using RPMs from Red Hat 7.3.


About the author   
Chris Wolf is an instructor at ECPI Technical College, as well as a leading industry consultant in enterprise storage, virtualization solutions, and network infrastructure management. He has a master's degree in information technology from Rochester Institute of Technology, and his IT certification list includes MCSE, MCT, and CCNA. Wolf authored MCSE Supporting and Maintaining NT Server 4.0 Exam Cram, Windows 2000 Enterprise Storage Solutions and Troubleshooting Microsoft Technologies, and he contributes frequently to Redmond Magazine and Windows IT Pro Magazine. Wolf also speaks at computer conferences across the nation.

Dig Deeper on Server virtualization technology and services

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.