The VMware High Availability (HA) feature was introduced in VMware VI3 to provide protection for virtual machines (VMs) in the event of a host failure. With VMware HA enabled on a cluster,
if a host failure occurs, VMs are automatically restarted on other hosts in the cluster. This was a much-needed feature because a host failure can take down a large number of VMs that are running on it. VMware HA helps reduce the risk that virtualization introduces of placing all your eggs in one basket. In this VMware High Availability guide, learn how VMware HA works, VMware HA requirements, how to enable Admission Control Policies and other VMware HA implementation tips.
How does VMware High Availability work?
VMware HA works by placing an agent on each host in a cluster that maintains a "heartbeat" with the other hosts in the cluster. The loss of a heartbeat initiates the process of restarting all affected VMs on other hosts. The vCenter Server doesn't provide a single point of failure as HA will continue to work even if the vCenter Server is unavailable. VMware HA relies on what are called "primary" and "secondary" hosts. The first five hosts powered on in an HA cluster are designated as primary hosts, and the remaining hosts in the cluster are all considered secondary hosts. The function of the primary hosts is to replicate and maintain the state of the cluster, and to initiate failover actions. If a primary hosts fails, a new primary is chosen at random from the secondary hosts.
HA uses a failure detection interval that is set by default to 15 seconds; a host failure is detected after the HA service on a host has stopped sending heartbeats to the other hosts in the cluster. The heartbeat check occurs every second; if a host does not respond after 15 seconds, it is considered failed. Fifteen seconds is actually considered pretty aggressive, anything less and you can experience false alarms from any brief loss of network connectivity that occurs. A host stops sending heartbeats if it is either isolated from the network, if the host crashes, or is completely down due to a hardware failure. Once a failure is detected, other hosts in the cluster treat this host as failed, while this host declares itself as isolated from the network.
What happens when a host failure occurs
By default, the isolated host leaves its VMs powered on but the isolation response for each VM is configurable on a per-VM basis. These VMs can then successfully fail over to other hosts in the cluster. HA also has a restart priority that can be set for each VM so that certain VMs are started before others. This priority can be set to either low, medium or high, and also can be disabled so VMs aren't automatically restarted on other hosts. Here's what happens when a host failure occurs:
1. One of the primary hosts is selected to coordinate the failover actions, and one of the remaining hosts with spare capacity becomes the failover target.
2. VMs affected by the failure are sorted by priority, and powered on until the failover target runs out of spare capacity in which case another host with sufficient capacity is chosen for the remaining VMs.
3. If the host selected as coordinator fails, another primary continues the effort.
4. If one of the hosts that fail was a primary node, one of the remaining secondary nodes is promoted to being a primary.
HA works best if used in conjunction with the Distributed Resource Scheduler (DRS) feature. Hosts that absorb the VMs from a failed host can become overloaded and DRS can help balance the VMs around to other hosts in the cluster.
VMware HA and Admission Control Policies
While HA may seem like a simple enough feature on the surface, under the covers it is actually rather complicated. The placement of VMs from a host that fails over to other hosts requires that enough free capacity be available to handle all of the workloads that exist on a failed host. VMware HA can be configured to handle multiple host failures so determining if enough free capacity exists in a cluster can become complicated. HA uses Admission Control Policies to ensure sufficient resources exist in a cluster to handle the VMs from a failed host(s). When Admission Control policies are enabled (see Figure 1 below), VMs aren't allowed to be powered on if they violate availability constraints.
Figure 1: Enabling Admission Control policies
There are three types of Admission Control policies that use different methods for reserving resources to handle host failures.
- Host Failures Cluster Tolerates: This is the total number of host failures that the cluster can handle with the reserved resource capacity. This number can be set from one to four; it can't exceed four because there are only five primary hosts that monitor HA. Whatever number is selected, HA will reserve sufficient CPU and memory resources by using a mechanism called “slots.” Slots are simply a unit of measurement of CPU and memory resources that is used to reserve resources for HA. Slot sizes by default are 0 MB for memory and 256 Mhz for CPU. If VM CPU/memory reservations exist, then slot sizes become whatever the highest value is for any CPU and memory reservations. Once a slot size is determined, it is used to ensure that sufficient CPU and memory resources exist throughout the cluster to handle the amount of host failures that is set. The more host failures that you tolerate will mean more resources are set aside for reserve capacity and cannot be used by VMs. You can view slot information by selecting a cluster and on the Summary tab, and click the "Advanced Runtime Info" on the VMware HA section (see Figure 2 below).
Figure 2: Selecting Advanced Runtime Info
- Percentage of Cluster Resources: This method doesn't rely on slots and instead uses
calculations to ensure that a percentage of the clusters resources are reserved for failover. It
does this by calculating the total resource requirements for all powered-on virtual machines in the
cluster. Next, it calculates the total host resources available for VM. Finally it calculates the
Current CPU Failover Capacity and Current Memory Failover Capacity for the cluster, and if it's
less than the percentage specified for the Configured Failover Capacity, then admission control
will be enforced. This method is a bit more balanced than specifying host failures but is not as
automated because you have to manually specify a percentage.
- Specify a failover host: With this method, The software will simply restart a failed VM onto a single specific host. If the specified host has failed, or doesn't have enough resources, then VMware HA will restart the VMs on another host in the cluster. You can only specify one failover host and HA will prevent VMs from being powered on or moved to the failover host during normal operations to ensure there is sufficient capacity on it.
VMware HA capacity and other important tips
Because VMware HA requires spare capacity to handle VMs from failed hosts, you should take this into consideration when designing your virtual environment. The nature of virtualization is to make the most of a host’s resources, but you should ensure that you size your hosts to have sufficient capacity to utilize the HA feature. While you can disable the HA admission control, it's not recommended as you may overwhelm your environment if a host fails and there are not enough resources available to handle the increase of VMs on other hosts.
The 15-second failure detection interval that HA uses as a trigger can be a bit aggressive, and you may want to increase this interval to prevent false alarms from occurring from a brief loss of network connectivity. The HA heartbeat is sent over the management interface on ESX (Service Console)/ESXi (VMkernel) hosts, so it is best to have NIC redundancy in the vSwitch that the management interface resides on. Alternatively, you can configure a second management interface on a second vSwitch to serve as a backup. You can also set an advanced setting to add a secondary isolation IP address as a safety net to ensure that a host is truly isolated. HA will attempt to ping the isolation address to ensure that it is truly isolated before triggering.
Another important setting determines the host isolation response for VMs that are powered on when an HA failure occurs. If a major failure like the loss of power were to occur, this setting does not come into play. But if a host remains operational and simply loses connectivity to the network, this setting determines what to do with the VMs that are still running on the host. The default setting is to shutdown VMs on an isolated host so they can be brought up on other hosts. This initiates a shutdown through the operating system so VMs are shutdown gracefully. The setting can also be set to power off VMs, which results in a quicker shutdown but risks VM data corruption.
The final setting is to leave the VMs running on the host, which can help if you experience false alarms but can complicate starting VMs on other hosts. If a host experiences a network failure, it may still be connected to its storage, especially if it’s on a Fibre Channel network. If a VM remains running on a failed host that is still connected to its storage, it will still maintain a lock on its disk file so it will be unable to be started on another host. Therefore, it's preferable in most cases to shutdown VMs on a failed host so they can be cleanly started on other hosts.
There is more to HA than meets the eye and a whole book has been written on this seemingly simple feature. There are also many advanced HA settings that can be set to customize the way HA functions for you, but you should know how to use them before implementing them. If you intend to use HA, make sure you understand the mechanics of how VMware HA works so you can more effectively use it in your VMware environment.
About this author: Eric Siebert is an IT industry veteran with more than 25 years experience covering many different areas but focusing on server administration and virtualization. He is a very active member in the VMware Vmtn support forums and has obtained the elite Guru status by helping others with their own problems and challenges. He is also a Vmtn user moderator and maintains his own VMware VI3 information website, vSphere-land. In addition, he is a regular blogger and feature article contributor on TechTarget's SearchServerVirtualization and SearchVMware websites.
This was first published in June 2011