Vmware FT Testing

Fail over scenarios Vs Test Result:

Scenario – 1	Physical server failure(FT Configuration Test)
Purpose of Test	To check when Active physical server fails, the application should be run without interruption of any services. (Provides cost effective, automated OS instant migration  within second in the event of hardware or operating system failures)
Expected Response Time:	2 seconds
Actual RTO	1 second
Detailed Scope of Test	–       Test FT fail our –       Auto start Virtual machine from secondary host –       There is no need to change anything, migration will done auto.
Resources Required for Test	–       System and Network Administrator
What went well	–       As expected Virtual machine was started from the passive(secondary host) server successfully and actual response time was less than the expected response time
What could be improved
What failed (if any)

Step-01: Virtual machine (192.168.10.55) is up.

 

 

 

 

Step-2 – 02: Physical host details where server virtual machine is resided.

As showing in screen 02, server virtual machine is hosted on physical host (192.168.10.30)

Step -03: Start FT Test failover

Step-04: Virtual machine is going on.

Step -05: Migration completed successfully without disruption of services and server is resided on secondary physical server (192.168.10.25)

step-06: user session has not expired while server migration process.

vSphere HA agent on host cannot reach some of the management network addresses of other hosts and thus HA may not able to restart VMs if a host failure occurs

May be genuine connection error, so check this using:

vmkping

between the hosts

If this is all fine, it may be just a leftover from a previous issue, which is now resolved. The error may go if you restart the management services on the affected host using:

services.sh restart

Troubleshooting vSphere HA Admission Control

vCenter Server uses admission control to ensure that sufficient resources in a vSphere HA cluster are

Reserved for virtual machine recovery in the event of host failure. If vSphere HA admission control does not function properly, there is no assurance that all virtual machines in the cluster can be restarted after a host failure.

Red Cluster Due to Insufficient Fail over Resources

When you use the Host Failures Cluster Tolerates admission control policy, vSphere HA clusters might

Become invalid (red) due to insufficient fail over resources.

Problem

If you select the Host Failures Cluster Tolerates admission control policy and certain problems arise, the

cluster turns red.

 

Cause

This problem can arise when hosts in the cluster are disconnected, in maintenance mode, not responding, or

Have a vSphere HA error. Disconnected and maintenance mode hosts are typically caused by user action.

Unresponsive or error-possessing hosts usually result from a more serious problem, for example, hosts or

Agents have failed or a networking problem exists.

Another possible cause of this problem is if your cluster contains any virtual machines that have much

Larger memory or CPU reservations than the others. The Host Failures Cluster Tolerates admission control

Policy is based on the calculation on a slot size consisting of two components, the CPU and memory

Reservations of a virtual machine. If the calculation of this slot size is skewed by outlier virtual machines, the

Admission control policy can become too restrictive and result in a red cluster.

VMware, Inc.  37

Solution

Check that all hosts in the cluster are healthy, that is, connected, not in maintenance mode and free of

vSphere HA errors. vSphere HA admission control only considers resources from healthy hosts.

Unable to Power On Virtual Machine Due to Insufficient Failover Resources

You might get a not enough failover resources fault when trying to power on a virtual machine in a

vSphere HA cluster.

 

Problem

If you select the Host Failures Cluster Tolerates admission control policy and certain problems arise, you

Might be prevented from powering on a virtual machine due to insufficient resources.

Cause

This problem can have several causes.

n Hosts in the cluster are disconnected, in maintenance mode, not responding, or have a vSphere HA

Error.

Disconnected and maintenance mode hosts are typically caused by user action. Unresponsive or error possessing hosts usually result from a more serious problem, for example, hosts or agents have failed or

a networking problem exists).

n Cluster contains virtual machines that have much larger memory or CPU reservations than the others.

The Host Failures Cluster Tolerates admission control policy is based on the calculation on a slot size

Comprised of two components, the CPU and memory reservations of a virtual machine. If the

Calculation of this slot size is skewed by outlier virtual machines, the admission control policy can

Become too restrictive and result in the inability to power on virtual machines.

No free slots in the cluster.

Problems occur if there are no free slots in the cluster or if powering on a virtual machine causes the slot

size to increase because it has a larger reservation than existing virtual machines. In either case, you

Should use the vSphere HA advanced options to reduce the slot size, use a different admission control

Policy, or modify the policy to tolerate fewer host failures.

Solution

View the Advanced Runtime Infopane that appears in the vSphere HA section of the cluster’s Monitortab

in the vSphere Web Client. This information pane shows the slot size and how many available slots there are

in the cluster. If the slot size appears too high, click on the Resource Allocationtab of the cluster and sort

the virtual machines by reservation to determine which have the largest CPU and memory reservations. If

there are outlier virtual machines with much higher reservations than the others, consider using a different

vSphere HA admission control policy (such as the Percentage of Cluster Resources Reserved admission

control policy) or use the vSphere HA advanced options to place an absolute cap on the slot size. Both

Error “The VMware VirtualCenter Server Service on Local Computer started then stopped”

One Silly issues has spoil 2 hours for solution….coincidentally we had installed PRTG Monitoring tool and V Center in same system.

Issue

VirtualCenter Server service fails starting on vCenter Server 5 with an error: The VMware VirtualCenter Server Service on Local Computer started then stopped . Some services stop automatically if they have no work to do, for example the Performance logs and Alerts service.

Solution

There is a conflict with TCP/IP port 8085 that is used by PRTG Network Monitor software and also by VirtualCenter Server.

Please refer to the following VMware KB article for a list of TCP and UDP ports required to access vCenter Server:

Click here to access the “VMware KB article” at: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012382.

 

VMware vSphere FT Configuration

1. Network configuration.

2. Turning on FT.

3. 54% through setting up FT.

4. FT setup complete.

5. vLockstep Interval and Log Bandwidth information are now updated.

6. Primary VM is located on 10.10.10.146.

7. Secondary VM is running on the Secondary Host- 10.10.10.145.

8. Testing FT using the built in Test Failover command.

9. Failover test completes. Notice that 10.10.10.146 has become the secondary location.

10. After the Failover Test, the primary host server is now 10.10.10.145 and the primary VM is running on it.

11. Furthermore, after the Failover Test, the secondary VM runs on the secondary host server, 10.10.10.146.

12. Failover test completion will result in showing the primary VM on the new primary host server and the new secondary host server with the accompanying vLockstep Interval and Log Bandwidth.

13. Finally, VMware vSphere FT is ready to provide continuous protection to your VMs

Host CPU Spikes at 100 Percent when install new VM

You experience high CPU usage in the guest operating system. However, when you examine Task Manager, no CPU usage issues are displayed in the host operating system.

There are instances where performance problems or symptoms may arise, but the cause may be due to the VM environment/configuration. The information and screenshots provided below are available to help determine if the performance problem may exist due to virtual instance instead of at the Traveler level.

Please check the following areas to determine if the VM is the cause of the performance related issue.

Check for VM Alarms

Click the Alarms tab to determine if there are any alerts available.

This example shows a high CPU Alarm condition for the timeframe being investigated. Engage your VMWare team as soon as possible to investigate errors such as these.

Check the CPU of the Traveler server by following these steps:
Click the Performance tab.
Click the “Advanced” button

Click the “Chart Options…” link

For CPU, choose “Past week” or the appropriate time frame for investigation.
Under the Counters section, check the boxes for “Usage” and “Ready”

Click Apply / OK

Save the chart (screenshot or click the save icon in the top right)
Notice that one of these (the Usage one) is in percentage of CPU. The key is to look for critical thresholds and peak times. Look for patterns.

Usage = CPU Usage as a percentage during the interval (during the amount of time that was selected)
Ready = Percentage of time that the virtual machine was ready, but could not get scheduled to run on the physical CPU.

A short spike in CPU usage or CPU ready indicates that the system is making the best use of the host resources. However, if both values are constantly high, the hosts are probably overcommitted. Generally, if the CPU usage value for a virtual machine is above 90% and the CPU ready value is above 20%, performance is impacted.

It can be many reason for 100 CPU spike by the host, one of the hardware compatibility issue can be also for 100 CPU spike.

Poor performance when virtual machines reside on local storage on VMware ESXi 5.0 Affected configurations The system may be any of the following IBM servers: BladeCenter HS23, type 7875, any model The system is configured with at least one of the following: UpdateXpress Service Pack Installer, any version VMware vSphere Hypervisor 5.0 with IBM Customization Installable, Base Install This tip is not option specific. The mpt2sas device driver for the VMware ESXi 5.0 is affected. VMware ESXi 5.0 Solution Issue the following esxcli commands in the ESXi Shell to remove ilfu and LSIProvider VIBs. esxcli software vib remove -n ilfu esxcli software vib remove -n LSIProvider

Virtual PC Performance Checklist

• Make sure your Host Operating System’s disk is defragmented.

  This includes the System Disk (the disk your OS boots off of) as well as the Disk that holds your Virtual Hard Disk File.
• Run Fewer Applications.
I’m continually amazed when folks complain about VM performance and when I get to their desk I see that they are running Outlook. That 200+megs could be better used by the system. Are you running a VM or checking your email? Consider checking your email on a schedule,  or using Outlook Web Access while you work on your VM.

• Enable Hardware Assisted Virtualization

If you’ve got this on your computer, turn it on. There IS some concern about really sophisticated Trojans that can use this technology for evil, but for me, it’s all good as it speeds most Guest Operating Systems (especially non-Microsoft ones) up quite a bit.
• Give your Virtual Machines LESS MEMORY
o I’ve found that 512 megs is just about the Ideal Amount of memory for 90% of your Virtual Machines. Don’t bother trying to give them 1024 megs, it’s just not worth the pressure it’ll put on the Host Operating System.

• Considering making a custom Windows install for your VMs.

Rather than going to all the effort to REMOVE things, why not create a Windows installation that can be shared across your organization that doesn’t include the crap ahead of time. There’s a Windows Installation Customizer called nLite that lets you prepare Windows installations so they never include the stuff you don’t want. Makes it easier if Solitaire is never installed

• Make sure the Guest Operating System is defragmented.

Disk Defragmenter that runs in that “Text Mode” place before Windows really starts up. This allows it to get at files that don’t always get defragmented.

• Don’t use NTFS Compression on the Virtual Machine Hard Drive File in the Host Operating System

NTFS Compression doesn’t work on files larger than 4 gigs, and can cause corruption.

• Don’t Remote Desktop or VNC into Host Operating Systems that are hosting Virtual Machines.

If you’re remoting into a machine where THAT machine is running a VM, note that to the Remote Desktop protocol (and VNC) the VM just looks like a big square bitmap that is constantly changing. That guarantees you slow performance. If you can, instead, Remote Desktop into the Virtual Machine itself.

• Make sure you’ve install the Virtual Machine Additions (or Tools, or Utilities, or Whatever)

Virtual PC and VMWare and Parallels all include drivers and tools that improve the performance of your Virtual Machine. They are there for good reason, make sure you’ve installed them.
 Also, if you’re running a Virtual Machine created under and older version, like Virtual PC 2004, and you’re now running under a newer one, like 2007, pay attention to the upgrade warnings and install the latest drivers and Virtual Machine Additions.

Virtual PC Performance Checklist

which VM backup should be more reliable ?

Type of VM backup through Symantec

agent-based backup is also known as guest based backup. An agent is installed in every virtual machine and treats each virtual machine as if it was a physical server. The agent in this scenario is reading data from disk and streaming the data to the backup server

Advantage-

• Both physical and virtual machines are protected using the same method
• Application owners can manage backups and restores to guest OS
• Time tested and proven solution
• Meets their recovery needs
• This is the only way to protect VMware Fault Tolerant virtual machines and VMs with Physical Raw Disk Mappings RDMS

Disadvantage-

• Significantly higher CPU, memory, I/O and network resources utilization on virtual host machines when backups run.
• Need to install and manage agents on each virtual machine
• Cost may be high for solutions that license on a per agent basis as opposed to per hypervisor based licensing
• lack of visibility into changing virtual infrastructure
• No visibility for backups from VM administrators’ point of view; for example, backups are not visible at vSphere client level
• Complex disaster recovery strategies
• Lack of SAN transport backups to offload backup processing job from virtual infrastructure
• No protection for offline virtual machines and virtual machine templates
• Slow file by file backup by agent sending the even unchanged data over and over again

 

Agentless backup-

Agentless backup, also known as host-based backup, refers to solutions that do not require an agent to be installed on each VM by the administrator. However, it’s important to note that the software may be injecting an agent onto the guest machines without your knowledge.

Advantage-

• VMs can be backed up online or offline
• Less CPU, memory, I/O and network impact on the virtual host
• An agentless architecture doesn’t require the management of agent software
• No per VM agent licensing fees

• Extremely difficult to recover granular object data – first restore the entire VM and its virtual disks
• Traditional login techniques to log into the server
• Temporary “injected” drivers can destabilize the system and compromise data integrity
• Troubleshooting is more complex when using injected (temporary) agents
• A centralized controller is a single-point-of-failure
• Requires a fully-virtualized environment. Physical machines still require agent-based backup. If you have physical and virtual you will need two backup solutions – one for physical and the other for virtual.

 

Agent-Assisted Backup-

Agent assisted backups are also known as host based backup and integrate with VMware’s VADP and Microsoft VSS to provide fast and efficient online and offline backups of ESX, vSphere and Hyper-V. The primary difference between agentless and this design is its perspective: it pairs the VMware VADP or Microsoft VSS with an agent that gathers application metadata to enable multiple avenues of recovery (full VM, applications, databases, files, folders and granular objects).

 

• The backup is for the entire virtual machine. This is important because it means the entire VM can be recovered from the image.  It also means that products like Backup Exec & NetBackup can offer “any-level” of recovery from the image contents: Files / Folders, Databases and Granular database contents, like email and documents.
• The backup can be offloaded from both VM as well as the hypervisor. This means that Backup Exec & NetBackup have the flexibility to offload VM backup onto an existing backup server, instead of burdening the hypervisor.  It also means that users have the option of deploying a dedicated VM, e.g. a virtual appliance, when a physical backup server is not practical.
• Application owner can self-serve restore requests: The application owner can request restores directly back to the application.
• Enhanced security: The agent installed for assisting with VM backup can be managed by the application owner. Thus you are avoiding the need to share guess OS credentials with backup administrator.

 

Best Way to store Backup-

1. Backup VM directly from storage location for example, SAN, iSCSI, NAS, without having to install any software agent inside the VMs

2. Centralized backups for Virtual machines
3. Keep all schedule in simple way for easily understand, thing wrest scenario and make backup plan
4. Make it documented for every backup policy.

 

 

Virtaul machine file structure

Once you understand virtual machines (VMs) from a hardware perspective, you can study the components that make up a VM on an ESX/ESXi host. These are the various VMware file types associated with a VM, located in the VM’s directory on the host (represented in the illustration below).

 

tools options, and power management options. While you can edit this file directly to make changes to a VM’s configuration, don’t do this unless you know what you are doing. If you do make changes directly to this file, make a backup copy first.

VMDK files. All virtual disks are made up of two files, a large data file equal to the size of the virtual disk and a small text disk descriptor file, which describes the size and geometry of the virtual disk file. The descriptor file also contains a pointer to the large data file as well as information on the virtual disks drive sectors, heads, cylinders and disk adapter type. In most cases these files will have the same name as the data file that it is associated with (i.e., myvm_1.vmdk and myvm_1-flat.vmdk). You can match the descriptor file to the data file by checking the Extent Description field in this file to see which -flat, -rdm or -delta file is linked to it. 

The different types of virtual disk data files that can be used with VMware virtual machines are:

The -flat.vmdk file
This is the default large virtual disk data file that is created when you add a virtual hard drive to your VM that is not an RDM. When using thick disks, this file will be approximately the same size as what you specify when you create your virtual hard drive. One of these files is created for each virtual hard drive that a VM has configured, as shown in the examples below.

The -flat.vmdk file
This is the default large virtual disk data file that is created when you add a virtual hard drive to your VM that is not an RDM. When using thick disks, this file will be

• The -delta.vmdk file
  These virtual disk data files are only used when making snapshots. When a snapshot is created, all writes to the original -flat.vmdk are halted and it becomes read-only; changes to the virtual disk are then written to these -delta files instead. The initial size of these files is 16 MB and they are grown as needed in 16 MB increments as changes are made to the VM’s virtual hard disk. Because these files are a bitmap of the changes made to a virtual disk, a single -delta.vmdk file cannot exceed the size of the original -flat.vmdk file. A delta file will be created for each snapshot that you create for a VM and their file names will be incremented numerically (i.e., myvm-000001-delta.vmdk, myvm-000002-delta.vmdk). When the snapshot is deleted, these files are automatically deleted after they are merged back into the original flat.vmdk file.

  

   

• The -rdm.vmdk file
  This is the mapping file for the raw device mapping (RDM) format that manages mapping data for the RDM device. The mapping file is presented to the ESX host as an ordinary disk file, available for the usual file system operations. However, to the VM, the storage virtualization layer presents the mapped device as a virtual SCSI device. The metadata in the mapping file includes the location of the mapped device (name resolution) and the locking state of the mapped device. If you do a directory listing, you will see that these files will appear to take up the same amount of disk space on the VMFS volume as the actual size of the LUN that it is mapped to, but in reality they just appear that way and their size is very small. One of these files is created for each RDM that is created on a VM.

  

The .vswp file. When you power on a VM, a memory swap file is created that can be used in lieu of physical host memory if an ESX host exhausts all of its physical memory because it is overcommitted. These files are created equal in size to the amount of memory assigned to a VM, minus any memory reservations (default is 0) that a VM may have set on it (i.e., a 4 GB VM with a 1 GB reservation will have a 3 GB VSWP file created). These files are always created for virtual machines but only used if a host exhausts all of its physical memory. As virtual machine memory that is read/written to disk is not as fast as physical host RAM, your VMs will have degraded performance if they do start using this file. These files can take up quite a large amount of disk space on your VMFS volumes, so ensure that you have adequate space available for them, as a VM will not power on if there is not enough room to create this file. These files are deleted when a VM is powered off or suspended.

Virtual machines will lock the .vswp, -flat.vmdk and -delta.vmdk, .vmx and .log files during runtime.

The .vmss file. This file is used when virtual machines are suspended and is used to preserve the memory contents of the VM so it can start up again where it left off. This file will be approximately the same size as the amount of RAM that is assigned to a VM (even empty memory contents are written). When a VM is brought out of a suspend state, the contents of this file are written back into the physical memory of a host server, however the file is not automatically deleted until a VM is powered off (an OS reboot won’t work). If a previous suspend file exists when a VM is suspended again, this file is re-used instead of deleted and re-created. If this file is deleted while the VM is suspended, then the VM will start normally and not from a suspended state.

The .vmsd file. This file is used with snapshots to store metadata and other information about each snapshot that is active on a VM. This text file is initially 0 bytes in size until a snapshot is created. A VMSD file updates with information every time snapshots are created or deleted. Only one of these files exists regardless of the number of snapshots running, as they all update this single file. The snapshot information in a VMSD file consists of the name of the VMDK and VMSN file used by each snapshot, the display name and description, and the UID of the snapshot. Once your snapshots are all deleted, this file retains old snapshot information but increments the snapshot UID to be used with new snapshots. It also renames the first snapshot to “Consolidate Helper,” presumably to be used with consolidated backups.

The .vmsn file. This file is used with snapshots to store the state of a virtual machine when a snapshot is taken. A separate .vmsn file is created for every snapshot that is created on a VM and is automatically deleted when the snapshot is deleted. The size of this file will vary based on whether or not you choose to include the VM’s memory state with your snapshot. If you do choose to store the memory state, this file will be slightly larger than the amount of RAM that has been assigned to the VM, as the entire memory contents, including empty memory, is copied to this file. If you do not choose to store the memory