Getting your fair share of AWS EC2, VMware, Hyper-V, etc.

One of the problems with virtualised infrastructure -- especially cloud servers -- is that your programs don't always get the CPU time that they need. The CPU time for your application is shared between CPU time given to other virtual machines on the same hardware. This is good for cost savings, but it would be nice to know how badly you are being affected by this.

I had a customer who had some really interesting VMware scheduling problems. They had a large number of multi-cpu virtual machines. VMware can't schedule a 4-cpu virtual machine unless there are 4 physical CPUs free. So even if you only have one tiny job to run on one CPU, VMware can't just schedule that one CPU -- it has to wait until at least 4 are available. (Incidentally, those other 3 CPUs that are scheduled but do nothing count towards co-stop% which is another interesting performance-and-tuning metric.)

As it turns out, almost all their virtual machines had some small tasks going on in the background (e.g. cluster heartbeats), so VMware tried very hard to schedule them all as best as it could, but the result wasn't pretty. How bad was it?

I wrote a little program that slept for a second, woke up, recorded the time, and then went back to sleep again. It kept statistics about how delayed it was. There were some horror stories --  there were virtual machines that received no CPU time for more than 90 seconds on occasions! No wonder their clusters kept crashing -- the cluster heartbeat time was only 30 seconds, so there was no way the cluster could stay up with VMware starving it of CPU time like that.

Anyway, I tidied up that program ("am-i-scheduled") and packaged it for RHEL7; the binary is so simple that it will also run on Ubuntu unchanged. I suspect it will run almost anywhere.

If you run AWS EC2 servers or other cloud-hosted servers, you really want to install this. This is the most convenient way you can find out how much CPU time you aren't getting: if your virtual machine is co-located with another virtual machine that is being used for Bitcoin mining, or password cracking, or a deep learning problem, you might want to terminate and try again. With am-i-scheduled, you can detect this easily, and measure the impact you are experiencing.

Here's the source: https://bitbucket.org/solresol/am-i-scheduled and the binaries (including RPMs) can be downloaded from here: https://bitbucket.org/solresol/am-i-scheduled/downloads/

GRE for Linux requirements

If you are using Data Protector to backup your VMware environment, and you have Linux boxes, you might have tried to use the Granular Recovery Extension (VMware GRE). The GRE lets you recover individual files from a VM-level or VMDK-level backup; it does this by mounting the VMDK file on a Linux GRE proxy.

There are three variations of GRE restore:

• If you backup to a StoreOnce device (e.g. a B6200, a D2D4500, a StoreOnce virtual appliance, or a software storeonce component on a Windows or Linux member of the cell)... then you need a very large disk and not much else.
• If you backup using 3PAR snapshots (which works very well indeed) and you are doing a GRE recovery from a snapshot, you don't need a big disk or anything else much at all -- just something connected to the 3PAR the meets the usual (documented) list of requirements.
• The very weirid case is if you use a SmartCache device. These are uncompressed, raw disk spaces for putting VMware backups on to. The SmartCache is accessed by the Windows and Linux proxies via Windows file sharing. Thus the Linux GRE proxy server needs to have Samba installed on it.

One remaining issue, that I hope gets fixed one day soon: the machine that you want to restore to has to have a world-writeable NFS share. Ouch: given that it is copying from one Linux box to another Linux box, I'm not quite sure why this couldn't have been done with SFTP. So I suspect everyone will just have a multi-stage restore for GRE on Linux boxes:

• Load the backup from StoreOnce onto a large disk.
• Restore files from that to a server with an insecure NFS share.
• Copy from the NFS share via SSH to the actual server where you needed the file restored.

I presume also that setuid / setgid binaries are therefore not supported for GRE restore. (Because who creates a world-writeable NFS share without the nosetuid,nosetgid options enabled?) Implication: you can't restore /usr or /sbin from a VMware backup reliably.

Of course, you might find it cheaper to use VMX (https://store.data-protector.net/collections/vmx-licenses) instead of Data Protector -- it is much cheaper than the equivalent GRE and VEAgent licenses. As far as I know, it doesn't face these limitations.

Greg Baker is an independent consultant who happens to do a lot of work on HPE DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HPE and HPE partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector, or visit the online store for Data Protector products, licenses and renewals at http://store.data-protector.net/ 

VMware 3PAR integrated backups

HPE recently added a feature to Data Protector which will appeal to customers running VMware and using 3PAR storage. You can now backup your VMware environment by taking 3PAR snapshots and then backing up the snapshot. This reduces load on your VMware environment. You can also leave the snapshot around in case you want to do a fast restore.

If you already have licenses for 3PAR snapshots (which most customers do) and you are using a Data Protector capacity license (which most recent customers are) then you can do this for free.

(Customers using Data Protector classic license need to pay for a ZDB license but they are surprisingly cheap.)

I've got a customer doing this right now, and they are getting around 250MB/s of backup speed (this is LAN-free, with a relatively old Windows server mounting the snapshot and then writing to a D2D4700 using Catalyst-over-Fibrechannel).

The documentation has a few holes (such as neglecting to mention that you need to use omnidbzdb --ompasswd to tell Data Protector which 3PAR account to use) but overall it seems to be more reliable than traditional VMware snapshots.

Here's the backup flow:

• Prepare all the virtual machines
• Tell VMware to snapshot the VMDK
• Tell the 3PAR to snapshot the LUN the VMDK is on.
• Tell VMware to release the VMDK snapshot
• Mount the LUN snapshot on a spare ESXi server (which can be part of the Vsphere environment, it just has to be explicitly also imported into to the client list)
• Mount the snapshot LUN on a physical host that has access to the 3PAR storage and your backup device
• Tell VMware to make a virtual machine (named after the original virtual machine)
• Attach the VMDK on that snapshot LUN to the new virtual machine
• Backup the virtual machines
• Tell the ESXi server to snapshot the VMDK (dumb, I know)
• The backup host reads from the LUN, and writes to the StoreOnce
• Unmount the LUN from the backup host
• Tell the ESXi server to release the VMDK snapshot
• Clean up each virtual machine
• Destroy the virtual machines on the ESXi server.
• Destroy the LUN snapshot on the 3PAR

If this is something you want to do -- and it really does work very well -- let me know and I can organise licenses and/or implementation for you.

Greg Baker is an independent consultant who happens to do a lot of work on HPE DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HPE and HPE partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector, or visit the online store for Data Protector products, licenses and renewals at http://store.data-protector.net/ 

Data Protector Granular Recovery Extension (GRE) and HP OneView don't play nicely together

Are you tearing your hair out about this mysterious error?

Http Error - 500 
{"message":"Unexpected error :There was a problem while sending query to service https://cellmgr.ifost.org.au:7116/dp-gui/grePlugin/GetMountProxies.jsp?newMachineTypeParam=MountProxy. Check that this agent is accessible."}

Then you are presumably trying to configure the HPE DataProtector Granular Recovery Extension (GRE) in version 9.05. And sometimes it works, and then.... it doesn't. Even when you haven't changed anything.

What's going on is that if anyone starts HP OneView on that Vsphere server, the GRE will start getting that error. The only workaround at the moment is to restart the Vsphere server (!) and try again, and hope that no-one looks at OneView.

Or uninstall OneView.

Or uninstall Data Protector VMware Granular Recovery Extension.

HPE's tracking code for this problem is QCCR2A65688, but at the moment it's still under investigation.

In the interim, just hope that you don't need to restore an individual file from a VMware-level backup because you will be manually restoring the VMDK, manually mounting it and then collecting the files out that you want.

You can also ditch VMware-level backups for the moment, install an agent and use file-level backups until it's resolved.

I'll update this blog post when a fix is released.


Greg Baker is an independent consultant who happens to do a lot of work on HPE DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HPE and HPE partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector, or visit the online store for Data Protector products, licenses and renewals at http://store.data-protector.net/ 

VMware ESX 6.0 bug with CBT

VMware has announced another CBT problem. Just a reminder, this is not a problem that HPE can do anything about in Data Protector -- it's a problem with the APIs that VMware have supplied for HPE to use.

If you are doing VEAgent backups of your VMware environment (which is quite common) and you have any incrementals scheduled (also quite common), and you are running ESX 6 (which is lots of people) and you are using CBT (which you really, really would want to do normally).... then you might want to be aware that (yet again) VMware have announced that your backups could well be painfully broken.

Here's VMware's KB article:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2136854

There are several solutions:

• Only do full backups. Hmm, that's a lot of data. Probably OK if you are going to a StoreOnce dedupe, but that's going to turn into a lot more tape.
• Turn off CBT. Ouch, that's going to hurt performance.
• Downgrade to ESX 5.5. I don't see anyone doing that.
• Using the DataProtector disk agent and automated disaster recovery module. This is actually cheaper (no extension licenses required!) and gets you both a file-level backup and an ability to restore a virtual machine from nothing. I recommend this as a better approach generally, but particularly now when we can't trust our VM-level backups.
• Apply the patch that VMware has now released.

Less easy solutions, but things to think about:

• Migrate all your virtual machines to Amazon machine images. (Or Google, or Azure. Pity it can't be HP any more). It's inevitable -- eventually -- that the economies of scale of the large cloud providers will overtake your ability to run things in your own data centre. So why not start planning for it now?
• Use a different virtualisation solution. This is not the first time that VMware have announced "by the way, all backups are broken". I suspect it won't be the last time either. KVM is very mature now and it's also free. Xen is in good shape too. Virtualisation technology is no longer cutting edge -- it's commoditised now. So why not pay commodity prices?

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector, or visit the online store for Data Protector products, licenses and renewals athttp://store.data-protector.net/ 

DataProtector and TCP wrappers (libwrap) etc.

While it's rare to run into a system using TCP wrappers rather than a host-based firewall, I ran into one today in the form of the Vsphere vCenter Server Appliance.

To cut a long story short, you can install the Data Protector agent quite happily (I pushed it from my Linux-based installation server which has OB2_SSH_ENABLED=1 by adding my ssh key to the server appliance's .ssh/authorized_keys). But then it can't be imported.

You will see lines like this appear in the appliance's /var/log/messages :

2015-01-19T01:25:06+00:00 app01 xinetd[19865]: libwrap refused connection to omni (libwrap=inet) from ::ffff:192.168.1.14

2015-01-19T01:25:06+00:00 app01 xinetd[19865]: FAIL: omni libwrap from=::ffff:192.168.1.14

It's been so long since I dealt with TCP wrappers that I spent ages remembering what to do. In the end, it's just a matter of putting the following into /etc/hosts.allow

inet: 192.168.1.0/255.255.255.0 : ALLOW

Adjust based on whatever IP ranges and subnets you need to allow. Or use "ALL" instead of 192.168.1.0/255.255.255.0 if you don't have any security concerns.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Permissions you need for Data Protector to be able to back up VMware

It's on pages 43-44 of the IntegrationVirtualization manual, but it's over a page break and I can never find it when I need it. These permissions need to be at the top level. Doing it at a lower level (e.g. at the datacentre level) doesn't seem to work.

• Datastore -> Allocate space
• Datastore -> Browse datastore
• Datastore -> Low level file operations
• Datastore -> Remove file
• Datastore -> Rename datastore
• Folder -> Delete folder
• Folder -> Rename folder
• Global -> Disable methods
• Global -> Enable methods
• Global -> Licenses
• Host -> Configuration -> Maintenance
• Host -> Inventory -> Add standalone host
• Network -> Assign network
• Resource -> Assign virtual machine to resource pool
• Resource -> Remove resource pool
• Resource -> Rename resource pool
• Sessions -> Validate session
• vApp -> Delete
• vApp -> Rename
• vApp -> Add virtual machine
• Virtual machine -> State -> Revert to snapshot
• ​Everything under ​Virtual machine -> Configuration
• Virtual machine -> Interaction -> Answer question
• Virtual machine -> Interaction -> Power Off
• Virtual machine -> Interaction -> Power On
• Virtual machine -> Inventory -> Create new
• Virtual machine -> Inventory -> Register
• Virtual machine -> Inventory -> Remove
• Virtual machine -> Inventory -> Unregister
• Everything under ​Virtual machine -> Provisioning
• Virtual machine -> State -> Create snapshot
• Virtual machine -> State -> Remove snapshot

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Auditing VMware backups

A customer asked me to report on whether every virtual machine in their VMware environment was getting backed up.

HP Data Protector includes a report on the last successful backup for various objects, but it doesn’t provide a convenient way of tying that in with what is on a VMware cluster.

So I wrote a program, imaginatively called vm-backup-audit.pl. You can get a copy from http://www.ifost.org.au/dataprotector/software/vm-backup-audit.pl

This program queries the vcenter-server given as a command-line argument, and identifies all the virtual machines on that server. It uses the VMware Perl SDK to do this (there's a program called vidiscover.pl which is makes use of.)

It also queries the Data Protector internal database for the last 14 days to find out what objects have been backed up during VEAgent backups. It then prepares a list of virtual machine names and shows when they were last (successfully or unsuccessfully) backed up. 


If the virtual machine has never been backed up successfully in the time frame , the message no full backup completed cleanly will be shown. Otherwise, the relevant session IDs will be shown in reverse chronological order.
This is the kind of output it gives:

[LaptopDatacenter:linuxvm1] 2014/12/29-6 (Full) 2014/12/29-5 (Incremental)
[LaptopDatacenter:unbacked-up-vm] <no full backup completed cleanly>
[LaptopDatacenter:linuxvm3] 2014/12/29-6 (Failed Full) 2014/12/29-5 (Incremental) 2014/12/28-1 (Full)

In full honesty, there are some obvious short-comings:

• It doesn’t correctly handle two virtual machines with the same name in the same data center. This is probably impossible anyway, so doesn’t matter. If they are in different data centers it is able to distinguish them.
• It’s not smart enough to understand that a virtual machine might be getting cloned or replicated between data centres.
• It might not cope very well with mixed Hyper-V and VMware environments. It might not cope very well with two instances running simultaneously.
• It has only been tested on a version 9.02 Linux-based cell console, talking to a Windows cell manager. It won't be hard to get working on anything else, but I just haven't done it yet.

If these matter to you and have a budget to cover fixing any of these, please get in touch and I'll see what I can do.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

VMware, Changed Block Tracking, disk expansion and silent backup corruption

VMware have released this Knowledge Base article.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2090639

If you have Change Block Tracking turned on (which is a sensible thing to do) and then you expand a virtual disk to be larger than 128GB, the data provided to the backup provider is wrong.

I haven't verified this with Data Protector, but I can't see any reason why DP wouldn't be affected in the same way as (say) Veeam, which uses the same mechanism.

Solution: turn off change block tracking, run a backup, and then turn change block tracking on again.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Data Protector 8.13 released

The list of fixes is very long, including a lot of VMware issues. If you have a support contract, you can download it from the following links:

• Windows
• Linux
• IA64 HP-UX

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

First look at Smart Cache

Data Protector 9.0 introduces a new kind of backup to disk device, imaginatively called "SmartCache". It is designed solely for VMware backups.

If you have backed up a virtual machine into a smart cache, and you happen to have the granular recovery extension installed, you can restore individual files from the VMDK very quickly. The end-to-end time to restore a file from a VMDK backup is not that different to restoring a file from a filesystem backup, actually. That is because instead of having to restore the whole VMDK file to a spooling location, the smart cache is essentially an already restored version, and it's just a matter of mounting it and browsing through it.

Starting in version 9, the granular recovery extension only works with VMware's web interface (the flash version). I spent a painful morning chasing up why the VMware Granular Recovery Extension comes up only in the "Available Plug-ins" section of the Vsphere Plug-in Manager with a status of "No client side download is needed for this plug-in". The answer is, it's like the VR Management plug-in: it only supports the web interface, not the vSphere client.

The smart cache device does no deduplication. VEAgent backups can't do in-line compression -- nor would it make sense here anyway -- so the amount of disk space required is exactly the same as the space required for the original VMDKs that you are backing up. Actually, it you want to keep a few days' worth, then you could well be looking at several times as much disk space required for your backups as for the originals. The client I'm doing this for has no Linux servers, otherwise I would have suggested putting the smart cache onto a BTRFS filesystem!

What this means is that you will probably pair the smart cache backup with an object copy job a few days later. Backup to the smart cache now, keep 2-3 copies in smart cache and then copy the oldest backup to a deduplication store where you can store it for a month or longer on disk, or possibly longer on tape.

Note that the default block size for the smart cache device is 1024Kb, which is larger than the defaults for (say) StoreOnce or most tape drives. This means that unless you change this (either by making the smart cache block size smaller, or the other device larger) that you won't be able to do any kind of object copy. Instead you will get errors like this:

[Major] No write device with suitable block size found.

If you don't have licenses for the VMware granular recovery extensions, then don't bother: just use a StoreOnce software device and save yourself a few TB of disk space.

Also, I would recommend bench-marking your VEAgent backups against a filesystem backup. You can create bootable disaster recovery images from filesystem backups (providing most of the advantages of a VMDK-level backup) and quite often the performance is comparable to (or sometimes better than) a VMware snapshot backup. Since filesystem backups can go to a StoreOnce store, it's much more space efficient than the combination of VEAgent + smart cache, and you can (obviously) restore individual files.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Data Protector 9.0 released

Normally the first I see of a new release is when it appears for evaluation on
http://www.hp.com/go/dataprotector but version 9 seems to have been released to customers on support contracts first.

The release notes are quite brief: nearly everything that has been announced as "new in version 9" was already available in the patches that took version 8.12 (Windows / Linux ). Integration with DataDomain, for instance.

The only obviously new option for most customers is that there is a new kind of backup-to-disk device ("Smart Cache") which you can use for VMware VEAgent backups. Then you can use the VMware GRE (Granular Recovery Extension) to extract out individual files from the Smart Cache device without having to restore the whole virtual disk first. This is obviously a big win for backing up virtual machines with large disks: it won't be necessary to do a file-level backup and a VMware backup.

Customers with large B6200 / B6500 arrays might find the federated de-duplication option useful because it means you don't have to assign engines to particular devices.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

VMware, Data Protector and virtual machines which won't consolidate

Working on a customer's systems recently,  there were a large number of virtual machines with the following error message:

Configuration Issues
Virtual machine disks consolidation is needed.

But if I tried to right-click in vCenter and select Snapshots -> Consolidate, what I got was "unable to access file <unspecified filename> since it is locked".

This was also causing error messages in the backup log, because HP Data Protector attempts to consolidate disks at the start of a full backup.

The VMware KB articles suggested various things to identify the lock. I ssh'ed in and ran

tail -f vmware.log | grep lock

to identify what the lock could be. As it turned out, it wasn't quite a lock. The file that couldn't be opened was a .vmdk file - no surprises there. So I ran

lsof | grep the-vmdk-file

This showed that two different processes had it open.

ps | grep process-id-from-the-previous-step

showed that the two processes were both /bin/vmx, but it was possible to distinguish them by their child vmx-vthread processes.

One of them was the process running the virtual machine (no surprises there), and the other was a process belonging to the hostname of the computer that runs their HP DataProtector VEPA agent.

This customer has a virtual machine inside their VMware environment which runs their VMware backups. They don't have to worry about correctly presenting LUNs or having an extra device attached to their SAN fabric. They do source-side deduplicated backups from this virtual machine, so it doesn't generate as much network traffic as it otherwise would.

What had happened was that some backup had failed spectacularly leaving the snapshots mounted on the VEPA agent virtual machine. Looking at the settings for the agent virtual machine it proudly said that it had 13 virtual disks - when it should only have had one, its boot disk.

Naturally, VMware couldn't consolidate the snapshots because as far as it was concerned, those snapshots were still in use. VMware also couldn't delete the virtual disks off the agent machine either, because there were snapshots depending on them.

So the solution was:

• Remove the snapshots on the agent machine.
• Remove the extraneous disks from the agent machine.
• Run the snapshot consolidation from the vCenter GUI.

When VMware NBD and NBDSSL backups fail

I was working on some VMware backups when I ran into this strange sequence of messages: a backup which is showing that that is quite possible to backup a VMX file, but not the VMDK files. And the error messages in the session aren't very informative!

[Normal] From: BSM@cell-manager.ifost.org.au "NBD test backup"  Time: 19/05/2014 10:30:16 AM
        Backup session 2014/05/19-5 started.

[Normal] From: BSM@cell-manager.ifost.org.au "NBD test backup"  Time: 19/05/2014 10:30:16 AM
        OB2BAR application on "vepa-agent.ifost.org.au" successfully started.

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:30:17 AM
        Resolving objects for backup on vCenter 'vcenter.ifost.org.au' ... 

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:30:33 AM
        Add Virtual Machine to the backup ... 
                Name: VM1
                Path: /DC/Discovered virtual machine/VM1
                InstanceUUID: 52dbf234-252e-c5dd-9df5-51c304bcf312

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:30:35 AM
        Virtual Machine 'VM1': Locking vMotion ... 

[Warning] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:30:35 AM
         Virtual Machine 'VM1': vMotion is in Progress.

[Warning] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:31:08 AM
        Virtual Machine 'VM1': Could not lock vMotion.

Everything's pretty much fine. There are lots of reasons for a vMotion lock to fail.

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:31:13 AM
        Creating folder /var/opt/omni/tmp/55fd5af8-f853-403e-bedf-2d1e60e418dd ... 

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:31:19 AM  
        Virtual Machine 'VM1': Backing up configuration file VM1.vmx ... 

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:31:20 AM
        Virtual Machine 'VM1': Creating snapshot ... 

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:31:59 AM
        Virtual Machine 'VM1': Optimizing disk scsi0:0 ... 

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:32:00 AM
        Virtual Machine 'VM1': Backing up VSS manifest  VM1/VM1-vss_manifests11.zip.

And now for the interesting part:

[Major] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:33:15 AM
        Virtual Machine 'VM1': Could not backup disk scsi0:0 ... 

[Major] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:33:15 AM
[172:162]       Virtual Machine 'VM1': No disk backed up ... 

[Critical] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:33:15 AM
        Backup of object failed.
                Name: VM1
                Path: /DC/Test_VMs/VM1
                InstanceUUID: 52dbf234-252e-c5dd-9df5-51c304bcf312

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.org.au "/DC"  Time: 19/05/2014 10:33:16 AM
        Virtual Machine 'VM1': Removing snapshot ... 

[Normal] From: VEPALIB_VMWARE@vepa-agent.ifost.or.gau "/DC"  Time: 19/05/2014 10:33:24 AM
        Virtual Machine 'VM1': Unlocking vMotion ... 

Deleted directory /var/opt/omni/tmp/564d01b7-7910-97a0-d54d-85c11ff8becd-vm-58/nbd
Deleted directory /var/opt/omni/tmp/564d01b7-7910-97a0-d54d-85c11ff8becd-vm-58/nbdssl
Deleted directory /var/opt/omni/tmp/564d01b7-7910-97a0-d54d-85c11ff8becd-vm-58/hotadd

[Normal] From: BSM@cell-manager.ifost.org.au "NBD test backup"  Time: 19/05/2014 10:33:53 AM
        OB2BAR application on "vepa-agent.ifost.org.au" disconnected.

I've truncated the rest of the messages.
Turning up the debugging level, the debug logs showed this:

[110] [VddkUtil::diskLibLog] NBD_ClientOpen: attempting to create connection to vpxa-nfcssl://[ESXi-MGMT-VMFS-1] VM1/VM1.vmdk@esxi1.ifost.org.au:902

[110] [VddkUtil::diskLibLog] Started up WSA

[110] [VddkUtil::diskLibLog] CnxOpenTCPSocket: Cannot connect to server esxi1.ifost.org.au:902: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

[110] [VddkUtil::diskLibLog] CnxAuthdConnect: Returning false because CnxAuthdConnectTCP failed

[110] [VddkUtil::diskLibLog] CnxConnectAuthd: Returning false because CnxAuthdConnect failed

[110] [VddkUtil::diskLibLog] Cnx_Connect: Returning false because CnxConnectAuthd failed

[110] [VddkUtil::diskLibLog] Cnx_Connect: Error message: Failed to connect to server esxi1.ifost.org.au:902

[ 20] [VddkUtil::diskLibWarning] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server esxi1.ifost.org.au:902

[110] [VddkUtil::diskLibLog] NBD_ClientOpen: Couldn't connect to esxi1.ifost.org.au:902 Failed to connect to server esxi1.ifost.org.au:902

The clue is the failed connection to esxi1.ifost.org.au. The VEPA backup agent obviously has to connect to the Vcenter server in order to start a backup, but because there was no SAN connectivity between the VEPA agent and the LUNs supporting the VM1 virtual machine's VMDK files, the VEPA agent ends up having to talk to the ESX server directly as well.

There can be many reason for this connection to fail: a firewall could be blocking the connection between the vepa agent and the esx server. Or in this case, there was no DNS entry for esxi.ifost.org.au didn't exist.


Greg Baker is an independent consultant working on HP DataProtector, LiveVault and many other technologies. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector