Auditing VMware backups

A customer asked me to report on whether every virtual machine in their VMware environment was getting backed up.

HP Data Protector includes a report on the last successful backup for various objects, but it doesn’t provide a convenient way of tying that in with what is on a VMware cluster.

So I wrote a program, imaginatively called vm-backup-audit.pl. You can get a copy from http://www.ifost.org.au/dataprotector/software/vm-backup-audit.pl

This program queries the vcenter-server given as a command-line argument, and identifies all the virtual machines on that server. It uses the VMware Perl SDK to do this (there's a program called vidiscover.pl which is makes use of.)

It also queries the Data Protector internal database for the last 14 days to find out what objects have been backed up during VEAgent backups. It then prepares a list of virtual machine names and shows when they were last (successfully or unsuccessfully) backed up. 


If the virtual machine has never been backed up successfully in the time frame , the message no full backup completed cleanly will be shown. Otherwise, the relevant session IDs will be shown in reverse chronological order.
This is the kind of output it gives:

[LaptopDatacenter:linuxvm1] 2014/12/29-6 (Full) 2014/12/29-5 (Incremental)
[LaptopDatacenter:unbacked-up-vm] <no full backup completed cleanly>
[LaptopDatacenter:linuxvm3] 2014/12/29-6 (Failed Full) 2014/12/29-5 (Incremental) 2014/12/28-1 (Full)

In full honesty, there are some obvious short-comings:

• It doesn’t correctly handle two virtual machines with the same name in the same data center. This is probably impossible anyway, so doesn’t matter. If they are in different data centers it is able to distinguish them.
• It’s not smart enough to understand that a virtual machine might be getting cloned or replicated between data centres.
• It might not cope very well with mixed Hyper-V and VMware environments. It might not cope very well with two instances running simultaneously.
• It has only been tested on a version 9.02 Linux-based cell console, talking to a Windows cell manager. It won't be hard to get working on anything else, but I just haven't done it yet.

If these matter to you and have a budget to cover fixing any of these, please get in touch and I'll see what I can do.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Lots of interesting new features and support in 9.02

• Data Protector can now back up mainframes: IBM System z running z/OS version 2.1. If there was any doubt whether HP is targeting IBM customers during IBM's current implosion, I think this should put it to rest. HP is obviously trying to position DP against TSM.
• There is now VMware 3PAR integration. I haven't researched this in any depth, but I presume what this means is that it is now possible for Data Protector to snapshot 3PAR arrays which are using LUNs to store VMFS volumes. If it works like Oracle, filesystems, MS-SQL, etc. then Data Protector should be able to mount this onto another system and back it up, and also restore snapshots instantaneously.
• Data Protector introduces support for pausing the lower priority backup sessions when the scheduled higher priority jobs share the same device. The paused jobs are resumed once the device is available.
• Viewing the names of VMs (rather than UUIDs) in VEagent backups is available again
• Catalyst over fibre channel
• I didn't see it documented, but it looks like this is the release where cloud backups [link to blog post] were introduced.
• The VMware Granular Recovery Extension has a new user interface, and a new way of being installed.

As a reminder, I keep a list of all the latest patches (including the 9.02 patch bundle) at http://www.ifost.org.au/Documents/dp-patch-list.html.

And as another reminder, if you are going to upgrade to 9.02 by running a new cell manager alongside an old cell manager, then you will want to buy my book on migrating and cloning cell managers ( http://www.ifost.org.au/press/#dp ) which looks like it was the best selling book on business software by an Australian author on Amazon Australia last week.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Migrating Linux or HP-UX cell manager configuration to and from a Windows cell manager

All the backup definitions (including integration backups, schedules and so on) in Data Protector are plain text files so they are easy to edit and easy to move around.

On Windows, these are UTF-16 encoded, and on HP-UX and Linux they are ASCII encoded. If you copy a backup specification out of C:\ProgramData\Omniback\config\server\datalists on a Windows system to /etc/opt/omni/server/datalists on a Linux machine, the resulting file won't work properly. It will appear to have every second character have a null in it.

Fortunately, Linux boxes come with a utility called iconv. Here is how to use iconv to convert a datalist which was on a Windows system after it has been transferred to a Linux system.

iconv -f UTF-16 -t ASCII datalists/main-backup > datalists/main-backup.tempmv datalists/main-backup.temp datalists/main-backup

You can't just redirect output back to datalist/main-backup because that will overwrite the file before it is read.

If you need to do a whole directory:

find . -type f -exec sh -c 'iconv -f UTF-16 -t ASCII "{}" > "{}.temp" && mv "{}.temp" "{}"'  ';'

Of course, if you are migrating between cell managers, then you should pick up a copy of my latest book on migrating and obsoleting Data Protector cell managers.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

A new Data Protector book

I've recently published another book based on my experiences with customers migrating off physical hardware and off older versions of Data Protector.

This is not a long book, which is why I've priced it at a fraction of anything I've written before, but I hope you find it valuable.

This book is for consultants and system administrators who need to migrate, clone or obsolete an instance of HP Data Protector.
• How to duplicate an existing cell manager on to a new system (one with new hardware, virtualised hardware, or a new operating system).
• How to merge two cells into one, leaving you with one less cell manager than before. In Data Protector version 7 and earlier there were a lot of limitations on the size of a Data Protector cell and how much a cell manager could do. This meant that many customers had several cells -- in version 8 and onwards this is not really necessary. This book walks step-by-step through how you would consolidate cell managers.

Read this book before you start planning your next Data Protector upgrade!

Links to buy from Amazon (in Kindle format) are here, but I've re-done the way I build these books and I can now produce them in other formats as well. Contact me if you need me to do this for you.

http://www.ifost.org.au/press#dp

Odd behaviour on a large incremental backup job

I've been investigating why one of my customer's incremental backups is so large.

They are doing incrementals forever -- they run one full backup a long time ago, and have been synthesising full backups each weekend ever since out of their incremental backups.

One backup object has 500GB backed up every night, and as it's a small third-world country branch office, this didn't make much sense.

When I went into the session (e.g. by pretending to start a restore) and tried to find which files had changed, I came up blank. The modification time of most of the files was way in the past. There was no reason they should have been backed up.

Adding to the mystery, when I turned off "Do not use archive attribute" the incremental shrunk down to something sensible.

So either Data Protector isn't resetting the archive attribute flag in this particular situation, or something else is turning the archive flag on.

The file server in question is running Win2k8 R2 with a Data Protector 8.11 connecting to a 9.01 cell manager. I'll have to upgrade it confirm it's still a problem, but I suspect it will continue to be.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Cannot write to device (JSONizer error: Invalid inputs)

I ran into the following problem at a customer site running 9.01 with all the latest patches. In context it was fairly obvious what was going on. But I pity anyone running into this cold because it would have been a very long and slow debugging process.

[Critical] From: VBDA@linux1.ifost.org.au "linux1.ifost.org.au [/home]" Time: 26/11/2014 4:03:41 PM
[80:1031] Received ABORT request from NET => aborting.

[Major] From: BMA@backup.ifost.org.au "LinuxBackupStorage [GW 4464:0:11286004362757246666]" Time: 26/11/2014 4:03:41 PM
[90:51] \\backup.ifost.org.au\LINUX_OS\1f00010a_54755f1f_06cc_0099
Cannot write to device (JSONizer error: Invalid inputs)

[Critical] From: VBDA@linux1.ifost.org.au "linux1.ifost.org.au [/home]" Time: 26/11/2014 4:03:41 PM
Connection to Media Agent broken => aborting.

The relevant part of the datalist file (in /etc/opt/omni/server/datalists on a Linux-based cell manager, and C:\programdata\omniback\config\server\datalists on a Windows-based cell manager) looked like this:

HOST "linux1.ifost.org.au" linux1.ifost.org.au
{
   -trees
      "/"
   -exclude
      "/home/scratch"
      "/home/pgdumps"
}

There's nothing wrong with that: it will work correctly when writing to a tape device. The same structure (host with exclusions) would probably work on a Windows box. But a Linux client backing up to a StoreOnce store causes the BMA to crash.

This also occurs when you split it into filesystems.

FILESYSTEM "linux1:/" linux1.ifost.org.au:/ {
}
FILESYSTEM "linux1:/home" linux1.ifost.org.au:/home { 

   -exclude 

      "/home/scratch" 

      "/home/pgdumps" 

}

Both of these backup specifications will work correctly when written to tape. Presumably one day a patch will be released to fix this. Contact HP support if necessary, and I'll try to update this posting when it's fixed.



Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector