GRE for Linux requirements

If you are using Data Protector to backup your VMware environment, and you have Linux boxes, you might have tried to use the Granular Recovery Extension (VMware GRE). The GRE lets you recover individual files from a VM-level or VMDK-level backup; it does this by mounting the VMDK file on a Linux GRE proxy.

There are three variations of GRE restore:

• If you backup to a StoreOnce device (e.g. a B6200, a D2D4500, a StoreOnce virtual appliance, or a software storeonce component on a Windows or Linux member of the cell)... then you need a very large disk and not much else.
• If you backup using 3PAR snapshots (which works very well indeed) and you are doing a GRE recovery from a snapshot, you don't need a big disk or anything else much at all -- just something connected to the 3PAR the meets the usual (documented) list of requirements.
• The very weirid case is if you use a SmartCache device. These are uncompressed, raw disk spaces for putting VMware backups on to. The SmartCache is accessed by the Windows and Linux proxies via Windows file sharing. Thus the Linux GRE proxy server needs to have Samba installed on it.

One remaining issue, that I hope gets fixed one day soon: the machine that you want to restore to has to have a world-writeable NFS share. Ouch: given that it is copying from one Linux box to another Linux box, I'm not quite sure why this couldn't have been done with SFTP. So I suspect everyone will just have a multi-stage restore for GRE on Linux boxes:

• Load the backup from StoreOnce onto a large disk.
• Restore files from that to a server with an insecure NFS share.
• Copy from the NFS share via SSH to the actual server where you needed the file restored.

I presume also that setuid / setgid binaries are therefore not supported for GRE restore. (Because who creates a world-writeable NFS share without the nosetuid,nosetgid options enabled?) Implication: you can't restore /usr or /sbin from a VMware backup reliably.

Of course, you might find it cheaper to use VMX (https://store.data-protector.net/collections/vmx-licenses) instead of Data Protector -- it is much cheaper than the equivalent GRE and VEAgent licenses. As far as I know, it doesn't face these limitations.

Greg Baker is an independent consultant who happens to do a lot of work on HPE DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HPE and HPE partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector, or visit the online store for Data Protector products, licenses and renewals at http://store.data-protector.net/ 

Planning, Deploying and Installing Data Protector 9: For the datacentre, the cloud and remote offices

This is the fourth book I've published on HP Data Protector, and I think it will be a very valuable resource to any sysadmin or consultant working with HP products.

If you are involved in pre-sales or implementation consulting you will find lots of useful material in this book.

• Why customers choose Data Protector, and what its strengths and weaknesses are.
• Step-by-step, everything you would need in order to backup remote branch offices.
• Very detailed designs on how you can backup Amazon-hosted cloud servers.
• How to backup data centres filled with nearly identical virtual machines.
• Guidance on handling media pools, tape drives and other physical devices.
• Doing disk-to-disk-to-tape backups.
• Very detailed information on the internal database that simply isn't available elsewhere -- the results of my reverse engineering what HP has done. If you have complicated reporting needs, this is the book to have.

If you are a system administrator / backup administrator, you will find this book a welcome alternative to the HP documentation. I talk honestly about bugs that I've found, what makes sense to do, and what doesn't, and more importantly why certain designs make more sense than others.

This book can also be used as a course book if you want to run a one-day training session. All the labs can be run up in the cloud.

To keep the price down on the paperback version, it's a B&W print for all 374 pages. The Kindle version is full-colour if you have a full-colour device.

The paperback version has a comprehensive index to make this into an ideal reference to have at your desk. It is enrolled in matchbook, so you can have both the paper version and the kindle version for only slightly more than the paper version. Buy both!

Topics covered:

• Reasons customers choose Data Protector 
• Areas of comparative weakness 
• Installing Windows and Linux cell managers 
• Agent deployment methods 
• StoreOnce de-duplication 
• Backing up filesystems 
• Backing up VMware 
• Backing up cloud-hosted servers 
• Configuring tape drives 
• Managing media 
• Backups spooled via disk 
• Remote office backups replicated to a central office 
• Reporting 

There is also a companion book on Data Protector for operators with a topics around restoring, supporting and maintaining.

Buy it on Amazon ( http://www.amazon.com/dp/B00T68FR64 ), or if you want to use an alternate store, they are listed here: http://www.ifost.org.au/books

Greg Baker is one of the world's leading experts on HP Data Protector. His consulting services are at http://www.ifost.org.au/dataprotector. He has written numerous books (see http://www.ifost.org.au/books) on it, and on other topics. His other interests are startup management, applications of automated image and text analysis and niche software development.

Details about the schema of the Data Protector internal database

As far as I can tell, HP don't document the schema of the PostgreSQL internal database. What follows is my investigations as I was chasing up a customer for whom thousands of sessions were giving strange results. (When they looked at a medium and went to the Objects tab to find, the GUI responded with "in order to delete this medium, export it first".)

There are seven important database tables which are affected in the normal operations of running a backup.

The first one is dp_management_session.

• It has a column "name" which looks like this: "2014/07/14 0007" - in other words, the session name as it appears in every command, except that it has a space instead of a dash.

The name is used as a unique key (as it should be unique!) in conjunction with the column 'application_uuid'. I haven't figured out exactly what that's doing, but I'm presuming it's something to do with the manager-of-managers product, where you might have centralised all your media into one cell. In this case you could have two or identical session names referred to in the database, one from each of the client cells and one from the manager cell. To simplify things, I've ignored the application_uuid column(s) in the diagram.

Obviously, the backup should write something. If this is the first time this filesystem (or database or Exchange server, etc.) has been backed up, then the dp_objects table will have a new row added to it, with the hostname, mountpoint and label of the object being backed up. The columns uuid and dp_numkey act like the primary key for this table, which means that if you aren't running manager-of-managers, the dp_numkey will be unique.

Each time a backup of that filesystem runs, a row is added to dp_catalog_object_version. If there are several filesystems being backed up in one job, this table may have many rows added for each backup run.

• There is a column backup_name which partly references dp_management_session.name. I say partly, because there is no foreign key between them, and in fact, sometimes backup_name is null. Presumably what's going on is that a backup could have a copy made, and then the original expires, delete the original session, leaving a catalog object version which doesn't correspond to a session.
• The column object_seq_id references dp_objects (together with the usual uuid story).
• The primary key is the combination of application_uuid (as usual) and a field called seq_id.

There is a row created in db_catalog_object_datastream and also one in dp_catalog_object_versession for each row added to dp_catalog_object_version. These don't seem very interesting: the former looks like it's something to do with enforcing device policies, and the latter a record of a post-backup verification.

The oddly and painfully named dp_catalog_position_seqacc_med maps backup objects to positions on tapes. This is obviously a very large table!

• The column objver_seq_id references the dp_catalog_object_version's seq_id column, essentially "what is backed up here?"
• The column medium_name references the unique header ID of the tape, for example '7b5ba8c0:53c3ae35:07eb:0014'

There is another table called dp_positions which is a little bit more accessible, but inserts, updates and deletions from this table trigger a function instead (presumably to update dp_catalog_position_seqacc_med). In a few tests this table got populated and dp_positions did not.

If you are backing up to a StoreOnce device, or to a file library then there's a good chance that this backup will cause a new medium to be created. This will also happen when you format a new tape in either a physical tape library of a virtual tape library.

The tapes are all listed in dp_medmng_media_pool_media. The column medium_seq_id (which is not the medium header, it's just an ID) is the key into the dp_medmng_medium tape.

There seems to be a distinction made between the medium itself, and the cartridge holding it. The dp_medmng_cartridge table has a barcode, a physical_location and a contained medium_seq_id. The dp_medmng_medium table has a unique seq_id and a name. The name is the header on the tape.

I still haven't figured out the way that file names are stored. Presumably this is in DCBF files until you run omnimigrate. In this transition time (which seems to be the default on new installs as well), some records get written to dp_catalog_dcbf_directory and dp_catalog_dcbf_info.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

DataProtector undocumented features -- running a script against the database

In my (in)famous write-up of how to un-break the DataProtector IDB after an upgrade (http://www.ifost.org.au/dataprotector/documents/dp8db-hack.pdf if you haven't seen it already), I gave a convoluted procedure for getting SQL-level access to the database.

I've since discovered that the password is simply Base64 encoded in the $OMNICONF/config/server/ID/idb.config file.

But today I discovered an undocumented omnidbutil feature:

omnidbutil -run_script sqlcommands.sql  -detail

Just create commands that you want to run in a text file. If you are on Windows, make sure the SQL file is saved in ASCII format rather than Unicode.

Much easier and I'm going to put that into the next edition of my Data Protector book.

Greg Baker is an independent consultant working on HP DataProtector, LiveVault and many other technologies. See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector.