Search This Blog

Wednesday 24 September 2014

Contender for the world's worst error message: Connection to CRS failed

I've just spent the afternoon chasing down this error message, trying to connect a Linux cell console client to query a Windows cell manager:

Connection to CRS failed.
To start the Data Protector daemons on the Cell Manager host use the command
omnisv -start on the Cell Manager
or check if the communication between the Cell Manager and client is encrypted with the command
omnicc -encryption -status -all on the Cell Manager.
Everything checked out on the Windows cell manager, and encrypted control was off on all nodes.

I ran strace on omnicc, omnicc -debug 1-200, tcpdump'ed every packet from the Linux system in question (there were none) and tried omnicc -server 1.2.3.4 (a non-existent address that should have been unroutable) which responded instantly.

Oh, and "omnicc -debug 1-200 2> /tmp/somewhere" doesn't put the same data into /tmp/somewhere as you get on stderr when you run "omnicc -debug 1-200" and stderr is a tty.

The root cause of the error message about being unable to connect to the server? The hostname of the Linux client didn't resolve correctly.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Disabling Data Protector encrypted control

By default, communications between a cell manager and all cell clients is not encrypted. It's just plain text, and you can intercept the network traffic and even modify it with something like netsed.

So, obviously some sites need to enable encryption. This is nicely documented in the Installation guide, and isn't that hard to figure out anyway (right click on the client and select "Enable encrypted communication").

But there is no documentation on how to disable this.

On the cell manager, there is a file /etc/opt/omni/server/config (which is usually equivalent to C:\ProgramData\OmniBack\config\server\config on Windows).

cellmgr.ifost.org.au={
encryption={
enabled=1;
certificate_chain_file='/etc/opt/omni/client/certificates/cacert.pem';
private_key_file='/etc/opt/omni/client/certificates/cacert.pem';
trusted_certificates_file='/etc/opt/omni/client/certificates/cacert.pem';
pkcs12_keystore_filename='/etc/opt/omni/client/certificates/hpdpcert.p12';
pkcs12_keystore_password='hpdpcert';
pkcs12_ca_certificate_filename='/etc/opt/omni/client/certificates/hpdpcert.p12';
pkcs12_ca_certificate_password='hpdpcert';
pkcs12_private_key_filename='/etc/opt/omni/client/certificates/hpdpcert.p12';
pkcs12_private_key_password='hpdpcert';
};
};
client.ifost.org.au={
encryption={
exception=1;
};
};

The first stanza (for cellmgr.ifost.org.au) has encryption enabled. The one client in the cell (client.ifost.org.au) does not.

In order to undo this, simply remove the content of the encryption clause.

cellmgr.ifost.org.au={
encryption={
};
};
If you restart Data Protector now (omnisv stop ; omnisv start), what will happen now is that you won't be able to connect to the cell manager using the GUI or the command line. This is the kind of error you will get:

Connection to CRS failed.
To start the Data Protector daemons on the Cell Manager host use the command
omnisv -start on the Cell Manager
or check if the communication between the Cell Manager and client is encrypted with the command
omnicc -encryption -status -all on the Cell Manager.
This is because the client-side programs think they are supposed to be encrypting their connections to the cell manager (even if we're running on the cell manager itself), and the cell manager isn't responding with a valid SSL response.

There's another file /etc/opt/omni/client/config ( C:\ProgramData\Omniback\config\client\config ) which looks somewhat similar to the server-side one:

encryption={
        enabled=1;
        certificate_chain_file='/etc/opt/omni/client/certificates/cacert.pem';
        private_key_file='/etc/opt/omni/client/certificates/cacert.pem';
        trusted_certificates_file='/etc/opt/omni/client/certificates/cacert.pem';
        pkcs12_keystore_filename='/etc/opt/omni/client/certificates/hpdpcert.p12';
        pkcs12_keystore_password='hpdpcert';
        pkcs12_ca_certificate_filename='/etc/opt/omni/client/certificates/hpdpcert.p12';
        pkcs12_ca_certificate_password='hpdpcert';
        pkcs12_private_key_filename='/etc/opt/omni/client/certificates/hpdpcert.p12';
        pkcs12_private_key_password='hpdpcert';
};

The plaintext version should look like this:

encryption={
        exception=1;
};

If you were turning off encrypted control for a client, then you will need to update the cell info file (/etc/opt/omni/cell/cell_info or C:\programdata\omniback\config\server\cell\cell_info ) and remove the reference to encryption there too.

-host "client.ifost.org.au" -os "gpl x86_64 linux-2.6.32-279.el6.x86_64" -encryption 1 -core A.09.01 -da A.09.01 -ma A.09.01  -cc A.09.01  -vepa A.09.01  -autodr A.09.01  -StoreOnceSoftware A.09.01 -ts_core A.09.01 

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Tuesday 9 September 2014

The sports wet weather line app

It's a grey Saturday morning, not quite wet enough to be sure that sport is cancelled, and not clear enough to be sure it's on.

You dial the school's wet weather line, and it's engaged because hundreds of other parents are doing the same thing. When you finally get through, you hear that sport is on, but it may get cancelled later. As you bundle the kids into the car you toss up getting a fine for using your mobile to call the wet weather line on the way, or getting there and discovering it was cancelled half an hour ago.

Sound familiar?

It was getting to a Trinity vs Cranbrook game at 7:15 in the morning (yawn!) that made me come up with a better solution.

I've set up some servers that dial the wet weather lines repeatedly and check for any change in message. If the sportsmaster has changed the voicemail message, my servers grab the new messages as an MP3 file.

Then I wrote an app (Android version here, iphone coming soon ) which connects to my servers and polls for any changes.

Just leave this app open on a Saturday morning, and you'll see when there's a new message. You can play it with the touch of a finger, or you can get the children to do it for you. They'll be using your phone in the car anyway, of course.

So far I'm only monitoring a handful of Sydney private schools: King's (TKS), Cranbrook, Trinity Prep, Trinity Grammar and Trinity Junior school. This website: http://wet-weather-line.appspot.com/ has the latest list.

If your school has a wet weather line that you dial (even if there are digits to send), my app can get updates from it. Let me know through this form if you want this done. I'm using Twilio for my phone services, so it can monitor a phone line in any country.

I've been asked to incorporate this into one school's information app (which was easily done). 

I also figured I might as well offer a voicemail service for schools and other sports organisations. If you are coaching a team and need a convenient way of getting wet weather information out, email me (gregb@ifost.org.au) or use this contact form.

My goal: to make sure no-one ever has to get out of bed early on a cancelled-sport Saturday morning again.

Changing the hostname of the Data Protector cell manager

HP DataProtector is very sensitive to changes in the name of the cell manager. There are several reasons for this:

  • Integration clients need to connect to the cell manager to get their credentials.
  • Clients will refuse to be upgraded from a computer that isn't their cell manager.

So if you need to change the cell manager's hostname (even if the domain name changes), there are several things you need to do.

Update the internal database

This is the easiest step of all:

  omnidbutil -change_cell_name

If you are on Windows, remember to run this in a terminal running as Administrator.

Update the cell server web service certificate (DP 8.x and later)

There will be existing certificates in /etc/opt/omni/config/server/certificates (on Linux/HP-UX) and in C:\programData\OmniBack\Config\Server\certificates on Windows. There are actually several levels of folders under this. I removed all the files I found there, leaving a skeleton of empty directories.

I don't know if this is necessary, but getting rid of the old certificates seemed like a good idea.

The new certificate will get used by the hpdp-as service which will decrypt the private key with a password. We might as well re-use the previous password to save on reconfiguring things.

The password is in /etc/opt/omni/client/components/webservice.properties on Linux/HP-UX and C:\ProgramData\OmniBack\client\components\webservice.properties on Windows (unless you chose a different data directory at install time). Yes, client, even though we're changing a server property.

Anyway, the file will look like this:

# global property file for all components
jce-serviceregistry.URL = https://hostname:7116/jce-serviceregistry/restws
keystorePath=/etc/opt/omni/server/certificates/client/client.keystore
truststorePath=/etc/opt/omni/server/certificates/client/client.truststore
keystorePassword=zys124ax52353
truststorePassword=zys124ax52353

Now we need to generate a new certificate:

omnigencert.pl -server_id the-new-cell-hostname -store_password xyz124ax52353 -user_id hpdp

On Windows, the .pl ending might not be associated with the right Perl interpreter (or even any interpreter at all). If so, give the full pathname for Perl and omnigencert.pl like this:
  "C:\Program Files\OmniBack\bin\perl.exe" "C:\Program Files\OmniBack\bin\omnigencert.pl"


Clients

Each client will now have the wrong cell server name. On Unix or Linux boxes, there's a file /etc/opt/omni/client/cell_server which is a text file just listing the name of the cell manager. Use whatever technique you normally push config files out with (puppet? cfengine? scp?) to update this.

Windows boxes keep the cell server in a registry key. Update this:

HKEY_LOCAL_MACHINE\SOFTWARE\Hewlett-Packard\OpenView\OmniBackII\Site\CellServer


Server cell info

The server itself is a client, and presumably the information in the cell_info file (/etc/opt/omni/server/cell/cell_info or C:\ProgramData\Omniback\config\server\cell\cell_info) will need to be updated.

If there were any devices attached to the cell manager, then they will need to be updated. You can do this with omnidownload / omniupload, or just by going into each device in the GUI and applying the relevant changes.

Advanced scheduling webservices (DP 8.1 and later)

Under /etc/opt/omni/client/components (C:\ProgramData\Omniback\config\client\components), there are several webservice.properties files scattered through the filesystem. They register web service components.

For example,  here is dp-jobexecution-backup/webservice.properties:

# dp-jobexecutionengine-backup property file
dp-jobexecutionengine-backup.URL = https://hostname:7116/dp-jobexecutionengine-backup/restws
jce-serviceregistry.URL = https://hostname:7116/jce-serviceregistry/restws

These all have to be updated:

  • webservice.properties
  • dp-jobexecutionengine-backup/webservice.properties
  • dp-jobexecutionengine-consolidation/webservice.properties
  • dp-jobexecutionengine-copy/webservice.properties
  • dp-jobexecutionengine-verification/webservice.properties
  • dp-loginprovider/webservice.properties
  • dp-scheduler-gui/webservice.properties
  • dp-webservice-server/webservice.properties
  • jce-dispatcher/webservice.properties
  • jce-serviceregistry/webservice.properties

Then restart DataProtector (omnisv stop ; omnisv start)

Licenses

Data Protector licenses are tied to the hostname and IP address of the cell manager. So you will need to visit http://webware.hp.com/ to get a new set of license keys for your cell manager.

Incidentally, if you are moving or renaming a cell manager, you might find it helpful to read this book: Migrating and Cloning Data Protector cell managers


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector, or visit the online store for Data Protector products, licenses and renewals at http://store.data-protector.net/