Search This Blog

Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

Friday, 10 June 2016

GRE for Linux requirements

If you are using Data Protector to backup your VMware environment, and you have Linux boxes, you might have tried to use the Granular Recovery Extension (VMware GRE). The GRE lets you recover individual files from a VM-level or VMDK-level backup; it does this by mounting the VMDK file on a Linux GRE proxy.

There are three variations of GRE restore:
  • If you backup to a StoreOnce device (e.g. a B6200, a D2D4500, a StoreOnce virtual appliance, or a software storeonce component on a Windows or Linux member of the cell)... then you need a very large disk and not much else.
  • If you backup using 3PAR snapshots (which works very well indeed) and you are doing a GRE recovery from a snapshot, you don't need a big disk or anything else much at all -- just something connected to the 3PAR the meets the usual (documented) list of requirements.
  • The very weirid case is if you use a SmartCache device. These are uncompressed, raw disk spaces for putting VMware backups on to. The SmartCache is accessed by the Windows and Linux proxies via Windows file sharing. Thus the Linux GRE proxy server needs to have Samba installed on it.

One remaining issue, that I hope gets fixed one day soon: the machine that you want to restore to has to have a world-writeable NFS share. Ouch: given that it is copying from one Linux box to another Linux box, I'm not quite sure why this couldn't have been done with SFTP. So I suspect everyone will just have a multi-stage restore for GRE on Linux boxes:
  • Load the backup from StoreOnce onto a large disk.
  • Restore files from that to a server with an insecure NFS share.
  • Copy from the NFS share via SSH to the actual server where you needed the file restored.
I presume also that setuid / setgid binaries are therefore not supported for GRE restore. (Because who creates a world-writeable NFS share without the nosetuid,nosetgid options enabled?) Implication: you can't restore /usr or /sbin from a VMware backup reliably.


Of course, you might find it cheaper to use VMX (https://store.data-protector.net/collections/vmx-licenses) instead of Data Protector -- it is much cheaper than the equivalent GRE and VEAgent licenses. As far as I know, it doesn't face these limitations.

Greg Baker is an independent consultant who happens to do a lot of work on HPE DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HPE and HPE partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector, or visit the online store for Data Protector products, licenses and renewals at http://store.data-protector.net/ 

Wednesday, 2 December 2015

My part in the making of WiFi



Between 1994 and 1996 I was working at CSIRO Radiophysics (which turned into Telecommunications and Industrial Physics). Terry Percival was my boss' boss, and Diet Ostry and I shared an office. This story happened just a little bit before Terry, Diet and the two Johns had starting applying the radio signal unsmearing algorithms that CSIRO ended up with patents for which formed part of the WiFi standard.

One day Dr Percival set me (fresh-faced, obnoxious, know-it-all graduate) the challenge of solving the hardest problem in radio communications at the time: how can A and B communicate reliably, if A can't detect C's signal, and C can interfere with B?

My thoughts on the matter was that everyone was mis-stating the problem. It's only going to be a serious problem if you want to broadcast at 2.4 GhZ. If you drop the frequency of the signal down to something so low that even an iron ore mountain is transparent to it, it would be a very strange environment where A & C couldn't communicate.

So therefore, the real problem was that we were trying to do high speed networking. On the contrary, what we should be researching is extremely low-speed networking. How could we have useful and reliable communication at only a few bits per second?

Latency, jitter, high-speed CPUs to perform processing -- all these hard problems go away when you are only dealing in bits per second.

There were three other very good reasons why I thought low-speed networking was the right thing to look at, too: Linux, mining and submarines.

At the time, Linux was just making inroads into our thinking. The business world was dominated by IBM mainframes and (even in 1996) Windows 3.11 crashing was a daily experience for most people's workday.

The prevailing opinion that the team in the signal processing wing of CSIRO Radiophysics developed was that source-available (free-to-modify) software was unstoppable, and in a short time would conquer everything else, particularly Microsoft. After all, if the source was available, the program could never truly become unavailable or die, like proprietary software would. Software distribution bloat was about to go away, because we would all be getting our software in source form and compiling it. The days of elegant software that did exactly what it was supposed to without cruft were just around the corner because of the massive growth in volunteer developers who would tidy up anything and everything.

Which led me to the conclusion that we wouldn't really need high speed networks. The future was going to be everyone having these extremely reliable, high performance desktops (32-bit Linux never crashed; and the difference in this and also in performance was night and day compared to 16-bit Windows 3.11). All the software we would ever want would already be on our local harddisks -- all of it free -- and that there simply wouldn't be enough "stuff" to send over a network to even justify upgrading existing 9.6k modems. (I used to dial in on a 2.4k modem most of the time, myself).

I had been working on a related geophysics project as well. It was deployed on Linux (tying into the future-of-operating-systems theme) and deployed radio transmitters and receivers down boreholes in order to draw conclusions about the kinds of rocks in a region. It seemed like geophysical technologies were going to be a significant part of Australia's research future (at least I got something right!), and the need to deliver communications down into mines (where very low bandwidth would be inevitable) seemed like a worthwhile research direction.

The issues with the Collins class submarines at the time (including: how do we communicate with a submarine deep underwater?) made it seem to me like all the arrows were pointing at low-speed rather than high-speed communication.

I was so convinced that Terry and Diet (and John Deane, who was just down the corridor; and John O'Sullivan whom I think I interacted with a couple of times) were on the wrong track that I ended up quitting CSIRO and joining a private consultancy. This probably diverted me away from academia altogether which is where I otherwise would have gone. With the funding cuts that have hammered Australian research in the last few years, I'm kind of glad about this.

And it was fortunate for everyone else that I quit; I suspect I would have been a pain to work with if I'd stayed, and I'm sure I would have tried (probably unsuccessfully) to push the research in all the wrong directions. I suspect that I might have done such a bad job on the team that they might well have never made any progress to what we now call 802.11b Wi-Fi. On this basis, can I claim that I played a role in the creation of WiFi? By leaving and letting the team hire someone who actually had a clue what they were doing?

I'd like to say that I learn from my mistakes.

Before I got the private consultancy job, I applied for a quant-like role at County Natwest which in the end I turned down (again another lucky save given their history later). I was asked how I thought that County Natwest could make use of the Internet. My answer was that since no-one in their right mind would transfer money over the internet, that all it could be was an information portal.

A decade later (in 2007) I left Google because I was fairly convinced that it was going to fall apart in a few years as Wikipedia became ever more trustworthy that it would become everyone's first point of call for search. It's 2015 now as I search using Google over my home WiFi connection from a proprietary operating system: I have to admit that Bill Gates, Eric Schmidt and Terry Percival were right, and I was wrong.

Based on this, feel free to ignore anything in this blog that you disagree with, since it's almost definitely wrong. But I still think I'm right when I say that my my book of nerd-geek poetry has the best poems about nuclear physics you'll ever see. (And some fun stuff with robots, AI, first contact, and all sorts of other topics. There's even a vampire-at-the-blood-bank.) You really should go and buy it for yourself or your nearest and dearest nerd-geek friends. Here's the Amazon link: When Medusa went on Chatroulette.

Thursday, 29 October 2015

CentOS Data Protector agent unable to be installed

A customer asked me today for some help with a CentOS server that wouldn't install properly, despite everything looking OK.

The session went like this.

# ./omnisetup.sh -server cellmgr.ifost.org.au -install da,autodr
Cannot access the Cell Manager system (inet is not responding)....

As the cell manager was known to be working, we didn't need to check connectivity to 5555 on the cell manager. I suggested just running the installation (without the cell manager import first)

# ./omnisetup.sh -install da,autodr

This worked fine. Was the disk agent listening?

# netstat -an | grep 5555
tcp6   0   0   :::5555 :::*    LISTEN

That's odd: why IPv6? The cell manager had IPv6 disabled, so that would certainly have stopped things working.

# grep FLAGS /etc/xinet.d/omni
FLAGS = IPV6

That's that one explained... use your favourite editor (vi, nano, emacs, gedit...) to set FLAGS = IPV4 if you happen to encounter it. (Don't forget to run service xinetd restart )

But things still weren't working: CentOS has a host-based firewall. As we didn't have a media agent, the only relevant port is tcp 5555.

# firewall-cmd --add-port 5555/tcp --permanent
# firewall-cmd --reload

And then everything worked correctly.

Thanks to Glen Thompson for doing most of the work investigating this one!

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Wednesday, 4 February 2015

Moment-in-time (snapshot) backup of Amazon AWS / EC2 instances

This post is the second in my series on backing up servers in the cloud. (The previous post was here: http://blog.ifost.org.au/2015/01/using-data-protector-to-back-up-your.html.) Obviously, there are some servers in the cloud that you simply won't need to backup, but the remainder which you do need to backup are much harder.

The big problem (which I will address in the next post in this series) is the cost of long-term storage. If you want to maintain backups for seven years, even Amazon Glacier will be absurdly expensive.

 The other problem is the challenge of getting a consistent moment in time backup. Only the very minor cloud infrastructure service providers are offering VMware or Hyper V as their main offering, so virtual disk snapshot backups aren't an option.

On Windows systems there is VSS, so a file system backup taken with a VSS snapshot option will be a reasonably consistent moment in time.

However, most servers run Linux. Many of them are running a database of some sort (PostgreSQL or MySQL). It is possible to arrange a pre-exec to dump the database to disk, but this gets increasingly impractical as the database gets larger.

I wrote a blog article about one method (LVM) to get consistent Linux backups last year (http://blog.ifost.org.au/2014/07/moment-in-time-snapshot-backups-of.html ). But with Amazon  it is possible to use the snapshot capability they provide.

The scripts on this page assume that your instance has one EBS volume that it boots from, and no other attached storage.

Create a backup job that will backup /mnt (even though the server doesn't have a mounted filesystem there). If necessary, just edit the data list:

FILESYSTEM "/mnt" cloud-server1.data-protector.net:"/"
{
}
Then put the following two scripts (snapshot-preexec.sh and snapshot-postexec.sh) into /opt/omni/lbin on the cloud hosted server, and modify the backup job to use these as the pre- and post-exec jobs for the backup.

If the Amazon EC2 tools are not already installed and configured in the environment, you will need to install them. Most of the Amazon-supplied AMIs have these already in-place, but the Redhat-supplied ones don't.

snapshot-preexec.sh

#!/bin/sh

# First, get our instance ID from Amazon
INSTANCE=$(wget -q -O - \
   http://169.254.169.254/latest/dynamic/instance-identity/document \
    | grep instanceId | cut -d'"' -f4)

# Next, find out zone we are in, for the volume creation later
ZONE=$(wget -q -O - \
   http://169.254.169.254/latest/dynamic/instance-identity/document \
    | grep availabilityZone | cut -d'"' -f4)

# This script only works for single volumes at the moment
VOLUME=$(ec2-describe-instances $INSTANCE \
   | grep BLOCKDEVICE | awk '{print $3}' | head -1)

# Flush everything we can out to disk before we take the snapshot
sync
sync
fsfreeze -f /

# Create a snapshot of our root volume
SNAPSHOT=$(ec2-create-snapshot $VOLUME | awk '{print $2}')
until ec2-describe-snapshots $SNAPSHOT | grep -q completed
do
    sleep 1
done

fsfreeze -u /

# Turn that snapshot into a volume
NEWVOL=$(ec2-create-volume --snapshot $SNAPSHOT -z $ZONE| awk '{print $2}')
until ec2-describe-volumes $NEWVOL | grep -q available
do
    sleep 5
done

# Connect that volume
ec2-attach-volume $NEWVOL -i $INSTANCE -d sdf
until ec2-describe-volumes $NEWVOL | grep -q attached
do
    sleep 5
done

# Mount it
mount /dev/xvdf /mnt

# Now we can back up. Remember what we had though
echo $SNAPSHOT > .snapshot-to-remove
echo $NEWVOL > .volume-to-remove


The first couple of lines will only work on the Amazon cloud. The EC2 instance queries a special Amazon address to find out its own details -- its instance id (e.g. i-121255) and its zone (e.g. us-west-2c).

Then we flush as much out to disk as we can (with the sync commands, and then freeze I/O on the root filesystem). Any thing that tries to write to disk will block until after the snapshot is completed. Read operations will still work. We run a busier loop checking to see if the snapshot is ready.

After that, we turn the snapshot into a volume, and attach that volume to a device which will probably be free. There will be an error in dmesg about the lack of a partition table on /dev/xvdf but it doesn't seem to matter.

Finally we mount /mnt (ready to be backed up) and remember what volumes we just created.


snapshot-postexec.sh

#!/bin/sh

SNAPSHOT=$(cat .snapshot-to-remove)
NEWVOL=$(cat .volume-to-remove)

umount /mnt

# Detach the volume and wait until it is gone
ec2-detach-volume  $NEWVOL
while ec2-describe-instances $NEWVOL | grep -q ATTACHMENT
do
    sleep 5
done

ec2-delete-volume $NEWVOL
ec2-delete-snapshot  $SNAPSHOT
After the backup, the post-exec removes the mount, the volume and the snapshot.

I hope you find this helpful.


Greg Baker is one of the world's leading experts on HP Data Protector. His consulting services are at http://www.ifost.org.au/dataprotector . He has written numerous books (see http://www.ifost.org.au/press ) on it, and on other topics. His other interests are startup management, applications of automated image and text analysis and niche software development.

Monday, 19 January 2015

DataProtector and TCP wrappers (libwrap) etc.

While it's rare to run into a system using TCP wrappers rather than a host-based firewall, I ran into one today in the form of the Vsphere vCenter Server Appliance.

To cut a long story short, you can install the Data Protector agent quite happily (I pushed it from my Linux-based installation server which has OB2_SSH_ENABLED=1 by adding my ssh key to the server appliance's .ssh/authorized_keys). But then it can't be imported.

You will see lines like this appear in the appliance's /var/log/messages :

2015-01-19T01:25:06+00:00 app01 xinetd[19865]: libwrap refused connection to omni (libwrap=inet) from ::ffff:192.168.1.14

2015-01-19T01:25:06+00:00 app01 xinetd[19865]: FAIL: omni libwrap from=::ffff:192.168.1.14

It's been so long since I dealt with TCP wrappers that I spent ages remembering what to do. In the end, it's just a matter of putting the following into /etc/hosts.allow

inet: 192.168.1.0/255.255.255.0 : ALLOW

Adjust based on whatever IP ranges and subnets you need to allow. Or use "ALL" instead of 192.168.1.0/255.255.255.0 if you don't have any security concerns.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/press/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Monday, 27 October 2014

Unknown error 1053 starting hpdp-as

As I mentioned in my post about 1053 error for hpdp-idp-cp ( http://blog.ifost.org.au/2014/06/unknown-error-1053-starting-hpdp-idp-cp.html ), error code 1053 on Linux doesn't have any particular meaning. It's just a catch-all to say "something went wrong during service startup".

I've experienced this twice now, and both times it was for different reasons.

Here's the output from omnisv status...

    ProcName      Status  [PID]    
===============================
    crs         : Active  [26506]
    mmd         : Active  [26504]
    kms         : Active  [26505]
    hpdp-idb    : Active  [26466]
    hpdp-idb-cp : Active  [26499]
    hpdp-as     : Down
    omnitrig    : Active
    Sending of traps disabled.
===============================


With a bit of inspired guessing, hpdp-as is supposed to be started by /etc/rc.d/init.d/hpdp-as. Despite what it looks like, this isn't actually used as SYSV init runscript -- it is invoked by /opt/omni/sbin/omnisv start. This in turn is invoked by /etc/rc.d/init.d/omni, which actually is a SYSV init runscript.

/etc/rc.d/init.d/hpdp-as isn't part of any RPM file; neither are any of the other Data Protector start up scripts.


  • /etc/init.d/omni is installed by the OB2-CS post-install scriptlet.
  • /etc/init.d/hpdp-as is created by the IDBsetup.sh script when it calls an internally-defined updateServices function

The first time I encounted "Unknown error 1053", it was simply because something had gone wrong during installation, and /etc/init.d/hpdp-as wasn't created. I just took the code from IDBsetup.sh (search for hpdp-as and you'll find an init script inside a heredoc) and recreated it. Then I checked that the appropriate /etc/services entry had been created ( "hpdp-idb-as 7116/tcp"  )

If this happens to you, here's a handy /etc/init.d/hpdp-as for reference. Just change lnx.ifost.org.au to whatever your cell manager's hostname is:


#!/bin/sh
# chkconfig: 35 99 08
# description: HP Data Protector Application Server.
# processname: hpdp-as

### BEGIN INIT INFO
# Provides: hpdp-as
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 3 5
# Default-Stop: 0 1 2 4 6
# Short-Description: HP Data Protector Application Server
### END INIT INFO

#Defining AS_HOME
AS_HOME=/opt/omni/AppServer

case "$1" in
start)
echo "Starting the HP Data Protector Application Server..."
nohup su - hpdp -c "${AS_HOME}/bin/standalone.sh -b lnx.ifost.org.au &"
;;
quick)
nohup su - hpdp -c "${AS_HOME}/bin/standalone.sh -b lnx.ifost.org.au > /dev/null &"
;;
stop)
echo "Stopping the HP Data Protector Application Server..."
su - hpdp -c "${AS_HOME}/bin/jboss-cli.sh --connect command=:shutdown"
;;
log)
echo "Showing server.log..."
tail -1000f /var/opt/omni/log/AppServer/server.log
;;
*)
echo "Usage: /etc/init.d/hpdp-as {start|stop|log}"
exit 1
;; esac
exit 0



The second time I encountered this (which was today), /etc/init.d/hpdp-as was present. A bit of digging into it revealed that it calls /opt/omni/AppServer/bin/standalone.sh -b cell-manager-hostname as the user hpdp (or whatever you are running Data Protector as). Helpfully, standard output is redirected to /dev/null, so whatever errors you might encounter are not reported anywhere.

When I ran that manually I saw (buried in the standard output which would otherwise have been discarded):

15:35:14,505 ERROR [org.jboss.msc.service.fail] MSC00001: Failed to start service jboss.logging.handler.FILE: org.jboss.msc.service.StartException in service jboss.logging.handler.FILE: java.io.FileNotFoundException: /var/opt/omni/log/AppServer/server.log (Permission denied)

And indeed, /var/opt/omni/log/AppServer is owned by root, and unwritable by hpdp-as.

[hpdp@cellmgr AppServer]$ ls -ld /var/opt/omni/log/AppServer
drwxr-xr-x. 2 root root 21 Sep 16 22:40 /var/opt/omni/log/AppServer

Various other problems crop up, all to do with permissions. As far as I can tell, the following chown command will fix all of them. 
sudo chown hpdp \
  /var/opt/omni/log/AppServer \
  /var/opt/omni/server/AppServer  \
  /opt/omni/AppServer/standalone/deployments  \ 
  /etc/opt/omni/server/AppServer

sudo /opt/omni/sbin/omnisv start


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Wednesday, 1 October 2014

Linux iptables firewalling rules for use with Data Protector

Every client (and the cell manager) needs port 5555 open, unless you've changed the default port for the omniinet service.

Do you have a special "backup" network? If it's accessible on (say) eth1, then
iptables -I INPUT -p tcp -i eth1 --dport 5555 -j ACCEPT
Or, if you want to restrict a client so that it only receives connections from the cell manager (if the cell manager has an IP address of 192.168.200.100:
iptables -I INPUT -p tcp -s 192.168.200.100 --dport 5555 -j ACCEPT
You could get the same effect by adding an only_from parameter in /etc/xinetd.d/omni or by turning on cell security.

If the client also has tape drives (or the robotic control for a tape library) attached then you will need to open up a range of port numbers. Here I've allowed 10 concurrent connections, which would be appropriate for a 9-drive tape library with a robotic controller:

test -e /opt/omni/.omnirc || cp /opt/omni/.omnirc.TMPL /opt/omni/.omnirc
echo OB2PORTRANGESPEC=xMA-NET:18000-18009 >> /opt/omni/.omnirc
for port in 18000 18001 18002 18003 18004 18005 18006 18007 18008 18009
do
  iptables -I INPUT -p tcp --dport $port -j ACCEPT
done

And if you are running the StoreOnce software component on this Linux machine, then you will need ports 9387 and 9388 (unless you have changed them).

iptables -I INPUT -p tcp --dport 9387 -j ACCEPT
iptables -I INPUT -p tcp --dport 9388 -j ACCEPT
Finally, save it for the next reboot:

service iptables save

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Monday, 28 July 2014

Moment-in-time snapshot backups of Linux systems with Data Protector

You can get moment-in-time snapshot backups of Linux systems with Data Protector.

Everybody knows the "Use Shadow Copy" flag for Windows backups gets something very close to a moment-in-time snapshot, but there's a persistent rumour that you can only get a good moment-in-time snapshot of Linux systems if you take a VMware snapshot backup.

If you are backing up a btrfs filesystem (which is supported on the Linux agent: you can see it listed in /opt/omni/.util), then you can use its built-in snapshot capabilities.

But even ext3 and xfs filesystems can do snapshots with the help of the LVM layer. This technique dates back to the early days of HP-UX, but it works on modern Linux boxes (and probably on modern HP-UX boxes as well). You will need to do four steps:

  1. Make sure that you have some spare space in the volume group which contains the volumes you are wanting to back up. Use vgdisplay and look for the lines about Free PE (free physical extents).
  2. Make a backup specification (through the Data Protector GUI if you want to) including all the filesystems you want. Don't tick the host, tick each of the filesystems, even if you want all of them. In the pre-exec and post-exec fields of the filesystem defaults (not the pre-exec and post-exec for the whole backup job) put snapshot-preexec.sh and snapshot-postexec.sh.
  3. Put snapshot-preexec.sh and snapshot-postexec.sh into /opt/omni/lbin
  4. Edit the backup specification. Look for the lines that say FILESYSTEM "/xxx" ... and replace them with FILESYSTEM "/mnt/backup/xxx" ... The end result should look like Linux-Snapshot-Example
Then you can run the backup as per normal.

The snapshot-preexec.sh script looks at the parent process which spawned it (which will be the vbda process) and finds the -volume parameter. Then it strips off the /mnt/backup part of it, and figures out the logical volume that the original filesystem is mounted on. It calls lvcreate --snapshot, runs fsck and then mounts that snapshot volume. So by the time the vbda process starts trying to read the /mnt/backup/xxx filesystem, the filesystem is mounted.

The snapshot-postexec.sh script cleans up these volumes.

There are two environment variables you can set in the backup specification which snapshot-preexec.sh will make use of.
  • SNAPSHOT_PREFIX (which defaults to /mnt/backup)
  • SNAPSHOT_SIZE (which defaults to 10m, which means that 10MB worth of block writes can be happen while the backup runs. For a busy filesystem this might be too little.)
Of course, the backups will be recorded as being of /mnt/backup/... which means that disaster recovery won't work properly (because it will never see your root filesystem as being backed up). It's a pity that there isn't an easy way of updating the internal database to make it think that it was a different filesystem backed up.

All in all, it's not that difficult to do. It only took me about an hour to set up. The mysterious thing is that 17 years after I first implemented this on an HP-UX system -- and I wasn't the first to do this -- it's still not out-of-the-box functionality.


Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Friday, 11 July 2014

Installing Data Protector cell manager on a minimal-install Redhat 7 / Centos 7



This is more of a note to myself, but if you do a "minimal" install of RedHat or Centos, you will be missing a number of important packages.

Here's what I do to fix this, before running omnisetup.sh

echo 'PATH=$PATH:/opt/omni/bin:/opt/omni/sbin:/opt/omni/lbin' \
        > /etc/profile.d/omni.sh
chmod +x /etc/profile.d/omni.sh
. /etc/profile.d/omni.sh

useradd -m hpdp
yum install net-tools bc xinetd glibc.i686 
yum install bind-utils psmisc mlocate telnet
     ;# not really necessary, but so useful...

mkdir -p /etc/opt/omni/server
chmod a+rx /etc/opt/omni/server

Then edit  /etc/man_db.conf and add the following two lines in the appropriate stanzas.
 MANPATH_MAP /opt/omni/bin /opt/omni/lib/man
 MANDB_MAP /opt/omni/lib/man /var/cache/man/omni

You probably won't need a firewall on your Data Protector cell manager. In any case, the installer doesn't add exceptions to the firewalling rules like it does on Windows, so the cell manager can't import itself or start properly.

systemctl stop firewalld
systemctl disable firewalld

Now you can run
  omnisetup.sh -CM -IS -install da,ma,cc,StoreOnceSoftware,autodr

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Monday, 23 June 2014

Unknown error 1053 starting hpdp-idp-cp

Today I was installing HP Data Protector on a Linux cell server. During the installation, I saw an error message as it tried to install the internal database connection pooling process. This is what it said:

ERROR: Unable to Start IDB CP (Return code = 1)For more detail please refer to /var/opt/omni/server/log/DPIDBsetup_5216.logerror: %post(OB2-CS-A.08.10-1.x86_64) scriptlet failed, exit status 3

(The 5216 is a process ID, it changes on each invocation.)

Running omnisv -start produces the delightfully unhelpful

Cannot start "hpdp-idb-cp" service, system error:[1053] Unknown error 1053

Error 1053 seems to be a Windows error message that someone has decided a Linux-based cell-manager needs to be compatible with!

Digging a bit deeper, and running the SYSV / upstart / init start-up script with "/etc/rc.d/init.d/hpdp-idp-cp start" was slightly more helpful:

FATAL Cannot load config filehpdp-idb-cp started

It hadn't actually started, of course, the init script just blindly assumes that it has without checking $? for an error code.

Walking through the init script, there's a line

su hpdp -c "LD_LIBRARY_PATH=/opt/omni/idb/lib:$LD_LIBRARY_PATH /opt/omni/idb/bin/pgbouncer -d /etc/opt/omni/server/idb//hpdp-idb-cp.cfg"

That makes sense, the IDB connection pooler runs as hpdp, and hpdp-idb-cp.cfg is the configuration file which says what port number to connect on, and various other useful parameters.

The file itself was readable, but for some reason the installer failed to set the right permissions on /etc/opt/omni/server (it was unreadable to anyone but root).

So with a quick
chmod a+rx /etc/opt/omni/server

And then I could restart the installation...
./omnisetup.sh ... -IS
See also: unknown errror 1053 starting hpdp-as.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector