Unknown error 1053 starting hpdp-as

As I mentioned in my post about 1053 error for hpdp-idp-cp ( http://blog.ifost.org.au/2014/06/unknown-error-1053-starting-hpdp-idp-cp.html ), error code 1053 on Linux doesn't have any particular meaning. It's just a catch-all to say "something went wrong during service startup".

I've experienced this twice now, and both times it was for different reasons.

Here's the output from omnisv status...

    ProcName      Status  [PID]    
===============================
    crs         : Active  [26506]
    mmd         : Active  [26504]
    kms         : Active  [26505]
    hpdp-idb    : Active  [26466]
    hpdp-idb-cp : Active  [26499]
    hpdp-as     : Down
    omnitrig    : Active
    Sending of traps disabled.
===============================

With a bit of inspired guessing, hpdp-as is supposed to be started by /etc/rc.d/init.d/hpdp-as. Despite what it looks like, this isn't actually used as SYSV init runscript -- it is invoked by /opt/omni/sbin/omnisv start. This in turn is invoked by /etc/rc.d/init.d/omni, which actually is a SYSV init runscript.

/etc/rc.d/init.d/hpdp-as isn't part of any RPM file; neither are any of the other Data Protector start up scripts.

• /etc/init.d/omni is installed by the OB2-CS post-install scriptlet.
• /etc/init.d/hpdp-as is created by the IDBsetup.sh script when it calls an internally-defined updateServices function

The first time I encounted "Unknown error 1053", it was simply because something had gone wrong during installation, and /etc/init.d/hpdp-as wasn't created. I just took the code from IDBsetup.sh (search for hpdp-as and you'll find an init script inside a heredoc) and recreated it. Then I checked that the appropriate /etc/services entry had been created ( "hpdp-idb-as 7116/tcp"  )

If this happens to you, here's a handy /etc/init.d/hpdp-as for reference. Just change lnx.ifost.org.au to whatever your cell manager's hostname is:

#!/bin/sh
# chkconfig: 35 99 08
# description: HP Data Protector Application Server.
# processname: hpdp-as

### BEGIN INIT INFO
# Provides: hpdp-as
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 3 5
# Default-Stop: 0 1 2 4 6
# Short-Description: HP Data Protector Application Server
### END INIT INFO

#Defining AS_HOME
AS_HOME=/opt/omni/AppServer

case "$1" in
start)
echo "Starting the HP Data Protector Application Server..."
nohup su - hpdp -c "${AS_HOME}/bin/standalone.sh -b lnx.ifost.org.au &"
;;
quick)
nohup su - hpdp -c "${AS_HOME}/bin/standalone.sh -b lnx.ifost.org.au > /dev/null &"
;;
stop)
echo "Stopping the HP Data Protector Application Server..."
su - hpdp -c "${AS_HOME}/bin/jboss-cli.sh --connect command=:shutdown"
;;
log)
echo "Showing server.log..."
tail -1000f /var/opt/omni/log/AppServer/server.log
;;
*)
echo "Usage: /etc/init.d/hpdp-as {start|stop|log}"
exit 1
;; esac
exit 0

The second time I encountered this (which was today), /etc/init.d/hpdp-as was present. A bit of digging into it revealed that it calls /opt/omni/AppServer/bin/standalone.sh -b cell-manager-hostname as the user hpdp (or whatever you are running Data Protector as). Helpfully, standard output is redirected to /dev/null, so whatever errors you might encounter are not reported anywhere.

When I ran that manually I saw (buried in the standard output which would otherwise have been discarded):

15:35:14,505 ERROR [org.jboss.msc.service.fail] MSC00001: Failed to start service jboss.logging.handler.FILE: org.jboss.msc.service.StartException in service jboss.logging.handler.FILE: java.io.FileNotFoundException: /var/opt/omni/log/AppServer/server.log (Permission denied)

And indeed, /var/opt/omni/log/AppServer is owned by root, and unwritable by hpdp-as.

[hpdp@cellmgr AppServer]$ ls -ld /var/opt/omni/log/AppServer

drwxr-xr-x. 2 root root 21 Sep 16 22:40 /var/opt/omni/log/AppServer

Various other problems crop up, all to do with permissions. As far as I can tell, the following chown command will fix all of them. 

sudo chown hpdp \
  /var/opt/omni/log/AppServer \
  /var/opt/omni/server/AppServer  \
  /opt/omni/AppServer/standalone/deployments  \ 
  /etc/opt/omni/server/AppServer

sudo /opt/omni/sbin/omnisv start

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Unknown error 1053 starting hpdp-idp-cp

Today I was installing HP Data Protector on a Linux cell server. During the installation, I saw an error message as it tried to install the internal database connection pooling process. This is what it said:

ERROR: Unable to Start IDB CP (Return code = 1)For more detail please refer to /var/opt/omni/server/log/DPIDBsetup_5216.logerror: %post(OB2-CS-A.08.10-1.x86_64) scriptlet failed, exit status 3

(The 5216 is a process ID, it changes on each invocation.)

Running omnisv -start produces the delightfully unhelpful

Cannot start "hpdp-idb-cp" service, system error:[1053] Unknown error 1053

Error 1053 seems to be a Windows error message that someone has decided a Linux-based cell-manager needs to be compatible with!

Digging a bit deeper, and running the SYSV / upstart / init start-up script with "/etc/rc.d/init.d/hpdp-idp-cp start" was slightly more helpful:

FATAL Cannot load config filehpdp-idb-cp started

It hadn't actually started, of course, the init script just blindly assumes that it has without checking $? for an error code.

Walking through the init script, there's a line

su hpdp -c "LD_LIBRARY_PATH=/opt/omni/idb/lib:$LD_LIBRARY_PATH /opt/omni/idb/bin/pgbouncer -d /etc/opt/omni/server/idb//hpdp-idb-cp.cfg"

That makes sense, the IDB connection pooler runs as hpdp, and hpdp-idb-cp.cfg is the configuration file which says what port number to connect on, and various other useful parameters.

The file itself was readable, but for some reason the installer failed to set the right permissions on /etc/opt/omni/server (it was unreadable to anyone but root).

So with a quick

chmod a+rx /etc/opt/omni/server

And then I could restart the installation...

./omnisetup.sh ... -IS

See also: unknown errror 1053 starting hpdp-as.

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published book on HP Data Protector (http://x.ifost.org.au/dp-book). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector