arssockd server job starts and terminates immediately

[Joe Wolken]

We have a new RHEL Server with CMOD v.10.5 and FP2 installed.  The system seems to have been working without issue.

Then we need to stop the server using:   arssockd -I ondp -T
We confirmed the job had terminated:   arssockd -I ondp -p

When we go to restart the server job:  arssockd -I ondp -S

We get the message:  ARS0351I The arssockd instance ONDP is ready
BUT; we then almost immediately get another message:  ARS0352I The arssockd instance ONDP is terminating

Are we doing something wrong?  Has anyone else experienced this?  Does anyone know why this would happen or where we can look for the cause? 

We are looking throughout the V10.5 directory and the ARS_TMP directory and do not see anything that looks like a log file or error file.

Ideas?

Thanks,
Joe

[Ed_Arnold]

Joe - have you tried this?

Here's an article I published in the CMOD Newsletter.

Linux 101 – How to debug when ARSSOCKD fails to start and no messages are produced
 
 
What usually causes this is either ARSSOCKD ends before it can write to the System Log or the console is not set up correctly.
 
Before doing anything else ensure that Db2 is up.
 
If that doesn’t resolve the problem capture the Linux system trace info thusly:
 
strace -s 256 -f /full/path/to/arssockd -I MyInstance -S 2> strace.out
 
Browse strace.out.  Look for “arslog”
 
Often behind “arslog” will be a CMOD message with the error.
 
Here’s an example from a recent problem record (yes, the line is very long in the trace output)
 
[pid  1927] execve("/opt/ibm/ondemand/V10.1/bin/arslog", ["/opt/ibm/ondemand/V10.1/bin/arslog", "ARCHIVE", "2021-06-08 19:41:32.980988", "0", "ARSSOCKD", "", "4", "213", "ARS0213I Unable to load the OnDemand DB2 dynamic load library (arsdb2).  Check to make sure DB2 is installed and that the db2ln command has been run", ""], [/* 74 vars */] <unfinished ...>
 
Note the ARS0213I message which in this case also gave the problem resolution:
 
ARS0213I Unable to load the OnDemand DB2 dynamic load library (arsdb2). Check to make sure DB2 is installed and that the db2ln command has been run
 
If that doesn’t resolve the issue do the following:
 
You can use the "ldd" command to check if there are other dependency issues with arssockd.
 
Example:  As the userid that would start arssockd, run "ldd /opt/ibm/ondemand/V10.1/bin/arssockd "
 
If at this point there are still issues then open up a problem record with support.

Ed

[Joe Wolken]

Thank you Ed.  This is useful information and we will look into creating the trace that you recommend.

Some added detail:  We are using Oracle for the DB.  The instance was up and running, apparently without issue.  We manually terminated the instance and then tried to start it again.  This is when we are unable to restart it.  I'm not an Oracle guy, but I would think that if the Oracle database was up and running before we terminated the instance then it should still be rup and ready for us to restart the instance.

Also, the only way we've found to restart the instance is to completely reboot the server.  Then the instance restarts without issue.  We should not have to reboot the server...

Thanks,
Joe