arssockd not starting on New RHEL Server

[Joe Wolken]

I am installing a new OnDemand v.10.5 server on RHEL v.7.9 with an Oracle v.19 database. There is no TSM or SSL requirements. We plan to use LDAP but do not have this configured yet.

The software seems to be successfully installed and the arsdb command ran successfully to create an instance called ondq (this will be the QA Server).  I have been able to run the commands to create the System Log and System Load files.  I think this next step is to try to start the server job.  When I run either of these commands I get NO response.  I just get returned quickly with a command prompt:

./arssockd ondq arssockd
or
arssockd -S -I ondq

I can not find any new log files being created with any information to go on.

I feel like I'm missing something simple??  Any ideas?

Thanks,
Joe

[rjrussel]

Edit the arslog script (opt/ibm/ondemand/v10.5/bin directory) add something like

echo $@ >> /tmp/arslog.out

This should pass the startup error to the file /tmp/arslog.out


That should give you the info to solve the problem.

Note: I am doing this from memory but I think the syntax is correct.

-RR

[Ed_Arnold]

Here's an article I published in the CMOD Newsletter.

Linux 101 – How to debug when ARSSOCKD fails to start and no messages are produced
 
 
What usually causes this is either ARSSOCKD ends before it can write to the System Log or the console is not set up correctly.
 
Before doing anything else ensure that Db2 is up.
 
If that doesn’t resolve the problem capture the Linux system trace info thusly:
 
strace -s 256 -f /full/path/to/arssockd -I MyInstance -S 2> strace.out
 
Browse strace.out.  Look for “arslog”
 
Often behind “arslog” will be a CMOD message with the error.
 
Here’s an example from a recent problem record (yes, the line is very long in the trace output)
 
[pid  1927] execve("/opt/ibm/ondemand/V10.1/bin/arslog", ["/opt/ibm/ondemand/V10.1/bin/arslog", "ARCHIVE", "2021-06-08 19:41:32.980988", "0", "ARSSOCKD", "", "4", "213", "ARS0213I Unable to load the OnDemand DB2 dynamic load library (arsdb2).  Check to make sure DB2 is installed and that the db2ln command has been run", ""], [/* 74 vars */] <unfinished ...>
 
Note the ARS0213I message which in this case also gave the problem resolution:
 
ARS0213I Unable to load the OnDemand DB2 dynamic load library (arsdb2). Check to make sure DB2 is installed and that the db2ln command has been run
 
If that doesn’t resolve the issue do the following:
 
You can use the "ldd" command to check if there are other dependency issues with arssockd.
 
Example:  As the userid that would start arssockd, run "ldd /opt/ibm/ondemand/V10.1/bin/arssockd "
 
If at this point there are still issues then open up a problem record with support.

[jsquizz]

I am installing a new OnDemand v.10.5 server on RHEL v.7.9 with an Oracle v.19 database. There is no TSM or SSL requirements. We plan to use LDAP but do not have this configured yet.

The software seems to be successfully installed and the arsdb command ran successfully to create an instance called ondq (this will be the QA Server).  I have been able to run the commands to create the System Log and System Load files.  I think this next step is to try to start the server job.  When I run either of these commands I get NO response.  I just get returned quickly with a command prompt:

./arssockd ondq arssockd
or
arssockd -S -I ondq

I can not find any new log files being created with any information to go on.

I feel like I'm missing something simple??  Any ideas?

Thanks,
Joe

One thing I do, and I am not sure if its the "IBM suggested" approach, is just use trace. that has worked most if not all of the time on numerous rhel/aix systems-

1) Edit /opt/ibm/ondemand/V10.5/config/trace.settings, I usually turn it all on, TRACE_LEVELS=ALL=15,   TRACE_SETTINGS=/somewhereWithSpace/ondqa.trace
2) Edit /opt/ibm/ondemand/V10.5/config/ars.cfg = TRACE_SETTINGS=trace.settings (Make sure you comment this out when done, requires arssockd restart)
3) Start arssockd, let it fail.
4) Execute /opt/ibm/ondemand/V10.5/bin/arstfmt -i /somewhereWithSpace/ondqa.trace -o /somewhereWithSpace/readable_trace.out
5) execute more /somewhereWithSpace/readable_trace.out | grep ERR (Or whatever way to search the file..)

Most of the time its because I forgot to make a setting change or something. Such as,

setting ars.cfg to = ARS_STORAGE_MANAGER=CACHE_ONLY (i think default is TSM, or- If using new api's for azure/aws/etc.. ARS_STORAGE_MANAGER=NO_TSM)
if you're using cache, you didnt define a file system in /opt/ibm/ondemand/V10.5/config/ars.cache, or - odqa/archive user can't read/write that file system

Curious as to what the culprit was.

[Joe Wolken]

Thanks everyone for the very helpful posts.

I actually set up the strace suggested by Ed.  It quickly became clear that when setting permissions on the various directories we overlooked the /Cache directory.  Once we updated the permissions per the OnDemand v.10.5 Installation manual the Instance started right up without issue.

It might be nice in a future release to get this information in a message back from the arssockd command instead of relying on a strace.

Thanks again,
Joe