Multiple arrsockd license servers?

[Justin Derrick]

Hi folks...

I've got a REALLY strange one here.  A customer has recently upgraded to CMOD 8.5.0.6, and is seeing a very strange behaviour after just a few hours of CMOD running.  The arssockd process spawns multiple 'child' processes of the 'license' server.  I've never worked on this server before (and wasn't involved in the upgrade) so this behaviour is totally mysterious.  Here's a sample of the process list after just 24 hours:

# ps -ef |grep arssockd
 archive  409676  798936   0   Feb 01      -  0:00 arssockd-dpycserr: license (Cur(135), Max(1000))
 archive  798936       1   0   Feb 01      - 78:57 arssockd-dpycserr: license (Cur(0), Max(1000))
 archive  495936  798936   0   Feb 01      -  0:00 arssockd-dpycserr: license (Cur(127), Max(1000))
 archive  926192  798936   0   Feb 01      -  0:00 arssockd-dpycserr: license (Cur(94), Max(1000))
 archive  995668  798936   0 23:27:59      -  0:00 arssockd-dpycserr: license (Cur(3), Max(1000))
 archive  999766  798936   0   Feb 01      -  0:00 arssockd-dpycserr: license (Cur(129), Max(1000))
 archive 1036594  798936   0   Feb 01      -  0:21 arssockd-dpycserr: cmd
 archive 1065250  798936   0 23:28:04      -  0:00 arssockd-dpycserr: license (Cur(6), Max(1000))
 archive 1089894  798936   0 01:30:31      -  0:00 arssockd-dpycserr: license (Cur(9), Max(1000))

The weirdest part of all is that the server seems to run just fine, despite the weirdness.

These folks are on AIX 5.3, DB2 9.5, and (as mentioned above) CMOD 8.5.0.6.

Any insight would be GREATLY appreciated!

-JD.

[Alessandro Perucchi]

Hello Justin,

I find it quite strange... because since CMOD 8.5, you should see only one entry in the process table like :

Code: [Select]

archive 25362552        1   0   Feb 02      -  2:52 arssockd-ARCHIVE: license (Cur(1), Max(3500))
archive 40960194 25362552   0   Feb 02      -  0:02 arssockd-ARCHIVE: cmd
have you tried something like

Code: [Select]

arssockd -p -I ARCHIVE
??

Is there something in the System Log? Are they using some user exits?
Are the permission ok?
Do you have something in the arserr.log file which is in the ARSTMP directory?
in the errpt maybe?

Sincerely yours,
Alessandro

[Justin Derrick]

Yes, it's very, very strange, and I've never seen anything like it.

I checked the threads, and it reports the results properly, but as I said, it looks like the main thread is creating 'child' processes that appear to be hung.

It happened twice, and since stopping arssockd and restarting it, it hasn't re-appeared in 48 hours.  I'll keep an eye on it, but if anyone else sees something similar, please let me know, so we can get this bug squashed!

Thanks Alessandro!

-JD.

[Alessandro Perucchi]

I've an idea... don't know if it could help...

I've had in the past some semaphor and shared memory problems.
So if you stop CMOD and DB2 (I assume that they are using DB2), then if you run the command:

Code: [Select]

ipcs |grep archive

You should have nothing... if you find something then you can do something like

Code: [Select]

ipcs |awk '/archive/ {print "ipcrm -"$1" "$2}' |ksh

to remove all semaphore and shared memory allocation.

Then restart DB2 and CMOD, and check if that helped a bit...

Sincerely yours,
Alessandro

[Steve Bechtolt]

We had this same issue and it ended up being a file permissions issue on the 'arslog' user exit script.

[Justin Derrick]

Yup, my customer confirmed that it was related to logging -- I think they had started CMOD under the archive ID, but the owner of the arslog exit was root -- and the archive user didn't have execute permission, so it failed at the OS level, causing strange behaviour in arssockd.