ARSSOCKD Crash when calling TSM.

[jsquizz]

Have a strange issue here on one of my old development servers. I am starting to think it's some kind of communication error..

First, here is my out of date "update in progress" system. Everything is on the same box. I do have a TSM 7.1 server configured in the dsm.sys file..

AIX 6.1
DB2 9.7
CMOD 8.5.0.6
TSM Client - Command Line Administrative Interface - Version 6, Release 2, Level 0.0
TSM Server - Server Version 5, Release 5, Level 7.0

Heres what does work:

-Loading to cache
-Retrieves from CACHE
-Loading from this server, to another tier's CMOD server, into THAT CMOD servers Cache/TSM (Same versions..)
-Creating/Updating/Exporting objects in CMOD
-Creating new dev classes / pools / nodes in TSM

I guess you could say the basics. Here is what does not work

-Loading into TSM, either the 5.5 server that is local, or even a remote 7.1 server
-ARSMAINT cache migration (-m)
-All of the objects in TSM are orphaned apparently...I have not tried dsmc retrieve.

arsload: 01/30/19 20:16:09 -- Loading started, 201946 bytes to process
Resource FILE.ARD.res will be added as resource >4-63-0<.  Compression Type(OD77) Original Size(13937709) Compressed Size(8041010)
Connection cannot be established for the >dev< server
Unable to store the object >4<.  Object size 8041010
arsload: 01/30/19 20:16:10 Loading failed
arsload: Processing failed for file >FILE.ARD<
Connection cannot be established for the >dev< server

From the TSM activity log when I load a file or run arsmaint, and arssockd crashes-

01/30/19   20:16:10      ANR0480W Session 10 for node MULTIOD (ONDEMAND) terminated
                                   - connection with client severed. (SESSION: 10)

ARSSOCK DEBUG-   Wed Jan 30 21:15:13 2019: OnDemand(3082) -> connect 1445, 127.0.0.1 errno = 79 rc = -1

Those are the only two messages that I currently have. Anyone have suggestions? I am pretty sure I checked everything at this point!

Thanks in advance.

[jsquizz]

Found a core.

---------------------------------------------------------------------------
LABEL:          CORE_DUMP
IDENTIFIER:     A924A5FC

Date/Time:       Wed Jan 30 22:19:01 EST 2019
Sequence Number: 2772919
Machine Id:      00C298F24C00
Node Id:         server
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSPROC         

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

   Recommended Actions
   CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

   Recommended Actions
   RERUN THE APPLICATION PROGRAM
   IF PROBLEM PERSISTS THEN DO THE FOLLOWING
   CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
          11
USER'S PROCESS ID:
              27066496
FILE SYSTEM SERIAL NUMBER
           5
INODE NUMBER
           0        4098
CORE FILE NAME
/tmp/ars_crash/core
PROGRAM NAME
arssockd
STACK EXECUTION DISABLED
           0
COME FROM ADDRESS REGISTER
??
PROCESSOR ID
  hw_fru_id: 0
  hw_cpu_id: 0

ADDITIONAL INFORMATION
ADSM_Make 15C
ADSM_Make 150
remove__7 E78
ArcSMS_De 330
ArcSMS_St D1C
ArcCSSMP_ 5D4
ArcCSSM_O 100
ArcCSP_SM 1A4
ArcXPORT_ 1C88
ArcSERVP_ 348
ArcSERVP_ 380
ArcSERVP_ 114
_pthread_ F4
??

Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/arssockd SIG/11 FLDS/ADSM_Make VALU/15c FLDS/ArcSMS_De

[Justin Derrick]

Well, a return code 79 on AIX is 'Connection Refused'.  So...  firewall?

Of course, I'd recommend less time figuring this out, and more time doing a 'pick and place' upgrade, where you back up the database and restore it to a newer version, and export the node data from TSM, and sync the caches to a new machine...  Trying to debug decade-old software will just be an exercise in futility.

Alessandro wrote a somewhat complete guide on doing these a while back.

-JD.

[jsquizz]

Well, a return code 79 on AIX is 'Connection Refused'.  So...  firewall?

Of course, I'd recommend less time figuring this out, and more time doing a 'pick and place' upgrade, where you back up the database and restore it to a newer version, and export the node data from TSM, and sync the caches to a new machine...  Trying to debug decade-old software will just be an exercise in futility.

Alessandro wrote a somewhat complete guide on doing these a while back.

-JD.

That may work for this environment. I don't even need the database or even the TSM node data. After looking at the occupancy of the nodes, The objects  There has not been a load to our TSM staging system in like 6-7 years. I could basically start from scratch, do an ARSXML export from QA, Setup some device classes, and arsdoc get from our test system directly into this box. 

This staging system is an absolute disaster

[Alessandro Perucchi]

Well putting aside the server is too old, if there was no connection with TSM in X years, then maybe the whole configuration with TSM is not correct anymore, not the correct port/server, etc...

Have you tried to check the dsm.sys and dsm.opt configuration? If they are still correct?

[jsquizz]

Well putting aside the server is too old, if there was no connection with TSM in X years, then maybe the whole configuration with TSM is not correct anymore, not the correct port/server, etc...

Have you tried to check the dsm.sys and dsm.opt configuration? If they are still correct?

I compared every CMOD / TSM setting to a working environment, including all dsm.sys/opt and they are all the same. My other environment works fine.

[jsquizz]

Update on my issue- It's more than TSM/ARSLOAD

I tried unloading a document that's loaded to cache-

jeff@server:/home1/jeff> arsadmin unload -u admin -p ondemand -h archive -g MULTI -L 6141-64-0-2FAA-17451-17451
LoadId matches existing LoadId in System Log
Connection cannot be established for the >archive< server
Unable to unload data from OnDemand - LoadId(6141-64-0-2FAA-17451-17451) Rows Deleted(0)

I'm def going to move on from this. We are moving from AIX to Linux and since this is a very low dev environment, and I already have a Linux box in place, I am probably going to extract data from my QA system and do my testing there. Thanks all for the guidance, I never like "starting over"..I usually like to play and figure things out, but since we are in a time crunch, yeah that's my best bet.

It's also good to see that others have that mindset too! Thanks again!