Cmmunication failure

[balaji]

Could some one help me on this problem, we are getting frequent messages from the server like
"ARS1106E Connection cannot be established for the >xxxxxxx.xxx.xxx.com< server . "

 When ever we got this message 1) we are able to connect to ondemand client and server and able to see logs
                                                2) we are able to connect the servers using putty and activity also normal
                                                3) some times we are getting load failures also like below

2019-05-25 05:00:14.486213: ARS1106E Connection cannot be established for the >SERVER_NAME< server
2019-05-25 05:00:14.486389: ARS1157E Unable to store the object >354894GAAA<.  Object size 10178
2019-05-25 05:00:14.486595: ARS4311E Loading failed
2019-05-25 05:00:16.645219: ARS4318E Processing failed for file >FILE_NAME<
2019-05-25 05:00:17.071489: ARS4327E Processing has stopped.  The remaining files will NOT be processed.
2019-05-25 05:00:07.055679: ARS4315I Processing file >file name<
2019-05-25 05:00:07.055806: ARS4334I Load Version <9.5.0.10>  Operating System <Linux> <#1 SMP Wed Feb 27 09:13:29 EST 2019.2.6.18-437.el5>  OS Userid <USRID>  Install Location </opt/app/workload/ibm/ondemand/V9.5/> Data(unlimited KB) Stack(10240 KB) Core(0 512-blocks) Cpu(unlimited seconds) File(unlimited 512-blocks) Nofiles(16384) Threads(0) Processes(515643)
2019-05-25 05:00:07.055854: ARS4335I Server Version <9.5.0.10>  Operating System <Linux> <#1 SMP Tue Jan 22 17:25:23 EST 2019.2.6.32-754.11.1.el6.x86_64>  Database <DB2> <10.05.0010>
2019-05-25 05:00:07.061983: ARS4339I Application Group >APPLICATION GROUP<
2019-05-25 05:00:07.062041: ARS4340I Application >APPLICATION<
2019-05-25 05:00:07.062078: ARS4341I Storage Set >Object Server<
2019-05-25 05:00:07.062113: ARS4342I Storage Node >CACHE_OBJECT<
2019-05-25 05:00:07.062241: ARS4312I Loading started, --UNKNOWN-- bytes to process
2019-05-25 05:00:07.155297: ARS1144I OnDemand Load Id = >LOAD_ID<
2019-05-25 05:00:14.486534: ARS1146I Loaded 0 rows into the database
2019-05-25 05:00:14.486653: ARS4320I Unloading started
2019-05-25 05:00:16.645146: ARS4321I Unloading of data was successful

Not sure what's the issue, how we need to be get out of it ? what would be the possible reasons?

[jsquizz]

I would start with a complete system trace, and go from there.

What is your server setup? Single library server? multiple Object single Library Server? Local DB2 database on the library server?

[balaji]

Yeah , its single library server , object server and DB2 on the library server.

Thanks

[jsquizz]

I saw this issue about 5 years ago, and it was actually a memory leak with our version of Xenos. Does it happen intermittently? Can you load other files?

I would first check your uLimits, I believe stack needs to be set to unlimited, no files a minimum of 791339. Can't seem to locate the IBM requirements- but that's two that jumped out at me that I remember setting

[balaji]

Yeah , its happening intermittently, files are loading to CMOD is continuous process, loads should trigger every 10 mints ...
suddenly we got the connection failure error then 4 - 5 reports fail to load, then automatically it recovered and loads will happen to CMOD
then after some time we got the same connection issue and loads are failing, there is no specific gap between failures ...

[Ed_Arnold]

Upgrade to 9.5.0.12 to see if the problem continues?

As my nephew the veterinarian says, "Couldn't hurt.  Might help."

Ed

[balaji]

@jsquizz

I would start with a complete system trace, and go from there.

can we use netstat command to trace the network stability, and how to trace DB2 to CMOD server connection hiccups