Unable to allocate enough memory / cannot connect

[jeffs42885]

Hello all.

Last night, we experienced this error message with files loading:

arsload: 02/09/10 14:34:07 -- Loading started, 442073904 bytes to process
Unable to allocate enough memory. File=arslacif.c, Line=711
Unable to read from offset 0 for length 442073904 from the file: inputFile.out
Loaded 0 rows into the database
arsload: 02/09/10 14:34:07 Loading failed
arsload: Processing failed for file >inputFile<
arsload: Processing has stopped. The remaining files will NOT be processed.

This happened a total of 11 times, followed by:

Connection cannot be established for the >ARCHIVE< server

Once this occured, our ARSLOAD process terminated. Between the first memory error, and the cannot connect error, there was 87 records.

I verified proper ulimit settings for the process id that runs arsload, also
made sure that there was nothing going on with the server (AIX 5.3) and that
the database was okay (DB2 9.1)

Has anyone experienced occurances like this, and what routes did you take to fix this.

[kbsiva]

You might want to check what other process was running on that time. Usually when we have issues like that there are other process that are running and chewing up resources. An SA should be able to look in the syslog or tell what other processes were running on that time. Hope this helps.

[jeffs42885]

At the time, we had our SA run a topas command. Nothing was out of the ordinary, resources were ok

[Justin Derrick]

I've recently run into this issue as well, although it wasn't arslacif.c that bombed -- it was arsadmp.c.

I'm on CMOD 8.5.0.5 on AIX 6.1 TL07 with DB2 9.7.

Any hints on how to resolve it?

[jeffs42885]

Hi JD

We had this happen a few times and as per PMR, it was a sporatic issue and the fix was to upgrade to 8.5.0.5

[demaya]

If you have any problems please upgrade to the newest version...

 

Anybody else here that hate this sentence?!?!

[Justin Derrick]

Hi JD

We had this happen a few times and as per PMR, it was a sporatic issue and the fix was to upgrade to 8.5.0.5

Heh!  We're on 8.5.0.5... 

*bangs head on desk*

-JD.

[Alessandro Perucchi]

JD,

as Mayach said with his lovely new sentence :

you have any problems please upgrade to the newest version...

Maybe with 8.5.0.6 is will solve the problem! :-D

Joke aside, maybe that's stupid and probably you've already tried.
Have you check the ulimit? Filesystems full? errpt ?

Sincerely yours,
Alessandro

[johnnoel]

Ok.. this is taking a stab in the dark here.......

ARSLOAD for z/OS and naturally AIX, uses a temporary folder on the Unix System (MVS in the z/OS case.) It creates temporary files in a location that you configure it to use and those temporary files are supposed to be deleted after loading. However, in some instances they are not. Later you try to load a report with a different ID and the system tries to create a temporary file with the same name as one already existing and it can't delete/overwrite it.

I know you are thinking "what does this have to do with out of memory error?" I don't trust that message you see there are the real root cause, but a mere side effect.

You need to delete all temp files from this location.

It will be the value of the ARS_TMP variable. In our case it is set as follows:

ARS_TMP=/fs/arstemp/KAHP/arstemp/tmp

If you don't have permissions to this folder, you will have to get the Unix Admin to delete them for you.

This was the message we saw (just like you)

FILE>       KAHXGD.KAH$GDBN.JOB32664.D0000106.?,2012.291,23:27:07:60       <FILE
arsload: Using /fs/arstemp/KAHP/arstemp/tmp for temporary files                
arsload: Processing file >/fs/arstmp/KAHP/tmp/kah16777584 (DD:OBJINPT-GD.D00H001
arsload: 10/17/12 23:26:55 -- Loading started, --UNKNOWN-- bytes to process    
arsload: 10/17/12 23:26:56 Loading failed                                      
arsload: Processing failed for file >/fs/arstmp/KAHP/tmp/kah16777584 (DD:OBJINPT
Unable to allocate enough memory.  File=arsadmin.c, Line=791                    
arsload: Unable to log load information                                        
arsload: Processing has stopped.  The remaining files will NOT be processed.    



When we dug deeper, we found this..

ICH408I USER(KAHXGD  ) GROUP(KAHONDMD) NAME(PROD APPLID VIA E
  /fs/arstemp/KAHP/arstemp/tmp/16777584.arsload.env          
  CL(FSOBJ   ) FID(01D2C1C8C6E2F90001050229C7640000)        
  INSUFFICIENT AUTHORITY TO OPEN                            
  ACCESS INTENT(-W-)  ACCESS ALLOWED(GROUP      R--)        
  EFFECTIVE UID(0000025172)  EFFECTIVE GID(0000001034)

Deleted those previously created temp files and load ran succesfully.

[ali.arsiwala]

We ran into the same issue too and the above as suggested by johnnoel worked for us. Although we had to raise a PMR and after a long delay and lot of haggling with IBM support, they were able to find the temporary files and asked us to get them removed. Loads started working fine after removing those temp files.

[pankaj.puranik]

We are on AIX.
I had theis issue today.
And surprisingly, I got the following error when I tried loading a zero byte file.

But when I ran the same command with a file having data, it loaded fine

arsload: Processing file >/cna1/cfsc/SystemLogMixed/11796736_systemLogMixedIND_132-23-2345-23-45-65<
arsload: Load Version <8.5.0.5> Operating System <AIX> <6.1>
arsload: Server Version <8.5.0.5> Operating System <AIX> <6.1> Database <DB2> <09.07.0005>
arsload: 12/13/12 07:15:59 -- Loading started, --UNKNOWN-- bytes to process
Unable to allocate enough memory.  File=arslacif.c, Line=1767
Loaded 0 rows into the database
arsload: 12/13/12 07:16:04 Loading failed
arsload: Processing failed for file >/cna1/cfsc/SystemLogMixed/11796736_systemLogMixedIND_132-23-2345-23-45-65<
arsload: Processing has stopped.  The remaining files will NOT be processed.

Cheers
Pankaj.

[Justin Derrick]

Yeah, the issue was resolved for us when we cleaned out our /arstmp directory.

-JD.

[jeffs42885]

Heres a question---

Is it a good idea to use a folder different than /tmp for this?

[Justin Derrick]

Well, I think the rationale for using a separate /arstmp directory is something like...

• Keeps I/O for the application out of your OS partition.
• An error won't fill up /tmp -- space in /tmp is critical to the OS.
• Vice-versa:  Another user might fill /tmp, and interrupt your load processing.
• You might be able to put /arstmp on faster, more reliable disks.
• You can blow away ALL the contents of /arstmp without worrying about disturbing another app/process/task.
• You can restrict access to /arstmp for security purposes.

That's all I can think of off the top of my head...  Anyone got any others?

[Alessandro Perucchi]

I have nothing to say from what Justin just said.

The only thing is...

NEVER USE THE /TMP FOR ANYTHING. LEAVE IT TO THE SYSTEM!!!

If you want a /tmp, then use one for you... (/var/tmp, /arstmp, /my/precious/tmp, /my/own/tmp, /my/love/my/precious/tmp, ... but NOT /tmp).
There are system (Solaris) which use /tmp as part of their RAM, so if /tmp is full, then your server is dead...

Sincerely yours,
Alessandro