OnDemand User Group
Support Forums => MP Server => Topic started by: jeffs42885 on February 17, 2012, 09:25:07 AM
-
Hello all.
Last night, we experienced this error message with files loading:
arsload: 02/09/10 14:34:07 -- Loading started, 442073904 bytes to process
Unable to allocate enough memory. File=arslacif.c, Line=711
Unable to read from offset 0 for length 442073904 from the file: inputFile.out
Loaded 0 rows into the database
arsload: 02/09/10 14:34:07 Loading failed
arsload: Processing failed for file >inputFile<
arsload: Processing has stopped. The remaining files will NOT be processed.
This happened a total of 11 times, followed by:
Connection cannot be established for the >ARCHIVE< server
Once this occured, our ARSLOAD process terminated. Between the first memory error, and the cannot connect error, there was 87 records.
I verified proper ulimit settings for the process id that runs arsload, also
made sure that there was nothing going on with the server (AIX 5.3) and that
the database was okay (DB2 9.1)
Has anyone experienced occurances like this, and what routes did you take to fix this.
-
You might want to check what other process was running on that time. Usually when we have issues like that there are other process that are running and chewing up resources. An SA should be able to look in the syslog or tell what other processes were running on that time. Hope this helps.
-
At the time, we had our SA run a topas command. Nothing was out of the ordinary, resources were ok
-
I've recently run into this issue as well, although it wasn't arslacif.c that bombed -- it was arsadmp.c.
I'm on CMOD 8.5.0.5 on AIX 6.1 TL07 with DB2 9.7.
Any hints on how to resolve it?
-
Hi JD
We had this happen a few times and as per PMR, it was a sporatic issue and the fix was to upgrade to 8.5.0.5
-
If you have any problems please upgrade to the newest version...
???
Anybody else here that hate this sentence?!?!
-
Hi JD
We had this happen a few times and as per PMR, it was a sporatic issue and the fix was to upgrade to 8.5.0.5
Heh! We're on 8.5.0.5... :D
*bangs head on desk*
-JD.
-
JD,
as Mayach said with his lovely new sentence :
you have any problems please upgrade to the newest version...
Maybe with 8.5.0.6 is will solve the problem! :-D
Joke aside, maybe that's stupid and probably you've already tried.
Have you check the ulimit? Filesystems full? errpt ?
Sincerely yours,
Alessandro
-
Ok.. this is taking a stab in the dark here.......
ARSLOAD for z/OS and naturally AIX, uses a temporary folder on the Unix System (MVS in the z/OS case.) It creates temporary files in a location that you configure it to use and those temporary files are supposed to be deleted after loading. However, in some instances they are not. Later you try to load a report with a different ID and the system tries to create a temporary file with the same name as one already existing and it can't delete/overwrite it.
I know you are thinking "what does this have to do with out of memory error?" I don't trust that message you see there are the real root cause, but a mere side effect.
You need to delete all temp files from this location.
It will be the value of the ARS_TMP variable. In our case it is set as follows:
ARS_TMP=/fs/arstemp/KAHP/arstemp/tmp
If you don't have permissions to this folder, you will have to get the Unix Admin to delete them for you.
This was the message we saw (just like you)
FILE> KAHXGD.KAH$GDBN.JOB32664.D0000106.?,2012.291,23:27:07:60 <FILE
arsload: Using /fs/arstemp/KAHP/arstemp/tmp for temporary files
arsload: Processing file >/fs/arstmp/KAHP/tmp/kah16777584 (DD:OBJINPT-GD.D00H001
arsload: 10/17/12 23:26:55 -- Loading started, --UNKNOWN-- bytes to process
arsload: 10/17/12 23:26:56 Loading failed
arsload: Processing failed for file >/fs/arstmp/KAHP/tmp/kah16777584 (DD:OBJINPT
Unable to allocate enough memory. File=arsadmin.c, Line=791
arsload: Unable to log load information
arsload: Processing has stopped. The remaining files will NOT be processed.
When we dug deeper, we found this..
ICH408I USER(KAHXGD ) GROUP(KAHONDMD) NAME(PROD APPLID VIA E
/fs/arstemp/KAHP/arstemp/tmp/16777584.arsload.env
CL(FSOBJ ) FID(01D2C1C8C6E2F90001050229C7640000)
INSUFFICIENT AUTHORITY TO OPEN
ACCESS INTENT(-W-) ACCESS ALLOWED(GROUP R--)
EFFECTIVE UID(0000025172) EFFECTIVE GID(0000001034)
Deleted those previously created temp files and load ran succesfully.
-
We ran into the same issue too and the above as suggested by johnnoel worked for us. Although we had to raise a PMR and after a long delay and lot of haggling with IBM support, they were able to find the temporary files and asked us to get them removed. Loads started working fine after removing those temp files.
-
We are on AIX.
I had theis issue today.
And surprisingly, I got the following error when I tried loading a zero byte file.
But when I ran the same command with a file having data, it loaded fine :)
arsload: Processing file >/cna1/cfsc/SystemLogMixed/11796736_systemLogMixedIND_132-23-2345-23-45-65<
arsload: Load Version <8.5.0.5> Operating System <AIX> <6.1>
arsload: Server Version <8.5.0.5> Operating System <AIX> <6.1> Database <DB2> <09.07.0005>
arsload: 12/13/12 07:15:59 -- Loading started, --UNKNOWN-- bytes to process
Unable to allocate enough memory. File=arslacif.c, Line=1767
Loaded 0 rows into the database
arsload: 12/13/12 07:16:04 Loading failed
arsload: Processing failed for file >/cna1/cfsc/SystemLogMixed/11796736_systemLogMixedIND_132-23-2345-23-45-65<
arsload: Processing has stopped. The remaining files will NOT be processed.
Cheers
Pankaj.
-
Yeah, the issue was resolved for us when we cleaned out our /arstmp directory.
-JD.
-
Heres a question---
Is it a good idea to use a folder different than /tmp for this?
-
Well, I think the rationale for using a separate /arstmp directory is something like...
- Keeps I/O for the application out of your OS partition.
- An error won't fill up /tmp -- space in /tmp is critical to the OS.
- Vice-versa: Another user might fill /tmp, and interrupt your load processing.
- You might be able to put /arstmp on faster, more reliable disks.
- You can blow away ALL the contents of /arstmp without worrying about disturbing another app/process/task.
- You can restrict access to /arstmp for security purposes.
That's all I can think of off the top of my head... Anyone got any others?
-
I have nothing to say from what Justin just said.
The only thing is...
NEVER USE THE /TMP FOR ANYTHING. LEAVE IT TO THE SYSTEM!!!
If you want a /tmp, then use one for you... (/var/tmp, /arstmp, /my/precious/tmp, /my/own/tmp, /my/love/my/precious/tmp, ... but NOT /tmp).
There are system (Solaris) which use /tmp as part of their RAM, so if /tmp is full, then your server is dead...
Sincerely yours,
Alessandro
-
This might make you laugh...
The 'Unable to allocate enough memory' error came back to haunt me this week.
After going through the configuration and limits on the main CMOD admin ID, it was revealed to me by the support team that there was a DIFFERENT ACCOUNT that the production support team was using for loading.
The problem? The ulimit was set too low for the user account that the production support team used to monitor (and correct) load failures. On AIX, user limits are set in the file /etc/security/limits. The documentation for CMOD recommends removing all limits (ie, set the value of the various flags to -1).
You can view the limits set on your current account with "ulimit -a" (to check the 'soft' limits), or "ulimit -Ha" (to check the 'hard' limits). Soft limits can be raised to the value of the (usually higher) hard limit at the command line, but the solution should be to have your UNIX admin change the limits file.
Good luck!
-JD.
-
We are on AIX.
I had theis issue today.
And surprisingly, I got the following error when I tried loading a zero byte file.
But when I ran the same command with a file having data, it loaded fine :)
arsload: Processing file >/cna1/cfsc/SystemLogMixed/11796736_systemLogMixedIND_132-23-2345-23-45-65<
arsload: Load Version <8.5.0.5> Operating System <AIX> <6.1>
arsload: Server Version <8.5.0.5> Operating System <AIX> <6.1> Database <DB2> <09.07.0005>
arsload: 12/13/12 07:15:59 -- Loading started, --UNKNOWN-- bytes to process
Unable to allocate enough memory. File=arslacif.c, Line=1767
Loaded 0 rows into the database
arsload: 12/13/12 07:16:04 Loading failed
arsload: Processing failed for file >/cna1/cfsc/SystemLogMixed/11796736_systemLogMixedIND_132-23-2345-23-45-65<
arsload: Processing has stopped. The remaining files will NOT be processed.
Cheers
Pankaj.
Hello Pankaj,
Yep :-) CMOD is unable to store a zero byte file! :-) it must be at least 1 byte! :-)
Sincerely yours,
Alessandro
-
On my z/OS system, I have the following temps set up:
ARS_TMP=/ars/V850/tmp
ARS_PRINT_PATH=/ars/tfs_for_ars_print
The print tmp is actually pointing to a TFS - filesystem in memory - and it works great.
I'm tempted to change ARS_TMP to also be a TFS and have it reinitialized after every IPL.
Does anybody out there ever need to save their tmp over an IPL?
Ed
-
I realize this thread is over 5 years old... But this recently bit me again, and I felt the need to re-iterate...
When your system administrators set up the CMOD user account (the one that runs arssockd) that user account should have VERY HIGH, or preferably UNLIMITED access to memory -- 25% of your server's physical RAM at a minimum, and no less than 75% of physical if your administrators insist on a limit.