ARS1109E Unable to allocate enough memory

[ibmarthin]

Hi out there

We face this problem more often - and have a PMR open for it.
But I just want to hear if anyone have seen it for OnDemand 9.5.0.6.
I can't figure what kind of "Memory" that is short. The /tmp is not the problem. The ARSLOAD is running with REGION=0K

2019-05-30 00:29:17.305779: ARS1109E Unable to allocate enough memory. File=SYS16180.T182004.RA000.ARNLOAD.SRCCMS.H01(A     
RNLOAD), Line=6311
2019-05-30 00:29:17.307399: ARS1157E Unable to store the object >3918<.  Object size 306
2019-05-30 00:29:17.307413: ARS4311E Loading failed                                                                           
2019-05-30 00:29:17.307969: ARS4318E Processing failed for file >/usr/lpp/ars1/DB2N/tmp//16777874.0000016B06256813

(DD:INPUT-SYS19150.T022836.RA000.PFLSD888.DCFOD.H01)<
ARS1109E Unable to allocate enough memory.  File=SYS16180.T182004.RA000.ARNLOAD.SRCCMS.H01(ARNLOAD), Line=15163               
ARS4323E Unable to log load information
2019-05-30 00:29:17.324308: ARS4327E Processing has stopped.  The remaining files will NOT be processed. 

And today we even get memory shortage on READ:

30-05-2019 12:16:27   ODREADX1       2041440   Error   No       24   Object >13115< in Application Group >FAA< not found in node >DB2G.VEAVTALE.ODSG155.MCOBJ155 <  Srvr->server.name 10.181.23.1 non-SSL<-
30-05-2019 12:16:27   ODREADX1       2041440   Error   No      430   OAM Error: ARSSMMVS line 1188: 00000015 13451599 bytes (EDC5132I Not enough memory. (errno2=0x00000000)), Return Code=21, Reason=0, File=arssmsms.cpp, Line=1893  Srvr->tServer_name 10.181.23.1 non-SSL<-

I I xheck ulimit, it looks ok to me:
Å ulimit -a                   
core file         unlimited   
cpu time          3602         
data size         unlimited   
file size         unlimited   
stack size        unlimited   
file descriptors  65535       
address space     75240k       
memory above bar  2048m       

/regards
Marthin

[Greg Ira]

Can you check the PROC for your CMOD server on z/OS (ARSSOCKD)?  What is the region size set there?

[ibmarthin]

Hi

REGION=512M

Embarrassing I did'nt check that first.
I will change this to REGION=0M ASAP

Thanks Greg 

[ibmarthin]

After changed REGION to 0M we still face the problem.
So for now both the ARSSOCKD task and the ARSLOAD step has REGION=0.

[Ed_Arnold]

> ARS1109E Unable to allocate enough memory.  File=SYS16180.T182004.RA000.ARNLOAD.SRCCMS.H01(ARNLOAD)

Hmmm - just a guess, but that's a work file.  Is there enough space on disk for that file?

Ed

[Ed_Arnold]

>ARS1157E Unable to store the object >3918<.  Object size 306

Is the "306" the line number in the z/OS console log or that number is truncated?  What is the full text of that line (might have wrapped)?
__________

> The /tmp is not the problem.

Is /tmp an HFS or a zFS?  If still an HFS, convert to zFS.  I have seen in my deep past HFS's getting corrupted and lying about how much free space they have.  In other words, you might do a df -kPt /tmp and it'll say 1 meg available, but 1 meg isn't available, only a much smaller number of bytes.
__________

If nothing else, it might be time for a "therapeutic IPL" of the LPAR.

Ed

[ibmarthin]

Hi
/tmp is zFS.
and 306  "Object size 306" is, yes, Object size 306.

I think we wait for solution from PMR.

But thanks for the interrest anyway.
/ Marthin

[scottnys]

have you looking into this exit on your LPAR?  We have gotten bit with it before.

IEFUSI — Step Initiation Exit

z/OS MVS Installation Exits
SA23-1381-00


Topics for This Exit Appear as Follows:

Comparing IEFUSI with IEALIMIT when Limiting Region Size
Defining the Exit in SMFPRMxx
Controlling the Exit Routine Through the Dynamic Exits Facility
Exit Routine Environment
Exit Recovery
Exit Routine Processing
Programming Considerations
Entry Specifications
Registers at Entry
Parameter Descriptions
Return Specifications
Registers at Exit
Examples
Example of Using IEFUSI to Limit Region Size
Examples of Storage Allocations Based on Values Set by IEFUSI

[Ed_Arnold]

And you might want to take a look at this ARS1109E technote:

https://www-01.ibm.com/support/docview.wss?uid=swg27041871&aid=1

Ed