Error loading large XSLS file into CMOD

[jeffs42885]

Seeing this strange error when trying to load a test file into our QA system. It is a large excel file and is throwing this 88 record:

arsload: Processing file ><arsload: Load Version <8.5.0.6> Operating System <AIX> <6.1>
arsload: Server Version <8.5.0.6> Operating System <AIX> <6.1> Database <DB2> <09.07.0004>
arsload: 10/08/13 15:42:16 -- Loading started, 97628158 bytes to process
Warning:  A document of size >97628158< bytes was processed.  It is not recommended to store documents in OnDemand greater than 50MB in size.  Although dat
OnDemand Load Id = >46996-14-0-1FAA-0-0<
Unable to allocate enough memory.  File=arsadmp.c, Line=1252
Unable to store the object >1FAAA<.  Object size 97628158
Loaded 0 rows into the database
arsload: 10/08/13 15:42:18 Loading failed
arsload: Processing failed for file ><
arsload: 10/08/13 15:42:18 -- Unloading started
OnDemand UnLoad Successful - LoadId(46996-14-0-1FAA-0-0) Rows Deleted(0)
 
10/08/13 15:42:19 -- Unloading of data was successful


It is loading fine in our DEV environment which which is  the same setup..shared enironment of the object server and the library server. This file isn't that big compared to some files we load..in production. Here are some more errors I am seeing associated with the 88 record.


Application Group Unload Database: Name(EXCEL) Agid(46996) LoadId(1FAA-0-0) Rows Deleted(0) SM UnLoad Ready(1)
Application Group Unload Storage Manager: Name(EXCEL) Agid(46996) NodeName(-CACHE-) Nid(0) Server(-LOCAL-) LoadId(1FAA) Objects Deleted(0)
Application Group Unload Storage Manager: Name(EXCEL) Agid(46996) NodeName(TSMNODE) Nid(14) Server(-LOCAL-) LoadId(1FAA) Objects Deleted(0)

I checked ulimits, /var, paging space..everything looked fine on the server. I tried recreating the application grou and the app from scratch and still the same error. Anyone have any suggestions on what I can look for?

[Justin Derrick]

Check for ulimits on root's account.  This bit a customer of mine, so much so that it ended up in the IBM OnDemand Newsletter. 

Also, I didn't see any messages about compression -- turn on compression to save space.

Also consider bumping up the object size from 10MB to 50 or 100MB so that the file isn't so heavily segmented (would you rather pull back one piece, or 10 pieces?).

-JD.

[jeffs42885]

JD- can you send me the newsletter.. also- should the ulimit matter on arsload since they are not being ran by root, we are running them on our process account?

Also- Segment size..Its set to 10000 across the board even where I can load large files.

[LWagner]

Is loading of excel files now a standard feature in CMOD 9 ?

[Justin Derrick]

You can find all the newsletters here:

http://www-01.ibm.com/support/docview.wss?uid=swg27024130

The article explains why root's ulimit is important if you're starting processes from crontab.

The snag with the object size is that you're trying to load a 97MB object without compression -- each document you load will require that 10 different objects be retrieved in order to 'reconstitute' your document.  Adding a zero at the end cuts the I/O's to complete the retrieval by a factor of 10.  Turning on compression will make this efficient, even if it only cuts the size of the file in half.

-JD.