Max report size.

[hlim]

Hi there!  Is there a size limit for text (ascii) file report (single report)?  I have a a single report that is about 1.5GB, and it failed to load with a simple load failed message.

[Alessandro Perucchi]

Hello hlim,

I am not aware of any kind of limitation. Maybe you had a memory problem, When CMOD needs to store such big report, then depending on the method you use to store it (ACIF Indexer, Generic Index, ...), the number of Index, how many index are generated from this report might have a huge influence on the memory that CMOD use.

On which platform are you trying to archive this report?

Cheers,
Alessandro

[Justin Derrick]

We need a little more info to be able to help you find the problem...

Platform, versions of software, maybe your index parameters and Application config... 

-JD.

[pankaj.puranik]

Hi

We had a similar problem some time back.
CMOD failed to process large files.

So what was done is we had written an exit to break the file into smaller chunks and then processed it.

I guess if you could ask the upstream systems to send smaller files, it would work.

Thanks
Pankaj.

[hlim]

Here is my setup.

Win2008.
MS SQL 2005.

I'm using generic indexer (I also tried ACIF and verified it failed).  Here is one of my index file value.  As you can see, it has about 250,000 pages, which is about 1.7GB file size report.

FIELD NAMES BEGIN:
cu_name
report_title
report_date
folder_id
pagenum
fileinfo
FIELD NAMES END:
GENERATIONS CU
ACH List
11/04/03
301
250337
10065.0325
C:\arsload\WrgAGC.GCU301G.35370044.0.0.0.ARD.OUT
0
0

[Justin Derrick]

This looks like the old generic index format.  I'm not sure how well supported it is anymore.

Can you cut-and-paste in the error message you get from arsload?  You can probably find it in your System Log.

-JD.

[Alessandro Perucchi]

Here is my setup.

Win2008.
MS SQL 2005.

I'm using generic indexer (I also tried ACIF and verified it failed).  Here is one of my index file value.  As you can see, it has about 250,000 pages, which is about 1.7GB file size report.

FIELD NAMES BEGIN:
cu_name
report_title
report_date
folder_id
pagenum
fileinfo
FIELD NAMES END:
GENERATIONS CU
ACH List
11/04/03
301
250337
10065.0325
C:\arsload\WrgAGC.GCU301G.35370044.0.0.0.ARD.OUT
0
0

Hello hlim,

here is the converted to the new generic index format:

CODEPAGE:850
GROUP_FIELD_NAME:cu_name
GROUP_FIELD_VALUE:GENERATIONS CU
GROUP_FIELD_NAME:report_title
GROUP_FIELD_VALUE:ACH List
GROUP_FIELD_NAME:report_date
GROUP_FIELD_VALUE:11/04/03
GROUP_FIELD_NAME:folder_id
GROUP_FIELD_VALUE:301
GROUP_FIELD_NAME:pagenum
GROUP_FIELD_VALUE:250337
GROUP_FIELD_NAME:fileinfo
GROUP_FIELD_VALUE:10065.0325
GROUP_OFFSET:0
GROUP_LENGTH:0
GROUP_FILENAME:C:\arsload\WrgAGC.GCU301G.35370044.0.0.0.ARD.OUT

Can you try if you see a difference? If yes, then maybe it would be worth to update your process which create this old index format to the new one.

If not, then... we need to look somewhere else... :-(

Cheers,
Alessandro

[hlim]

Sorry for late response.  I tried new index file, but it still fails.  According to IBM, there is size limit for OnDemand MP for Windows.  They also recommand me to divide the report into several files.  Bummer!

[Justin Derrick]

It would certainly make sense to use Large Object Support for this report, and index it with ACIF.  Nobody wants to retrieve or view 250,000 pages at a time.

-JD.

[pankaj.puranik]

Yes, you would probably need to break the input file. As I mentioned in my earlier post we had a similar problem. We had written a shell script and a C program to check the input file size and break it into smaller pieces. But you need to be cautious when doing this as you need to maintain the AFP tags, BNG, TLE, BPG, etc or your ACIF indexer would fail to process the file.

Cheers
Pankaj.