Author Topic: Max File Size in CMOD 8.5  (Read 3775 times)

stephen.arnott

  • Newbie
  • *
  • Posts: 4
    • View Profile
Max File Size in CMOD 8.5
« on: January 17, 2013, 06:54:53 AM »
Hi All

Is the an actual physical file size limit for ingestion to a "standard" CMOD 8.5 Archive on Solaris 10 / Oracle 11g.

There is an IBM technote that quotes 2GB for files and 4GB for PDF, but several people have mentioned that this is not in fact a hard / physical limit.

Can anyone clarify this issue?

I currently have a project which woudl like to archive DB backup files where single files could be in excess of 40GB to a max of 150GB (upper limit highly unlikely to be reached)

Any help / pointers would be gratefully recieved

Kind Regards
Stephen Arnott

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2230
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: Max File Size in CMOD 8.5
« Reply #1 on: January 17, 2013, 08:49:10 PM »
Hi Stephen.

I'm not sure this is really a defined use case for CMOD -- especially because it breaks loaded files into many little pieces.  However, I have many customers that load 'BLOBs' (Binary Large OBjects) into CMOD.

There's no harm in trying this out and reporting back to us with your findings.  Here's a few things I can think of off the top of my head:

I've loaded many extremely large files (32GB uncompressed) into CMOD, and not had problems -- that having been said, CMOD compressed the documents down to under 4GB.  That 32GB file contained tens of thousands of documents, so each file was still handled in the way CMOD was designed to hold them:  Split into documents, compressed, and packaged into 10MB pieces.

You'll want to increase the object size dramatically.  I'm not sure what the upper limit for this value is (Default is 10,000K, or 10MB) but you should make this as large as CMOD will accept.

You'll want to pre-compress the file with another utility (bzip2 comes to mind because it's open source, multiplatform, handles large files, and compresses very well) and set CMOD's compression setting to 'Disable'.

All in all, it sounds like this would be better handled directly by TSM.  TSM is built to manage objects like this (backups, archives, etc.), keep them in their original format, provide compression on the fly with the built-in hardware compression chips in modern tape drives, and migrate the data from one device to another.

Let us know what your research uncovers.
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

pankaj.puranik

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Max File Size in CMOD 8.5
« Reply #2 on: January 21, 2013, 09:01:31 AM »
In the past we had issues loading PDFs above 4 GB.
THe IBM documentation also mentions this.

See this link that clearly mentions this.
http://www-01.ibm.com/support/docview.wss?uid=swg21170676


QuestionWhat is the maximum input file and document size IBM? Content Manager OnDemand supports?

AnswerWhen the input file is indexed, it is usually broken up into smaller documents called "groups" that are concatenated together in the .out file. Each group is represented by a set of indexes stored in the database; the group and indexes together is known as a single logical document.

The limit on the input file size is determined by the file system of the operating system, with the exception of PDF. The limit of the input file size is 4 GB for PDF.

A single group (including large objects) cannot be larger than 2 GB. This is true for all data types.

The .out file can be as large as the file system, memory, and operating system permit.

IBM recommends that the size of a PDF input file not exceed 500 MB.

As a best practice, IBM recommends that a single group should not be larger than 50 MB.

If Xenos? is used, the script file must contain the following line at the beginning of the script to support files greater than 4 GB: numeric digits 12