Author Topic: Loading large CSV using Generic indexer  (Read 2802 times)

DDP021

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Loading large CSV using Generic indexer
« on: March 04, 2021, 07:35:49 AM »
We loaded a large test CSV file to CMOD.  File loaded successfully although the 87 message contained  this information:

Loading started, 570696197 bytes to process
Warning:  A document of size >570696197< bytes was processed.  It is not recommended to store documents in OnDemand greater than 50MB in size.  Although data may
OnDemand Load Id = >90353-10-0-1FAA-0-0-90354<
Loaded 1 rows into the database
Document compression type used - OD77.  Bytes Stored = >44760080< Rows = >1<
Loading completed

When the users attempted to view in Content Navigator they received this, FILE NOT LOADED COMPLETELY

Spreadsheet contained 1196248 rows but they were only able to see 1048576.

Does anyone know if CMOD has a limit to size it can ingest?




jsquizz

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 573
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #1 on: March 04, 2021, 12:54:40 PM »
What version of CMOD?
#CMOD #DB2 #AFP2PDF #TSM #AIX #RHEL #AWS #AZURE #GCP #EVERYTHING

DDP021

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #2 on: March 04, 2021, 06:20:06 PM »
10.1.0.5

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2228
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: Loading large CSV using Generic indexer
« Reply #3 on: March 05, 2021, 06:12:17 AM »
What is your Application Configuration in CMOD?  Are you treating this document as line data?  You may have exceeded a 'number of lines per page' limit, since it appears to cut off at 210.

It's probably best to treat this as a user defined data type, so that OnDemand treats it as a binary file.

-JD.
   
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

DDP021

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #4 on: March 05, 2021, 06:23:04 AM »
Justin,

We do have it defined as USER DEFINED (CSV).  We did increase the Max Rows on the Application Group to 20,000,000 and having them retry sending file again.

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2228
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: Loading large CSV using Generic indexer
« Reply #5 on: March 05, 2021, 07:30:21 AM »
Hi.  Max rows is for the database table, not documents. 

I'd open a ticket with IBM to see which limitation you're running into -- I haven't seen this specific issue previously.

Is there any chance you could break up this CSV file somehow?  If it's truly archival (i.e., end users won't access it), you could pre-compress it with .zip or .gz, and store it as a binary.

-JD.
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

DDP021

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #6 on: March 05, 2021, 07:38:52 AM »
Agree..changing the max rows made no difference.  After loading file was not able to view all rows in CSV.  We are going back to application to see if they are able to split the file.  If they are resistant, we will have to go to IBM.  As always, appreciate the quick response!!   Take care

Dave

jsquizz

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 573
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #7 on: March 05, 2021, 08:22:10 AM »
Agree..changing the max rows made no difference.  After loading file was not able to view all rows in CSV.  We are going back to application to see if they are able to split the file.  If they are resistant, we will have to go to IBM.  As always, appreciate the quick response!!   Take care

Dave

Since you mention that..

I've seen this before, Granted- IT was a very VERY old CMOD system.. Maybe 8.5?

Our business partners were sending CSV files with like, 200-300k rows. They wanted to pickup something on each row of the file. CMOD would get to like the ..10th to last row and just quit. PMR didn't resolvge anything. We just told them to send it as smaller files.
#CMOD #DB2 #AFP2PDF #TSM #AIX #RHEL #AWS #AZURE #GCP #EVERYTHING

DDP021

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #8 on: March 05, 2021, 08:27:21 AM »
Thanks for the reply!...Not going to bother opening a PMR for this.   Application end is looking to splitting this up in 2 pieces. 

« Last Edit: March 05, 2021, 03:52:28 PM by jsquizz »

jsquizz

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 573
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #9 on: March 05, 2021, 03:52:38 PM »
Thanks for the reply!...Not going to bother opening a PMR for this.   Application end is looking to splitting this up in 2 pieces.

Ironic. I'm trying to open up a very large data dump from oracle with excel, it's a .del file with maybe 500k rows? Excel isnt liking it. I got the same error message you got.

Sounds to me like it's probably hitting some kind of resource limit on local PC, memory or cpu..or something
#CMOD #DB2 #AFP2PDF #TSM #AIX #RHEL #AWS #AZURE #GCP #EVERYTHING

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2228
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

DDP021

  • Sr. Member
  • ****
  • Posts: 343
    • View Profile
Re: Loading large CSV using Generic indexer
« Reply #11 on: March 07, 2021, 05:02:50 PM »
Good information to know!!!!….Especially if the application comes back to us blaming CMOD!!!  I appreciate it!!

Take care