Alternate of arsload

[kasim]

hi all

i have a requirement of some custom utility that can index and load data to cmod without using arsload.
anybody have idea on this. please share your experience.

thanks

[Justin Derrick]

It doesn't really exist.  arsload is how you load data into CMOD.  You could use arsdoc add, but that's not the same.

What are your requirements and how is arsload not filling those requirements?

[kasim]

Hi Justin

thanks for reply

actually we want to develop a custom utility that can work same as arsload but faster then arsload. I want to know whether anybody worked on this type activity in past.

will it be feasible to develop a custom utility using Java , C or any other language.

thanks

[Justin Derrick]

If you have a performance issue, your best bet is to open a PMR and have support & development work on it.

I've got customers ingesting a terabyte a day with time to spare -- so I'd be looking at your system configuration before I'd undertake re-writing proprietary software.

Can you share some numbers with us?  How much data are you trying to load, how long is it taking? How long do you feel it should take?

-JD.

[jeffs42885]

I have seen ARSLOAD load very large PDF and line data files very quickly, more inparticular 45000-50000 page AFP statements that were transformed from either AFP2PDF or LCDS2PDF and then loaded via ARSLOAD. What kind of files are these?

[kasim]

Hi JD , Jeff

We have simple files. pdf, text , xls etc. ARSLOAD is absolutely fine.

I was asked by my superior to look into the possibility that will work faster then ARSLOAD . we have 200 TB data to be migrated to CMOD.  for example ARSLOAD will take 2-3 minutes to load 1 GB file. Can we reduce this time to 30 to 40 seconds in any way. so that we can ingest 200 TB in half time.

My questions are -
Is there any third party developed tool that can work faster then arsload?
Is there any possibility that we can develop some utility that can work faster then arsload.

thanks
Kasim
 

[Justin Derrick]

You have a configuration issue, or need bigger/better hardware if you're taking 1 hour to load 1GB.  I have customers who routinely load 1GB files (including indexing, compression, etc.) in about a minute.

If you can't easily expand your CMOD server's hardware, use a separate system with CMOD installed as a 'load server' to outsource all the indexing & compression to a system with lots of CPU & RAM.

-JD.

[jeffs42885]

I agree with justin. I've ran LCDS data through the following steps:

Infoprint XT
Ricoh AVE
AFP2PDF
ARSLOAD

in about half an hour for a 3GB file

[kasim]

Hi JD & Jeff


Actually i am ok with asload its working  absolutely fine as expected. But still we are looking something better if possible. may be someone has done this ?

[jeffs42885]

It might not be possible but try throwing more CPU or memory at it.

I've ingested 1+GB AFP files with overlays, fonts, other resources, etc (40-50k pages) in about 45 minutes and this is on OLD hardware running 8.5.0.6, you must be hitting a bottleneck somewhere.

[Justin Derrick]

kasim...

To answer the question you keep asking, no, there is no alternate file loading tool that is either publicly available or has been developed for sale by a third party that I'm aware of.  All tools that work with CMOD that I've worked with rely on arsload for loading data into CMOD.

As Jeff has mentioned, you're hitting a wall somewhere else.  You can either increase the RAM / CPUs available on your existing server, or use multiple 'load' servers running the CMOD software to increase throughput.

You may also have the option of pre-indexing (arsload option '-i') and then running arsload at a later date to simply compress the data and load it into the database.  You may also be able to run several streams in parallel.

-JD.

[Stephen McNulty]

Kasim,  I think you answered your own question

"Actually i am ok with asload its working  absolutely fine as expected. But still we are looking something better Smiley if possible. may be someone has done this HuhHuh? "

I have not heard of anyone trying to invest in making it better, as it works today.  Better spending time and dollars on the other areas already mentioned in this thread.

[kasim]

Hi Stephen McNulty

thanks for reply. 

[ewirtz]

Hi Kasim,
just a remark. ARSLOADS calls OAM or TSM. Most of the work during storing the document is done by OAM or TSM. If you only look for the load phase, I don't see any chance to be faster. But if you look for the indexing phase you will be much faster if you are able to use a generic indexer.

regards Egon