OnDemand User Group

Support Forums => Report Indexing => Topic started by: kasim on May 30, 2014, 06:08:42 AM

Title: Alternate of arsload
Post by: kasim on May 30, 2014, 06:08:42 AM: hi all

i have a requirement of some custom utility that can index and load data to cmod without using arsload.
anybody have idea on this. please share your experience.

thanks
Title: Re: Alternate of arsload
Post by: Justin Derrick on May 31, 2014, 01:52:12 PM: It doesn't really exist. arsload is how you load data into CMOD. You could use arsdoc add, but that's not the same.

What are your requirements and how is arsload not filling those requirements?
Title: Re: Alternate of arsload
Post by: kasim on June 03, 2014, 01:33:12 AM: Hi Justin

thanks for reply

actually we want to develop a custom utility that can work same as arsload but faster then arsload. I want to know whether anybody worked on this type activity in past.

will it be feasible to develop a custom utility using Java , C or any other language.

thanks
Title: Re: Alternate of arsload
Post by: Justin Derrick on June 03, 2014, 04:04:06 AM: If you have a performance issue, your best bet is to open a PMR and have support & development work on it.

I've got customers ingesting a terabyte a day with time to spare -- so I'd be looking at your system configuration before I'd undertake re-writing proprietary software.

Can you share some numbers with us? How much data are you trying to load, how long is it taking? How long do you feel it should take?

-JD.
Title: Re: Alternate of arsload
Post by: jeffs42885 on June 03, 2014, 06:34:04 AM: I have seen ARSLOAD load very large PDF and line data files very quickly, more inparticular 45000-50000 page AFP statements that were transformed from either AFP2PDF or LCDS2PDF and then loaded via ARSLOAD. What kind of files are these?
Title: Re: Alternate of arsload
Post by: kasim on June 03, 2014, 09:05:46 PM: Hi JD , Jeff

We have simple files. pdf, text , xls etc. ARSLOAD is absolutely fine.

I was asked by my superior to look into the possibility that will work faster then ARSLOAD . we have 200 TB data to be migrated to CMOD. for example ARSLOAD will take 2-3 minutes to load 1 GB file. Can we reduce this time to 30 to 40 seconds in any way. so that we can ingest 200 TB in half time.

My questions are -
Is there any third party developed tool that can work faster then arsload?
Is there any possibility that we can develop some utility that can work faster then arsload.

thanks
Kasim
Title: Re: Alternate of arsload
Post by: Justin Derrick on June 04, 2014, 03:13:11 AM: You have a configuration issue, or need bigger/better hardware if you're taking 1 hour to load 1GB. I have customers who routinely load 1GB files (including indexing, compression, etc.) in about a minute.

If you can't easily expand your CMOD server's hardware, use a separate system with CMOD installed as a 'load server' to outsource all the indexing & compression to a system with lots of CPU & RAM.

-JD.
Title: Re: Alternate of arsload
Post by: jeffs42885 on June 04, 2014, 05:22:58 AM: I agree with justin. I've ran LCDS data through the following steps:

Infoprint XT
Ricoh AVE
AFP2PDF
ARSLOAD

in about half an hour for a 3GB file
Title: Re: Alternate of arsload
Post by: kasim on June 05, 2014, 09:12:54 PM: Hi JD & Jeff

Actually i am ok with asload its working absolutely fine as expected. But still we are looking something better :) if possible. may be someone has done this ???????
Title: Re: Alternate of arsload
Post by: jeffs42885 on June 06, 2014, 02:03:32 PM: It might not be possible but try throwing more CPU or memory at it.

I've ingested 1+GB AFP files with overlays, fonts, other resources, etc (40-50k pages) in about 45 minutes and this is on OLD hardware running 8.5.0.6, you must be hitting a bottleneck somewhere.
Title: Re: Alternate of arsload
Post by: Justin Derrick on June 09, 2014, 04:31:24 AM: kasim...

To answer the question you keep asking, no, there is no alternate file loading tool that is either publicly available or has been developed for sale by a third party that I'm aware of. All tools that work with CMOD that I've worked with rely on arsload for loading data into CMOD.

As Jeff has mentioned, you're hitting a wall somewhere else. You can either increase the RAM / CPUs available on your existing server, or use multiple 'load' servers running the CMOD software to increase throughput.

You may also have the option of pre-indexing (arsload option '-i') and then running arsload at a later date to simply compress the data and load it into the database. You may also be able to run several streams in parallel.

-JD.
Title: Re: Alternate of arsload
Post by: Stephen McNulty on June 09, 2014, 07:26:24 AM: Kasim, I think you answered your own question

"Actually i am ok with asload its working absolutely fine as expected. But still we are looking something better Smiley if possible. may be someone has done this HuhHuh? "

I have not heard of anyone trying to invest in making it better, as it works today. Better spending time and dollars on the other areas already mentioned in this thread.
Title: Re: Alternate of arsload
Post by: kasim on June 11, 2014, 10:02:41 PM: Hi Stephen McNulty

thanks for reply. :)
Title: Re: Alternate of arsload
Post by: ewirtz on June 11, 2014, 10:45:49 PM: Hi Kasim,
just a remark. ARSLOADS calls OAM or TSM. Most of the work during storing the document is done by OAM or TSM. If you only look for the load phase, I don't see any chance to be faster. But if you look for the indexing phase you will be much faster if you are able to use a generic indexer.

regards Egon