OnDemand User Group

Support Forums => MP Server => Topic started by: jfgonzalez on November 19, 2019, 02:41:30 PM

Title: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jfgonzalez on November 19, 2019, 02:41:30 PM
Hello,

We have 3 identical application groups, same compression ratio, similar structure, only cache storage, similar number of documents and none of them have segment tables.

All of them with one associated application. 2 of the applications takes 5 seconds to retrieve each document and the other application takes 15 seconds (This one is the more noticeable), it doesn’t matter if it’s the web client, thick client or recovery from arsdoc.

Currently we’re extracting the reports from one environment to another, using arsdoc get with the recovery by LoadID.
There’re around 4 millions of documents each application, so you can imagine how long it takes to retrieve the documents.

In the 3 application groups the search fields are set to filter.

What would you recommend to enhance the retrieval time?

Thanks in advance.
Title: Re: arsdoc retrieve too slow
Post by: jsquizz on November 20, 2019, 11:12:47 AM
What kind of data is it? PDF? AFP?
Title: Re: arsdoc retrieve too slow
Post by: jfgonzalez on November 20, 2019, 12:22:16 PM
The 3 application are PDF files
Title: Re: arsdoc retrieve too slow
Post by: jsquizz on November 20, 2019, 01:29:53 PM
I figured.

Are all of the PDF input files exactly structured the same? Here's possible scenario I am thinking

Application ABC with app group XYZ retrieves fast, because its a simple pdf, no colors, images, not a lot of resources
Application DEF with app group LMO retrieves slow, because they are complex pdfs with lots of colors, possibly lots of resources.

Here's a good start-

Retrieve a "Fast aka normal" retrieval via ars doc, and a "Slow one"...Note the times. Then, use the system log to find the original load time and see if they were also quick to load or slow to load.

I would imagine there would be some kind of correlation.
Title: Re: arsdoc retrieve too slow
Post by: jfgonzalez on November 20, 2019, 02:55:24 PM
The interesting thing about that is..

The application group ABC and XYZ have even more colors and images in the PDF (the faster one)
And the one that is taking longer, is only text lines.

I'll check the system load like you said.
Title: Re: arsdoc retrieve too slow
Post by: jsquizz on November 21, 2019, 09:36:25 AM
The interesting thing about that is..

The application group ABC and XYZ have even more colors and images in the PDF (the faster one)
And the one that is taking longer, is only text lines.

I'll check the system load like you said.

Could be dependent on the generating system, pdf version, etc. I've seen that happen before.
Title: Re: arsdoc retrieve too slow
Post by: Ed_Arnold on November 21, 2019, 12:40:15 PM
Are there funky fonts in the slow retrievals?

If I recall correctly, there's a standard set of fonts that's always available with PDFs.

If you use other than those standard fonts then they have to be archived as resources, recalled, mapped, converted, sliced and diced so that they can be displayed.

Ed
Title: Re: arsdoc retrieve too slow
Post by: jsquizz on November 21, 2019, 01:35:12 PM
Are there funky fonts in the slow retrievals?

If I recall correctly, there's a standard set of fonts that's always available with PDFs.

If you use other than those standard fonts then they have to be archived as resources, recalled, mapped, converted, sliced and diced so that they can be displayed.

Ed

That is where I was going. :)
Title: Re: arsdoc retrieve too slow
Post by: jfgonzalez on November 21, 2019, 06:04:52 PM
Hello. thanks for the response

All the PDFs use the Standard fonts, none of the funky fonts.

I'll post some images to explain more detailed the problem

Ok, the first image (IMG1) this one uploads fast and the retraival time is fast as well

The second image (IMG2) this one uploaded slow (around 25 seconds) but the retreival time is relative fast

And the last image (IMG3) is the one taking long, uploading (25 seconds) and slow retrieving (25 seconds) (As i mentioned, no funky fonts nor images or colored text.)

Just to clarify, all the documents are already uploaded, so the only thing that matter right now is how to retrieve them.

Thanks for the help


Title: Re: arsdoc retrieve too slow
Post by: Justin Derrick on November 21, 2019, 07:51:49 PM
You haven't given us any information on your environment - which version of CMOD, your OS & version, and database engine and version.  All versions should include the fixpack level.

Providing this information in your original post is the best way to help us to help you!  :)

-JD.
Title: Re: arsdoc retrieve too slow
Post by: jfgonzalez on November 22, 2019, 08:01:45 AM
You haven't given us any information on your environment - which version of CMOD, your OS & version, and database engine and version.  All versions should include the fixpack level.

Providing this information in your original post is the best way to help us to help you!  :)

-JD.

Hello Justin,

Here is the information on the environment


If you need any more information let me know

thank you in advance.

Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jsquizz on November 22, 2019, 08:43:58 AM
Are the PDF's that are taking forever older loads?
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jfgonzalez on November 22, 2019, 09:11:44 AM
Are the PDF's that are taking forever older loads?

No. They were loaded at similiar time
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: Lars Bencze on November 25, 2019, 09:31:41 AM
Just a detail; Your "H" column is measuring "Rows per second", so a higher number here is better/faster.
The IMG1 seems to be the slowest, it only indexes 5-7 rows per second, while the other two handle around 25-30 rows per second.
It is not "seconds per row", it's the other way araound.

I haven't seen anything else that looks suspicious in your Images. Yet, at least.
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: Lars Bencze on November 26, 2019, 04:15:15 AM
I have another question.
You mentioned that you are exporting from one environment, and these images (IMG1-3) are from when you load them back into the new environment, correct?
It looks like you are loading them using, hmm, the "Graphical PDF Indexer", correct? Or are you using "Internal indexing" (PPDs)?

The load process spends HUGE amounts of time indexing. Therefore, one way to speed up the load would be to use the -g flag during export (arsdoc get ... -g = Create generic indexer file), and then to use the flags -X G (="Find a generic indexing file") during arsload.
From what I can see, the PDF Indexer (which I assume that you are using) does not manage to compress the PDF documents during load. They actually wind up a bit BIGGER than they came, which is not unusual, at least not for smaller files. So you will not lose anything in size by using Generic Indexing instead of PDF Indexing.

WHICH, brings me to the next possible issue. Seeing that OnDemand is unable to compress the multi-document files to any degree makes me suspect that the PDF files may have "PDF Compression" turned on. This is a known factor that slows down OnDemand indexing a lot. Sometimes to a factor of 50 or 100.
Now, since these files are already created and you are just moving them, I guess you can't re-build them. But for future files, talk to the guys creating these PDF files and ask them to turn PDF Compression COMPLETELY OFF. If I recall Bud Paton correct, that is "Level 0 PDF Compression". Yes, the files will be much larger at delivery, but OnDemand should process them a huge lot faster.

I hope that either or both of these tips may help you!
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jfgonzalez on November 29, 2019, 05:10:54 PM
You mentioned that you are exporting from one environment, and these images (IMG1-3) are from when you load them back into the new environment, correct?

No.  The images (IMG1-3) are from the original loads to the "old" environment

It looks like you are loading them using, hmm, the "Graphical PDF Indexer", correct? Or are you using "Internal indexing" (PPDs)?

Yes, in the original load we used the Graphical PDF Indexer.

The load process spends HUGE amounts of time indexing. Therefore, one way to speed up the load would be to use the -g flag during export (arsdoc get ... -g = Create generic indexer file), and then to use the flags -X G (="Find a generic indexing file") during arsload.

Yes, the files were loaded with pdf indexer originally, but to retrieve them from the original environment we're using the generic indexer (with -g flag). To load on the new environment we're using the flag -X -G.

I don't know if I create a file with the generic indexer it has to do something with the type of compression in the PDF file that originally loaded. Is there a way to change the type of compression once is already loaded? the development team says it's  an OD77 compression

Sorry for the delay
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jfgonzalez on March 24, 2020, 11:17:09 AM
Hi,

Sorry for refloating the thread, but we still have the problem.

We have detected that the arsdump program varies too much in the execution times, it does not matter if it is from the ICN, the Windows client or the arsdoc.

Search is fast for all applications, queries in DB apparently respond the same way for all applications. But it takes almost 3 times longer when recovery is invoked with the arsdump program (from ICN, Windows Client, arsdoc).

What can we do to investigate the cause of this? Is there a way to modify the execution parameters of arsdump?
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jsquizz on March 24, 2020, 01:37:17 PM
Hi,

Sorry for refloating the thread, but we still have the problem.

We have detected that the arsdump program varies too much in the execution times, it does not matter if it is from the ICN, the Windows client or the arsdoc.

Search is fast for all applications, queries in DB apparently respond the same way for all applications. But it takes almost 3 times longer when recovery is invoked with the arsdump program (from ICN, Windows Client, arsdoc).

What can we do to investigate the cause of this? Is there a way to modify the execution parameters of arsdump?

Anything else happening in your system logs when these retrieves are slow?
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jfgonzalez on March 26, 2020, 04:28:08 AM
Hi jsquizz,

Nothing really is observed, only the exaggerated recovery time for the documents for the application group PAN_EstadoCuentaRetail.

I attached the file from SystemLog where "Message" column can be filtered by:
PAN_EstadoCtaTC (fast retrival time)
CRI_EstadoCtaTC (fast retrival time)

PAN_EstadoCuentaRetail (slow retrival time)

thanks in advance!
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: Justin Derrick on March 26, 2020, 05:18:33 AM
Just out of curiousity, is Tivoli Storage Manager or Spectrum Protect installed on the server?

You can provide the output of: 

Code: [Select]
lslpp -l |grep -i tsm
and

Code: [Select]
lslpp -l |grep -i tivoli
To show us which TSM / Tivoli products are installed on your AIX server where CMOD is running.

-JD.
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jsquizz on March 26, 2020, 05:52:21 AM
Hi jsquizz,

Nothing really is observed, only the exaggerated recovery time for the documents for the application group PAN_EstadoCuentaRetail.

I attached the file from SystemLog where "Message" column can be filtered by:
PAN_EstadoCtaTC (fast retrival time)
CRI_EstadoCtaTC (fast retrival time)

PAN_EstadoCuentaRetail (slow retrival time)

thanks in advance!

Was looking down the path of excessive logging, an issue I've seen bite two clients in the past and cause tons of issues. Looks like you're not doing it and your system log is clean.

I removed the attachment because it had IP addresses listed, please remove them next time prior to uploading/posting any log messages. Thanks :)
Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: jfgonzalez on March 27, 2020, 01:13:41 PM
Hi derrick

If apparently it is installed, I attach the images.

It is worth mentioning that we are only using Cache storage.

I have done tests to show with the command "more", the physical files at the operating system level (they obviously appear strange characters because they are PDF) and I have no problems with any.

Regards.

Title: Re: arsdoc retrieve too slow AIX 7.1 CMOD 9.5 DB10.5
Post by: Justin Derrick on March 30, 2020, 06:53:51 AM
Okay, so you're using TSM - that's awesome.  Can you check to see if you have the HSM module installed?

Code: [Select]
lslpp -l |grep -i hsm
I didn't see any server components in the previous output.  Can you confirm for me that you're using a remote TSM server?

-JD.