Slow retrieval times

[jeffs42885]

Hi All,

We randomly experience slow retrieval times for documents retrieved via ODWEK.

We are currently in the process of running an ARSMAINT to migrate our files from CACHE to TSM DR550.

The command is setup in the crontab to run at 9PM, and usually finishes up around 11, 11:30PM.

Today I came in, and we were getting alerts that there were slow response times (via WILY) and ARSMAINt was still running, I had our DBA kill it. This is the command that was running:

/usr/lpp/ars/bin/arsmaint -I INSTANCE -m

Once our DBA killed the command, we are still seeing long retrieval times. Our usual fix is to recycle arssockd.


By the way, we are running 8.4.0.3, currently in the process of upgrading to 8.5.0.6

What do you all think?

[Justin Derrick]

Are you seeing a single slow retrieval, or all retrievals are slow, or retrievals are fast until you hit a "critical mass" of retrieval volumes?

It's really hard to diagnose without some more info.  Also, define 'slow' -- measured in seconds?  Or minutes?

The thing I see most is admins saying "Retrievals from TSM are fast.  Let's just push everything to TSM."  The snag is that TSM retrievals are (at best) 10x slower than cache retrievals.  So you're going to hit the wall (in terms of retrievals per second) 10 times faster.  Even modest retrieval rates at a busy time of day can start bogging down your system.

I'd also check the DR550's activity log/system logs...  There might be a disk failure in there which is causing slow retrievals by virtue of the fact that it's either rebuilding the RAID array, or performance is degraded as it's forced to calculate the value of each byte from remaining volumes.

-JD.

[jeffs42885]

Hi JD,

It is multiple slow retrievals, documents that usually take sub-second to retrieve are taking 20-30 seconds, and that's a big nono from our business partner. We have a monitoring tool setup that records response times for various servlets based on business unit.

Once we killed the ARSMAINT, it seems as though everything went back to normal. We recycled ARSSOCKD and ARSOBJD just to be sure.

[frasert]

You may see an improvement when you go to 8.5.0.6.  IBM included "TSM connection pooling" into that fixpack which effectively removes unnecessary TSM database queries when a connection from CMOD is being established.  We were having a major performance problem for a long time and finally we saw an instant and dramatic improvement when the 8.5.0.6 patch was applied.  We have a large number of application groups which I think contributed to the performance problem.

[jeffs42885]

Good to know.

We just went live with 8.5.0.6 in our contingency site this past weekend!

[Justin Derrick]

It's been a little over a month -- any updates jeffs42885?

-JD.