ODWEK - OnDemand slow retrievals when arsmaint is running

[Nikesh PJ]

We have a custom UI using ODWEK to connect to OnDemand.
Problem: The retrievals are very slow (>10 seconds for 23323 bytes document) via ODWEK.
The retrieval took 0.010 seconds using CMOD Client for the same document.

We have scheduled ARSMAINT to run every morning.
For some reason ARSMAINT was running for the whole day. Do you know why?? 
This usually doesn't run for more than 1 hour. And there were not much documents to expire.

When I stopped the ARSMAINT and restarted the Object server, the retrieval time was back to normal via ODWEK. 
I found an old post with similar issue but reason for the slow retrievals were not explained.
Why does ARSMAINT affect retrievals via ODWEK.

Details:
CMOD V9.8
DB2 V10.5
RHEL 6.9
Document retrieved from Cache (Stored both on Cache(30 days) and TSM on load)

 

[jsquizz]

Ive always ran ARSMAINT off hours due to behind the scenes work to the DB/Storage (Ive seen issues like yours in DR550 and Centera..)

Also- Check your system log for 65/226 records

[Nikesh PJ]

As I mentioned before, we had scheduled ARSMAINT during off business hours.
And that used to complete within 1 hour (for expiring x documents ).
Suddenly it started running for more than 24 hours.

Does some one know the reason for ARSMAINT to take too long time to complete?
I mean the expiration was too slow (for expiring same x documents).

[Ed_Arnold]

The only time I've seen ARSMAINT expiration suddenly running a long time is when a migration from another platform to CMOD was performed some years ago (say 7 for example).

At that time a lot of docs were all set to expire on the same date.

In 7 years' time everyone forgot (or was no longer there) that there was a big expiration coming up.

Suddenly, tons of expiration processing.

Oh no!  Is expiration processing running wild?

Are things getting expired that aren't supposed to be expired?

No, it was an unexpected increase in volume.

This is what we call the "pig in a python" problem.

Ed