Issue with Ondemand load service.

[Abhishek Gupta]

Dear All,

I need your help/expertise in helping me out with the issue of Ondemand load service.

Background of the Issue:

We have one load service configured against a load directory which resides in other server than the OD server.
This load service would pick up the files and load it into the OD.
However, the performance was pretty good in terms of the speed of loading of the files to OD server, until one day after which suddenly the performance of the load service went down and this is resulting in piling up of the files in the load directory. As a result of which, the availability of the documents in the system is badly affected.
From business perspective, we have the SLA of having the documents available in the system within 3 minutes.
But since the load service performance is gone down the documents are not getting loaded in time.

Apparently, other load services configured in the same OD server are having good performance.

Steps taken to resolve:
We have done the restart of the affected OD load service, changed the polling interval to 2 seconds, but after which it still is loading the files pretty slowly.

Further we don't see any latency at the network level between the OD server and the server where the load directory is configured.

Could please suggest me on how I could go ahead with troubleshooting this issue?

Version: IBM CMOD v10.1 for multiplatform.

Hope to get some suggestion!!!

Thank You
Abhishek

[Michel de Kraker]

Hi Abhishek

Are you using NFS of SAMBA for getting the files from the remote server?

Kind regards

Michel.

[Abhishek Gupta]

Hi Michel,

We are using SMB for getting the files using the native protocols 139 and 445.

Thanks
Abhishek

[Michel de Kraker]

Hi Abhishek

Can you sent me an extract of the system log about 1 complete load ?
It will show where it takes time.

regards

Michel.

[Abhishek Gupta]

Hi Michel,

I have attached one.
It says it took around 28 seconds to load the file.

Thanks
Abhishek

[Michel de Kraker]

Hi Abhishek

Looks to me that this is DB2 related. Does the db2diag.log show any usefull messages around the same time?
Do you ever run arsmaint on the Application Group?


Regards

Michel.

[Abhishek Gupta]

Hi Michel,

I checked the db2diag.log for the specific timelines as present in the uploaded log files, however, I am unable to find any kind of error logged for the same.

Further, I am going to try out uninstalling/deleting the faulty load service and reconfiguring it in a fresh manner.
Let's see how it goes.

Please share your thoughts and any further suggestions.

Thank You
Abhishek

[Justin Derrick]

How old is this server?  Has it been in production for a long time?  You haven't mentioned the version & fix pack levels of the software you're running -- and those are important to mention when asking for help! 

-JD.

[Abhishek Gupta]

Hi Justin,

We have recently upgraded our Ondemand from 9.0.0.1 version to 10.1.0.0.
So we are using our new server from Aug 17, 2018.

However, to below discussion as suggested by Michel this could be a DB2 issue.

One thing I would like to mention here is the time when we upgraded our Ondemand the load service was working fine and the performance was upto the mark.

However, one change I had done in September was I updated the "Cache Document Data" timelines from 90 days to 30 days for all the application group.

I updated it by modifying the respective ARSAG table from DB2.

Could that modification have created any issue? Further, I do not see any concrete errors logged in DB2DIAG.log for the same.

Please suggest.

Thanks
Abhishek

[Justin Derrick]

A few more questions...

Did you also upgrade DB2?  To which version?  Did you apply any maintenance or fix packs to DB2?

You said you upgraded CMOD v9 to 10.1.0.0 -- why did you not apply fixpacks?  Fix Pack 4 for IBM CMOD v10.1 was just released.

-JD.

[Abhishek Gupta]

Yes, we upgraded DB2 as well to 11.1.2020, however, we have not applied any fix packs for OD, it is still at 10.1.0.0.

Please note post upgrade the load service was working fine, However, one change I had done in September was I updated the "Cache Document Data" timelines from 90 days to 30 days for all the application group.

I updated it by modifying the respective ARSAG table from DB2.

Could that modification have created any issue? Further, I do not see any concrete errors logged in DB2DIAG.log for the same.

It would be interesting to know, how the fix pack 4 would resolve this slow loading issue in OD.

Please suggest.

Thanks
Abhishek

[Justin Derrick]

Have you checked the README file to see if there were any APARs related to slow loading fixed?

-JD.

[Abhishek Gupta]

Hi, Just to update, I have resolved the issue by clearing the stuck items from the content queue table of the object store where the metadata of the OD documents get federated to.

Thanks all for your help.

Regards
Abhishek