Little background of our environment, 1Lib Server, 1 Obj Serv, RHEL + DB2, DB2 storage is on SAN Veritas Cluster, CMOD Cache is also on Veritas Cluster
We just upgraded from Redhat 6.8 to Redhat 7.9 Maipo. After a few days we noticed slow response times from our API's calling via ODWEK (Weblogic). These requests would timeout after around 20 seconds, causing impact to our customers. We also see instances where loading is slow, example- 12.8 Seconds to load a file with 11 Docs..500kb size.. using generic indexer (user defined html file).
We opened up a case with IBM, and the first suggestion was to upgrade CMOD to 10.5, so we did that, the CMOD upgrade went perfectly fine - and the issue persisted. We checked all of our DB2 (V11.1fp4) logs, and we don't see anything. CMOD says they cant see anything wrong, and I agree with them. the only thing they are seeing is a pause/delay in the system log..There was a 40-50 second delay where there was absolutely no activity, whereas for the whole day there is usually stuff going on. After that pause, there's some 6second long queries in the 226 records.
We also pulled in DB2 Support, and we are giving them updates/db2mon/db2logs - but i dont feel like we are getting anywhere.
Veritas is looking at our system, along with Redhat support.
We've engaged several of our DBA's, Storage Engineers, and System Admins to address the issues.
One of our SA's suggested rolling back the RHEL kernel to 7.8, due to some major features added in 7.9. We did that overnight, and sure enough when I logged on this morning, no bueno.
The biggest issue, is that the problem is intermittent. Our next step is to run on one library server instead of lib/obj to rule out any kind of network latency or anything like that. I also think it might be a shot moving the DB2 volumes off veritas to local disk, or something.
I am open to any possible feedback, if anyone has encountered issues going from RHEL6-7.