Can bugs in Java apps cripple a CMOD server?

[Justin Derrick]

Hi everyone.

I've got a customer with a weird problem with CMOD 8.5.0.5 & ODWEK 8.5.0.5, on AIX 6.1.  Their CMOD server seems to choke on about 4-5% of query & retrieval requests -- going from 0.05 seconds per request to several hundred seconds.  But here's the catch:  The System Log doesn't see anything wrong, and reports that it served ALL requests in the low-hundredths of a second.  It's not even necessarily during times of the highest load -- the lousy performance is frequent seen during periods of modest user activity. 

There aren't any 'gaps' in the System Log either, which would indicate that requests were queueing up (somewhere) and then suddenly being served in a flash.  The icing on the cake is that during these periods of poor performance, all CMOD clients are affected -- the Java front end, plus CGI and Windows thick client.

The front-end Application lives on a WebSphere server, and connects to the ODWEK Java API through the IBM HTTP Server, which runs on the same server as CMOD.

I guess I'm asking if anyone has ever seen anything like this (response time is occasionally terrible, but with no indication that anything is wrong, anywhere), and if so, where they looked for the problem, or how they solved it.

Thanks in advance for any thoughts or insight!

-JD.

[jeffs42885]

I am curious to this as well. JD- do you use any kind of monitoring tool for servlets and such if you are retrieving via web apps?

[Justin Derrick]

It turns out that this is a known bug in the Global Security Kit (GSK) that is required by CMOD 8.5.x.  Right now, it appears to primarily affect large, high volume CMOD installations on AIX servers with Power7 CPUs.

The GSK is used by CMOD to provide the new (and more secure) password-hashing algorithm -- even if you don't have encryption enabled through Secure Sockets Layer (SSL).

It's especially prevalent when using the CGI interface of ODWEK, or when loading a large number of files -- and drags down the performance of other clients like WEBi, the Windows Clients, and custom apps using the Java API.

While there isn't a patch that's been verified to resolve the issue, I'll be implementing a workaround on a customer system this weekend, and if it's successful, I'll post the results on the MP forum.

-JD.

[javed.khan]

JD...
Did you implemented the workaround? If yes did you see any improvements?

[Justin Derrick]

I implemented it, and it didn't resolve the issue to my customer's satisfaction -- in some cases it made the situation worse.

That's why I've recommended large installations (who are at a previous version like 8.4.x or 7.1.2.x) to avoid 8.5 -- or at the very least do considerable testing to see if they're affected.

-JD.