large page file on windows server

[bruce.mchendry]

Hi,
I've noticed something the last while on both our stage and prod windows 2012 servers. Our pagefile.sys keeps ballooning up to 90+ GB. One of the server guys suggested it may be the remote SQL 2012 server ( we have to have a remote SQL server ) doing some kind of caching / pooling that's contributing.
I do have a PMR open with IBM and an internal ticket with the DBA's to investigate as well too. Has anyone seem this ? If so did you find out what it was ?
Thanks and cheers,
Bruce

[Justin Derrick]

That's weird.  I wouldn't suspect the database in this scenario.  The CMOD server wouldn't have any use for caching database pages locally -- it would still need to access them via the database engine.

I'd be looking for a memory leak.  Does it happen quickly (within a day of rebooting) or is this only something that happens after a few weeks?

Does Windows have tools like top/topas or ps from the Unix world?  That would tell you which process has the biggest VM footprint.

-JD.

[bruce.mchendry]

Hi,

Thanks for the response ! Well interestingly we had to apply what we call "Emergency Zero Day" security patch and rebooted Stage Tuesday night and the pagefile is at 94GB again as of this morning. I will have to talk to my server guys about looking into the memory leak side.

Thanks for the info on SQL, to be honest I was kind of hoping it would be that in that it might be easier to find and adjust. Most times memory leaks are a pain to find. Good info though and I will pass it on.

Cheers

[Justin Derrick]

Yeah, 94GB in 24 hours is a BIG problem.  The good news is that it should be easy to find, and it'll be growing rapidly -- like 3-4GB/hr.

Good luck, and don't forget to tell us how it turns out!

-JD.

[bruce.mchendry]

Hi,
So far issue is still ongoing. I've had some of the usual fun getting someone to take ownership. I have gone on the server myself and enabled some performance monitoring to get a handle on threads etc but typically it takes a little while to gather data. We will also have a reboot tonight so I want to get the counters running after that and see what I find.
Cheers

[bruce.mchendry]

Sadly no real progress in this issue. After spending a bunch of time trying some things myself and having our server team review they keep saying it's the app. Now we're in Prod my IBM project team isn't looking at it. I am opening a PMR although I'm not overly optimistic at this point. One intersting note is that this Prod VM server has and exact ( configuration) DR server. Exact in the we run an app called Double Take to replicate the data over to the DR box. Since the Prod box does all the actual processing it's interesting that as of today the Prod server pagefile is max at 47.8 GB but the DR server is only at 4.25 GB. We will see if there's a solution . Also of note is our files are all sizes including some large ones like 100's of MB daily.
Cheers

[Justin Derrick]

It wouldn't be unusual for a tool that tries to keep a list of all files on the system to use an enormous amount of RAM.  Filesystems like the CMOD Cache contain millions of files, and the retr subdirectory contains millions more links.  Can you disable that synchronization tool for a weekend to see if that's your culprit?

-JD.

[bruce.mchendry]

Hey Justin, TGIF  ! That's an interesting idea and one I'll follow up on today as soon as I track down one of my server team buddies. Appreciate the input !
Cheers

[bruce.mchendry]

While we are still having issues with large page files we have collected a lot of data and are presenting it to our vendor with the intent they'll do a code review of our daemon. There were a number of documents and spreadsheets created to show we had an issue. One of the interesting ones was the Windows Sys Event logs over the last 30 days have shown the ODDaemonLT.exe process averaging 75 GB a day in virtual memory use. We'll see what the code review produces.
Cheers

[Justin Derrick]

Thanks for keeping us posted.  What does that ODDaemonLT process do, if you can tell us?

-JD.

[bruce.mchendry]

Hi,
This is the executable for the processing of our files to the specific folders that the end users search on in ICN. Best way to explain is this from our design document.
The OnDemand load daemon is an automated function to search for expected files in a predefined location and when found to perform the necessary actions on the file to index and load it into the OnDemand repository. The actions to be performed include the invocation of preprocessor functions, decompression, indexing, post indexing exits and finally the actual OnDemand load operation. The results of the operation are summarized and presented back to a specified FTP location to act as part of the audit trail of the overall process and for problem determination activity.   
The ‘daemon’ will monitor on a user-specified time cycle for the receipt of the expected files from the host system. The receipt of a file is determined by monitoring the appropriate directories, one for each of the statement files as described in the Detail section below. Upon finding a file, it will then perform the operations as described in the file group table entry for decompressing or preprocessing and then initiate the OnDemand load process. A successful load will then trigger an ftp transmission of a control (load log) file back to the host to the appropriate ID to indicate that the load was either successful or not. 

[bruce.mchendry]

I have mixed feelings about this. One thing we are doing now is having the server team do a weekly maintenance reboot of the server. They have an automated process where they do about 50 or 60 of them every Sunday and we just had them add ours to the list. That sort of resolves the issue from one perspective and being it's a Windows server we know it doesn't hurt.
The second part I'm not as happy about. We have this custom daemon program running and to save resources it had been set up to start and stop every 5 minutes. "Somehow", I know it wasn't me that had been changed to every 5 seconds. Big difference ! We also felt there was a memory leak so IBM reworked the code and we know have a changed version in Stage that we are watching. I think we'll end up changing Prod too but in the mean time the weekly reboots have certainly kept the pagefile down and the Stage pagefile seems better with the new daemon code and settings. I'm going to think positive and believe we've turned a corner on this. Thanks for all the help and suggestions along the way !
Cheers