Ondemand Cache disk balancing

[j4jackycheng]

Hi All,

Greeting!

First of all, thanks everyone in this Ondemand User Group, i have learn a lot from this forum throughtout the years.

According to IBM content manager ondemand document and case note, "If there are multiple file systems in the ARS.CACHE file, OnDemand uses the file system with the greatest amount of space free to store the objects".   
However we are not seeing this behaviour in our system.                           
                                                                       
We have always been following the 1st method in technote (http://www-01.ibm.
com/support/docview.wss?uid=swg21409251)
to extend the size of the current cache file systems.                                                                                   

We have 7 cache File systems all together, configurations for these FS already exists in file ars.cache
and we did not modify the ars.cache file for few years.  On the beginning of Jan-2013, all of our cache file systems are of same
size 318GB.                                                 
On Aug-2013, we have added in more disk space to the existing file   
system, each of them becomes 408GB
                 
                                                                       
Our Question are: 
1.  We have noticed that for the last few months, documents being uploaed always go to 4 particular cache FS,
which is NOT the one with most free space.  Now these 4 file systems are around 90% full each, while the others are seating at 50-60% full.
Is there anything we are missing so that it doesn' know some disk is very free? How can we force Ondmenad to use those FS which is more free at the moment?

2.  What would happen if one/few of the Cache disk reaches 100% full?       
Would the Ondemand server stop working and crash? Or would it stop us form uploading any new documents?               
   
Another thing is we are still at version 8.4.  I have ask IBM support the same question but they are basically saying "try upgrade to 8.5 first then we will talk  ".  That is why i'm asking for help from ODUG.  Thanks a lot.

Cheers,
Jacky

[jeffs42885]

My suggestion is to migrate to something like TSM if it is an option, and if you don't have the internal resources to do so, i know for a fact that IBM has some GREAT people that can help you. I've worked with him, alongside our DBA, TSM admin, and he was an absolute genius with doing everything. Very responsive, we never had to track him down. Just a great person to work with.

He also threw out some funny puns every once in a while . . . . lol

[Justin Derrick]

If you've been making changes to your system while it's online, you may simply need to restart your system.  Check your ars.cache configuration file to make sure all the filesystems are listed.  Also check to ensure that the permissions on the filesystems are correct, and all are the same.

When the cache filesystems hit 100%, files stop loading.  It wouldn't 'crash'.

Also, if you are not using TSM, you should.  For almost all customers, the majority of data stored in systems is inactive -- nobody is accessing it, ever.  It should be on tape, where it costs the least amount of money (electricity, cooling, etc.) to store long-term.

-JD.

[j4jackycheng]

Thanks jeffs42885.  I'm sure IBM has many nice guys, but i think it also depends on the contract and lisence terms.  I'm afraid in my case they insist us to upgrade to at least 8.5 first.

Hi Justin,

Thanks for your quick response as well.
1. When you say "restart my system", does it means ondemand server restart OR the entire Unix OS?
Because the disk were added quite a time ago Aug-2013, and from top of my head, we have done at least once Ondemand restart, and another one Unix server restart.  And there is no effect on this matter.

2. Check your ars.cache configuration file to make sure all the filesystems are listed.
This is the very first thing i checked.  And i can confirm that all the filesystems are listed, plus we are only extending the disk size for existing file system here, there is no addition to the ars.cache file.

3. "Also check to ensure that the permissions on the filesystems are correct, and all are the same."

This is a interesting point, is there any specific permission pattern i should follow?

Original i was looking into this direction as well.
And i have discovered the following:

e.g. these are the FS entries in the ARS.CACHE file, and the relative free space % (note all of them are same size):
/ars04/archive/cache01          (62% free)                                       
/ars05/archive/cache02          (62% free)                                       
/ars06/archive/cache03          (62% free)                                       
/ars08/archive/cache04          (89% free)                                         
/ars09/archive/cache05          (89% free)                                         
/ars10/archive/cache06          (89% free)                                         
/ars11/archive/cache07          (89% free)

My problem is: files likes to goes to cache04 to cache07 and ignoring cache01 to cache03.

Our ondemand instance owner is "archive"
Our ondemadn daemon jobs e.g. arssockd, and db2 jobs are started by "root"

Ininitally, i have found that the folder permission is different like folloiwng:

e.g.

> ls -la /ars06/archive/
total 128
drwxr-xr-x   3 archive  sysadm1       96 Jul 28  2001 .
drwxr-xr-x   4 archive  sysadm1       96 Jul 27  2013 ..
drwx------ 3444 root     root       65536 Aug 18  2013 cache03

> ls -la /ars11/archive/
total 96
drwxr-xr-x   3 archive  sysadm1       96 May  9  2010 .
drwxr-xr-x   4 archive  sysadm1       96 Jul 27  2013 ..
drwxr-xr-x 2773 root     root       49152 Feb 19 02:13 cache07

I try to test it out in our test environment (which also have similar issue since added disk), and then change the
cache01 to cache03 folder from "drwx------" to "drwxr-xr-x", and then restarted the ondemand server, and then try
upload new documents, the documents still do not want to go to cache01 to cache03.

Thanks again,

Cheers,
Jacky

[Justin Derrick]

*DING!*  You definitely have a permissions issue.  All cache directories should all be owned by whichever ID runs arssockd, and should only be readable/writable/executable for that user ID.

In the future, consider switching over to having all processes run under regular user ID's.  It's just a security precaution.

-JD.

[jeffs42885]

*DING!*  You definitely have a permissions issue.  All cache directories should all be owned by whichever ID runs arssockd, and should only be readable/writable/executable for that user ID.

In the future, consider switching over to having all processes run under regular user ID's.  It's just a security precaution.

-JD.

We had this happen before and- some of these directories within the migr directory were changed from the database instance owner / app that ran ARSOBJD to our regular process account (The same one that runs arsload and that we'd use to login via putty (bad, i know..)

needless to say, it was either a permissions issue or an ownership issue. In the business world, it was a SEV1 issue, random files were failing and not retrievable. Of course this was in the middle of a cache to TSM migration

Again, I suggest going the professional services route if your budget supports it.