Message 111 - OnDemand is unable to determine the link for the file

[jsquizz]

Hello all,

We recently had to move our arscache file system (jfs) to the jfs2 file system. Our unix team created the new volumes and then migrated everything over.

After the migration, I am seeing 71 errors in our system log, similar to this:

OnDemand is unable to determine the link for the file >/arscache/cache2/0/TFA/RES/6849<.  The error number is 2  Srvr->prod 10.10.10.10 non-SSL<-
 (Also same errors pointing to additional /arscache/cache1/0/TFA/RES/68** files)

We started off by looking at the links and everything looks good. So far, we haven't had any reports from users about their statements appearing incorrect. I tried duplicating the issue in our test environment and I also didn't see any issue with the output.

Has anyone ever seen this error? If so, what was your resolution?

8.5.0.6 on AIX btw.. Thanks in advance.

[Michel de Kraker]

Hi jsquizz,

My guess is that you are missing the symlinks to the /arscache/cache2/retr/..... directorie for some files.

See this example:

To make the search easier, CMOD creates symbolic links for the cache files, and stores the links in the path with names like /arscache/cache1/retr/CKM/DOC (where retr is a fixed value). If you issue

ls -l /arscache/cache1/retr/CKM/DOC

you should see output similar to the following:
lrwxrwxrwx 1 root root 36 May 9 13:47 26843FAA -> /arscache/cache1/18011/CKM/DOC/26843FAA
lrwxrwxrwx 1 root root 36 May 9 13:47 26843FAB -> /arscache/cache1/18011/CKM/DOC/26843FAB
lrwxrwxrwx 1 root root 36 May 9 13:47 26843FAC -> /arscache/cache1/18011/CKM/DOC/26843FAC



Regards

Michel.

[jsquizz]

It looks like all the links are still in tact in production,but every night we get the same 71 messages.

They are all related to the same application group, and it all seems like the resources are in numerical order, 6800-6871, as example (Across arscache1/arscache2)

I broke the link in our test environment and ran the same ARSMAINT and I'm not getting a 111 message.

weird

[Ed_Arnold]

Does the arsmaint -v  (validate) command show anything?

Ed

[jsquizz]

Excluding the 111/164 messages from our last ARSMAINT are below. We are running - "arsmaint -cvsr" nightly.

The 199 messages we were told are due to a bug in 8.5.0.6...The effected AG is "TFA"

   124   Filesystem Stats Name(/arscache/cache2) Data(65%) Inodes(0%)  Srvr->prod non-SSL<-
    32   Logoff
   124   Filesystem Stats Name(/arscache/cache1) Data(65%) Inodes(2%)  Srvr->prod non-SSL<-
   107   An unknown file or directory in found in the OnDemand cache >/arscache/cache2/0/GDA<  Srvr->prod non-SSL<-
   199   Unexpected empty directory in cache >/arscache/cache1/retr/HEA/DOC<  Srvr->prod non-SSL<-
   199   Unexpected empty directory in cache >/arscache/cache1/retr/AEA/DOC<  Srvr->prod non-SSL<-
   107   An unknown file or directory in found in the OnDemand cache >/arscache/cache1/0/LEA<  Srvr->prod
   109   Cache Expiration: Internal Date(17906) MinPct(80) MaxPct(80) Server(prod)
   107   An unknown file or directory in found in the OnDemand cache >/arscache/cache1/0/CEA<  Srvr->prod non-SSL<-
   107   An unknown file or directory in found in the OnDemand cache >/arscache/cache1/0/EDA<  Srvr->prod non-SSL<-
   110   Cache Migration: Internal Date(17906) Server(prod)
    30   Login: loopback 127.0.0.1 non-SSL (AIX) (ARSMAINT) (8.5.0.6)

[Justin Derrick]

Looks like you're missing some resources -- I think there's an undocumented feature in arsmaint -- with -R option should restore resources to the cache - you should probably specify the Application Group to restore resources on, to keep it from running for a long time.  I'm not sure it'll fix your cache issue though.

-JD.

[jsquizz]

Thanks. I will look into this. I just need to figure out what happened and how to duplicate the error in my test system.

After running -R against a file I unlinked, I am getting this-

   114   Unable to open file >/arscache/cache1/0/WDA/RES/3<.  The error number is 2  Srvr->TEST non-SSL<-

[Alessandro Perucchi]

   114   Unable to open file >/arscache/cache1/0/WDA/RES/3<.  The error number is 2  Srvr->TEST non-SSL<-
Modify message
Report to moderator   Logged

When I see this error, it means for me the following:

* The file located in your system at the following location /arscache/cache1/0/WDA/RES/3 does not exist
* The link located in your system at the following location /arscache/cache1/retr/WDA/RES/3 still exists
* Maybe, but the error message doesn't seem to indicate that, but probably the permissions are not correct.

What to do? Well, I am working with one of our old CMOD server, that nobody took care of... and it takes ages depending on the size of the cache, and how many files to solve all the problems.
The thing I would advice strongly is to have a backup of your cache before doing anything on your cache FS.
We had a backup, and we could solve our problem with great luck, otherwise millions of document could have been lost, since the files were not stored in TSM, but only in cache.... *big sweaty experience*

Basically do you use TSM, or only cache? If you have a TSM then you might need to try to restore the file /arscache/cache1/0/WDA/RES/3 which in your TSM should be located in /WDA/RES/3 or maybe /ARCHIVE/WDA/RES/3 .
The other solution would be if you don't know how to use the TSM retrieval of an archive object (dsmc command or some gui interface), then to use the command "arsmain retrieve", that way you can get your file back in a temp directory or directly in /arscache/cache1/0/WDA/RES/3
If you don't have TSM, then you must go to your Backup system to retrieve the missing file.

Maybe, because of my current scary experience I should write a memo on how to clean up/maintain a CMOD cache... because this is not rocket science, BUT this is so important, and lots of people have no idea how it works and how to cure it.