Migration Document stored in CACHE to TSM

[yousuf_gani]

Hello Alessandro,

I opened another Thread for understanding this Scenrio of how to Migrate the Object which was Only Stored in CACHE to TSM

As you had suggested this is Possible but not easy

"no problem :-) This is not impossible to do it, but this is not that easy. If you really want, then I can explain it to you :-) in another thread, so we don't confuse the readers"

As for Testing We Loaded many Docs to the CACHE but now instead of reloading again those DOc we thought to migrate it to the TSM.
If you have any such solution please let me know in your free time thanks a lot for all your help .

Thanks
Mohd Yousuf gani

[Alessandro Perucchi]

Hi Yousuf,

 

Well I will explain how to do it then 

This works on AIX/Solaris/Linux/HP
For Windows, you cannot use that method exactly, there are some "adaptation" to perform 

Please before doing it in production/test/...
Please test it, please do a backup before!!!!!!!!!!!!!!!!!

Basically, you need to follow the following steps:

1) You need to have a storage set defined for your AG, either you have it already, or you need to create it.
2) In the CMOD database, you need to get the SID from the storage set (table ARSSET), and the NID from the storage node that you will archive documents into (table ARSNODE)
3) In the cache, you need to create a directory call "migr" at the same location where the directory "retr" exists.
4) You need to know the AGID_NAME from your AG (table ARSAG or you can find it in the Admin Client on the advanced button of the storage management) (for example BAA)
5) in the directory migr, you create a directory with a number, this number is the internal date format of CMOD, which means the number of days since 1.1.1970. That's the moment that CMOD will migrate really the content into TSM, so I assume that you want to migrate it as soon as possible. That means, that you can use a number like 2 (mkdir 2)
6) in the directory migr/2 (for example) you need to create a link for ALL the objects that you need to move into TSM.
6.1) You can have the list of all your files like that with such a command:

Code: [Select]

find CacheDirectory/retr/BAA -type l -ls | awk '{print $NF}'

you'll get something like that:

Code: [Select]

...
/data/ars/archive/cache/01/15667/BAA/DOC/2FAAA
/data/ars/archive/cache/01/15667/BAA/DOC/2FAA1
/data/ars/archive/cache/01/16603/BAA/DOC/6FAAA
/data/ars/archive/cache/01/16603/BAA/DOC/6FAA1
...

6.2) Now with each of these files, you need to create a soft link with the following naming convention:

Code: [Select]

<AGID_NAME><DOC_NAME>.<PRI_NID>.0

So with my example, here are the commands I will have (I consider that my NID or PRI_NID is 18 as an example which was found in step 2) )

Code: [Select]

ln -sf /data/ars/archive/cache/01/15667/BAA/DOC/2FAAA /data/ars/archive/cache/01/migr/2/BAA2FAAA.18.0
ln -sf /data/ars/archive/cache/01/15667/BAA/DOC/2FAA1 /data/ars/archive/cache/01/migr/2/BAA2FAA1.18.0
ln -sf /data/ars/archive/cache/01/16603/BAA/DOC/6FAAA /data/ars/archive/cache/01/migr/2/BAA6FAAA.18.0
ln -sf /data/ars/archive/cache/01/16603/BAA/DOC/6FAA1 /data/ars/archive/cache/01/migr/2/BAA6FAA1.18.0

7) You need to stop CMOD (Maybe you don't need to do it, but I prefer to be safe than sorry)
8 ) You need to update the ARSAG table to point to the correct SID you got from point 2)

Code: [Select]

update ARSAG set SID=18 where agid_name='BAA'
9) You need to update all your segment tables to point to the correct Storage Node / Storage Set
9.1) First get all the tables you must update

Code: [Select]

select table_name from arsseg where agid in (select agid from arsag where agid_name='BAA')

you'll get a list of tables like this one:

Code: [Select]

TABLE_NAME       
------------------
BAA1             
BAA10             
BAA11             
BAA12             
BAA13             
...

9.2) Now for each table found in the previous step, you need to change the PRI_NID with the NID you found in step 2) :

Code: [Select]

update BAA1 set PRI_NID=18
update BAA10 set PRI_NID=18
update BAA11 set PRI_NID=18
update BAA12 set PRI_NID=18
update BAA13 set PRI_NID=18
...
10) Start CMOD
11) Check that you can still retrieve your document
12) Run the command

Code: [Select]

arsmaint -h <CMODInstance> -m
it will take some time...


I hope that I could explain in a clear way how to do it!! :-)

Again... please be really careful at each step, because this is not something trivial, and you're doing some changes in the database...

---
Alex

[yousuf_gani]

Thank a lot Alessandro

Your So Cool to Explain in Details ...
I am Trying this Option only in DEV environment (not in PROD), as we are actually setting up a New DEV enviornment for CMOD9.5.0.1  and TSM 7. 1. 1
I will try this option as you said .

Thanks again for all your support...

[Alessandro Perucchi]

Hi Yousuf,

please upgrade to 9.5.0.3 as soon as possible, since 9.5.0.1 has a bug with the delete of documents with TSM, so please don't stay at that level.

Cheers,
Alessandro

[Maciej Mieczakowski]

There is another approach to use arsadmin store command. If you have few objects to migrate it does the job as well

I do that in few steps:
1. Get the object name you are interested in  - use systemlog message 82 ex.

Application Group Object Store: Name(APP1) Agid(5679) NodeName(TA0410Y) Nid(85) Server(-LOCAL-) ObjName(4009FAAA) Time(0.057)

2. Get Primary Node ID - like from above example Nid(85)

3. Get FILESPACE_NAME as it's called in TSM or Application Group Identifier  as it's called in OnDemand (in my example APP1 = Agid 5679 = WYE), you can get this from ARSAG table (AGID_NAME)

SELECT NAME,AGID,AGID_NAME FROM ROOT."ARSAG" WHERE NAME='APP1'

NAME                                                                    AGID      AGID_NAME
------------------------------------------------------------ ----------- ---------
APP1                                                                    5679       WYE


or from Administration Client in AG properties, under Storage Management tab ->Advanced -> Application Group Identifier label

4. Find a a symbolic link in /arscache/retr/AGID_NAME/DOC directory to find relative path to the object stored in cache:

ls -l /cache_path/retr/AGID_NAME/DOC | grep ObjName

ex.
ls -l /ars/arscache1/retr/WYE/DOC/ | grep 4009FAAA
lrwxrwxrwx    1 root     system           35 Jun 29 2015  4009FAAA-> /ars/arscache1/16853/WYE/DOC/4009FAAA

5. Build a final command that will migrate data from cache to TSM

./arsadmin store -h serv_name -u user -p passwd -g app_gr_name -n prinid-secnid -d dir_of_cached_obj ObjName

ex.
/usr/lpp/ars/bin/arsadmin store -h ondemand.server.com -u admin -p admin -g APP1 -n 85-0 -d /ars/arscache1/16853/WYE/DOC 4009FAAA

And that's it File will be copied to TSM, it will be removed after all from cache location where it was stored previously and stored for another period of time in cache that is set in APPGR, same as during a the reload. So the relative location of that cache file will change.

I hope it's clear

[Alessandro Perucchi]

Hello Maciej,

I don't know what you are trying to do here... because the original question that created this thread has to do with an application group that has a storage set as "Cache Only - Library Server" or no storage set defined at all.

Then every document will go into the cache, and your cannot define a TSM node for them.

I don't understand the goal of procedure, because you have already an application group with a special storage set, you are already storing documents in TSM (I can see in your example, that you are using the node name TA0410Y)... so after the whole procedure, you are storing twice the object in TSM, meaning you are taking twice the disk space for every objects that you are archiving with arsadmin store.


Therefore, please explain me in details what is the goal of your post and method, because for me it simply doesn't work, and as described, and it doesn't do what it is supposed to do.
If you want do it with an actual example, that shows an application group that has NO binding with TSM, and how do you transfer the data to TSM with the cmod going to TSM to retrieve the documents, AND that the future archiving of document are going to TSM.
Because, again, your method doesn't work in that case, which was in fact the original question of Yousuf.

--
Alessandro

[Maciej Mieczakowski]

Indeed I see my mistakes here. I choosed a wrong entry from systemlog about exact destination of stored object, it supposed to be CACHE destination

Application Group Object Store: Name(APP1) Agid(5679) NodeName(-CACHE-) Nid(85) Server(-LOCAL-) ObjName(4009FAAA) Time(0.000)

in parallel we sent to TSM storage node
Application Group Object Store: Name(APP1) Agid(5679) NodeName(TA0410Y) Nid(85) Server(-LOCAL-) ObjName(4009FAAA) Time(0.057)

The procedure I described above I used when we have lost data in TSM due DB crash (unexpected server reboot), and we had to do point in time restore as roll-forward restore did not worked out due to last transaction crash and TSM didn't wanted to start. So we are loading data to cache and to TSM in fly, and we simply have lost data entries from TSM DB for certain period of time as we go back to latest backup (point in time restore). I hope it's clear now that we simply had to move data from CACHE to TSM and this is what this procedure is about.

Sorry I should have describe that earlier , my mistake And you realized me it's off topic. So @Yousuf it's different scenario and indeed you should use Alessandro's procedure

[Alessandro Perucchi]

Hello Maciej,

Ahhhhhhhhhh :-) that way I'm totally ok with your procedure!! :-) that's the way to do it!!! :-D

That's good, that way we have 2 kind of cache to tsm method in case one needs one or the other method!

Thanks for your input!

[Don Branton]

Alessandro,

Is there a way you can verify the documents get stored to TSM? I tried your above procedure and after I cleared the cache the documents were not retrievable. I have the same issue. I have two nodes in my storage set. One TSM and one CACHE only. I now need to migrate the cache only docs to TSM before they have to be cleared by the arsmaint daily run.

Thanks for your information and expertise.

[Alessandro Perucchi]

Alessandro,

Is there a way you can verify the documents get stored to TSM? I tried your above procedure and after I cleared the cache the documents were not retrievable. I have the same issue. I have two nodes in my storage set. One TSM and one CACHE only. I now need to migrate the cache only docs to TSM before they have to be cleared by the arsmaint daily run.

I am sorry to hear that you couldn't retrieve the documents after you cleared the cache. I hope you did all the test in the test environment and didn't do it in production until you were certain it was working as expected??

You said, that you followed my instructions, but what did you do exactly? Something went wrong at some point? Did you miss some steps?
Did you simplify something? Did you assume something? well anything that can help to understand if the instructions that I left contains a "bug" or not. I am at 99% sure that they are correct as they are...


How can you ensure that it was correctly saved in TSM?
Well 2 ways:
1) You look in the System Log of CMOD, and check for the message that says that it was sent to TSM. A message like this: Application Group Object Store: Name(APP1) Agid(5679) NodeName(TA0410Y) Nid(85) Server(-LOCAL-) ObjName(4009FAAA) Time(0.057)
2) You check in the TSM activity logs (act log) that you see the connections from CMOD and that he saved a file.

And if you can also in the TSM admin console you can try to display the content of the filespace for your application group in you node. It will display tons of files, and you should see the one you've migrated.

[Don Branton]

Alessandro,

I went through the steps again with the same results once I cleared the cache.
I did not have an entry in the system log showing that the documents have been stored in TSM.
I only see the entry for the initial storage to cache.
I did see one discrepancy in your explanation where you said the value for the SID as 18 and PRI_NID is 18. The SID is most likely going to be a different number than PRI_NID. In my case the SID is 1507 and the NID is 2. Just got a little confused on that.

Once I completed these steps I checked the migr/2 directory and it was no longer there. I assume that is supposed to happen since I ran the arsmaint -m command. Once I expired the cache using arsmaint -c -t 16820(yesterdays unix date) I saw the different icon in the OnDemand Client for the records I was trying to migrate. But got an error saying the server failed while retrieving the document. These are PDF files. Should I have migrated the resource files as well? The cach only node is defined in the same Storage Set as the TSM/cache node and set to Load Data.

I really appreciate your time and help with this. I made a rookie mistake and caused a major issue. This forum has been a great tool for me.


Step 1

Loaded new documents into cache only node.

Step 2

ecmcmt01(archive1)/cache>find /cache/cache1/ARCHIVE1/retr/FGA -type l -ls | awk '{print $NF}'
/cache/cache1/ARCHIVE1/16821/FGA/DOC/417FAA1
/cache/cache1/ARCHIVE1/16821/FGA/DOC/417FAAA
/cache/cache1/ARCHIVE1/16821/FGA/RES/369

Step 3
mkdir /cache/cache1/ARCHIVE2/migr/2
ln -sf /cache/cache1/ARCHIVE1/16821/FGA/DOC/417FAA1 /cache/cache1/ARCHIVE1/migr/2/FGA417FAA1.2.0
ln -sf /cache/cache1/ARCHIVE1/16821/FGA/DOC/417FAAA /cache/cache1/ARCHIVE1/migr/2/FGA417FAAA.2.0

Step 4

update FGA1 set PRI_NID=2

Step 5

check the retrieval of the documents - successful

Step 6

arsmaint -h archive1 -m

Step 7

check retrieval - success

Step 8

arsmaint -c -t 16820

Step 9

check retrieval - failure but the icon changed in OnDemand Client for the records

[Alessandro Perucchi]

ok, thank you I've seen that my explanation were a little bit too fast, and I didn't explain some parts...
That means basically that what you did is not "wrong" but they are not correct either, or at least from what you've shown me.

Tell me the application group corresponding to the AGID_NAME FGA what was the original SID for it? The one with the storage set "Local Cache" ?? Or something else?
If that's the case, then the missing step was to use or create a storage set specifically for this application group, because it is not good to have in the storage set "local cache only" something with tsm... bad practise... therfore you need to use a new one, or reuse an old one.

If it was something else, you had only one node at one point, which was pointing to the cache, then you can either change the cache to tsm in the node, or you can create a second node for tsm load.

In all these 2 cases, you will have a new SID which will replace (only for a change of storage set) the SID you have in the arsag table of the your AG.
And then you need to use the correct NID for all the steps I've explained.
And then it will work.

But from what you were describing such steps were not really done... and therefore it will not work.

I will find the time today or tomorrow to be more clear on this step with the storage sets and storage node... since this is a critical point, if you screw up there, then ... you can lose data.

[Don Branton]

The SID is the same for both nodes.

SID 5017 has  NID 2, NID 3

NID 3 is CACHE only
NID 2 is TSM/cache

I need to move from 3 to 2 in the same Storage Set. I think what I did worked its just looking for the resource files which I did not include in the migration.

Thanks,

[Alessandro Perucchi]

The SID is the same for both nodes.

SID 5017 has  NID 2, NID 3

NID 3 is CACHE only
NID 2 is TSM/cache

I need to move from 3 to 2 in the same Storage Set. I think what I did worked its just looking for the resource files which I did not include in the migration.

In that case, you did it correctly yes.
The only additional step to perform is to change the corresponding resources PRI_NID, SEC_NID entries in the ARSRES table. You will need to check the one that are related to your AG.

And that should be it.

I will also add that step in the description of this post, when I have time to review it correctly.

Thanks

[Alessandro Perucchi]

One additional thing about this process to migrate objects from one place to another... please be aware that the LoadID that was generated at the archiving time... in fact is void, useless.
You cannot use it any more... So if you saved your loadID, or need to unload some documents afterwards... then be aware of the consequences.

Of course, if you have the old load ID, you can tweak it, in order to be valid again. BUT the original one will not work any more.