TSM expiration

[pankaj.puranik]

Hi

I know that TSM expiration is independent of CMOD migration.
But I just wanted to confirm this.
Here's the scenario.

I have CMOD set to load data directly into archive (0 day cache retention).
So the data is directly moved to archive as soon as it is loaded.
Assume that the TSM retention is set to 1 year.
Now i delete a file using arsadmin unload in the mid of the year.

This means that the database entry from CMOD would be deleted for all records in that file.
Immediately after this if I run TSM expiration processing, would it remove the objects for this file or it will still keep it till the 1 year period is over?

Thanks
Pankaj.

[Alessandro Perucchi]

Hello Pankaj,

Well it depends. I will try to give all the scenario I can (so this will be a long post :-D)

1) TSM defined with the storage node in Creation Mode

Then the retention of documents is asynchronous from the retention of the library, so it is important to keep both synchronous.

So if like your example you have 1 year defined, and after 6 month you unload the document, then in the library it will be gone, and it will in TSM that the full year is gone before deleting really the documents.

2) TSM defined with the storage node in Event Mode

TSM will wait until CMOD tell him to delete the document, and at this moment it will start the count down for deletion.
If you do nothing in CMOD, then the documents will stay forever in TSM.
If like your example, you unload the document after 6 month, then TSM will receive the order to delete the Load NOW.
Then depending on the setup of your storage node, here is what will happen:

if the option retinit has been setup during the creation of the storage node, then TSM will wait until at least this number of days has passed, in your case, if we have set retinit=365, then he will wait 182 days before being really deleted.
If the option retver has been setup during the creation of the storage node, then TSM will wait an additional X days. So if you put rever=30, then it will wait after receiving the delete event 30 days before deleting it.

If you put them together, then in our case, it will wait 182 + 30 days before really deleting the file.

If you delete the documents after 2 years (assuming that you didn't run arsmaint for 2 years :-D), then it will wait only 30 days, since the minimum of 365 days are already gone...


I hope this is a bit clearer.

We advice now to use the Event method for all the storage node in TSM, so the control is always in CMOD, and you don't have this problem of asynchronous deleted...

Sincerely yours,
Alessandro

[pankaj.puranik]

Thanks Alessandro.

I have to check but I think in our case it is asynchronous.

Now I have to delete the object from TSM manually/or using a script.
For this I need the object name.

In the Load ID below is 19FAA the object name that I can use to delete the whole file?

5097-3-0-19FAA-15522-15522

Thanks
Pankaj.

[pankaj.puranik]

Found this nice tech note on IBM website.

http://www-01.ibm.com/support/docview.wss?uid=swg21206601


In what situations will OnDemand expire document data from cache or Tivoli Storage Manager?
Technote (FAQ)


Question
When using IBM? Content Manager OnDemand server with application groups and storage sets defined for cache only, Tivoli Storage Manager only, or both cache and Tivoli Storage Manager storage, you notice document data is not consistently being expired.

In what situations will OnDemand expire or delete document data from OnDemand cache and Tivoli Storage Manager (TSM) storage?

Answer
In most cases, each load in OnDemand creates one or more internal storage objects. These storage objects contain the actual document data.

OnDemand attempts to expire a storage object from OnDemand cache in the following situations:

If an application group is deleted.
All storage objects in the application group are immediately deleted from OnDemand cache. If an arsadmin unload command is issued.
It will delete the load from OnDemand cache. A load consists of document index values from the database and storage objects from storage. Cache expiration is run using the arsmaint -c command.
OnDemand will expire storage objects from cache if all of the following conditions are met:
◦The maximum cache threshold is reached for a cache file system. This is specified when you run arsmaint with the '-x' parameter. The default value is 80%. ◦Cache Document Data for X days configuration in the application group has been met.
◦If the storage object has been configured to migrate to TSM and has successfully been migrated.
Note: If a TSM storage node was added to an application group's storage set at a later time, previously loaded data in OnDemand cache will not be migrated to TSM and will qualify for expiration. ◦Arsmaint does not encounter an error during expiration processing. If an error is encountered, the error must be corrected before cache expiration can continue. Run cache validation (arsmaint -v) and review the OnDemand System Log for cache errors. Index expiration is run using the arsmaint -d command.
If the application group is configured as Expiration Type of Load, OnDemand expires a load at a time based on the Life of Data and Indexes retention period that is configured in the application group. A load consists of index values that are inserted into the OnDemand database and storage objects in OnDemand cache and TSM.
OnDemand attempts to expire a storage object from TSM in the following situations:
If an application group is deleted.
OnDemand immediately sends a DELETE FILESPACE command to TSM, that deletes all storage objects that are archived in TSM under this application group. Index expiration is run using the arsmaint -d command.
If the application group is configured as Expiration Type of Load, OnDemand deletes a load at a time based on the Life of Data and Indexes retention period that is configured in the application group. A load consists of index values that are inserted into the OnDemand database and storage objects in OnDemand cache and TSM. OnDemand issues the TSM Client API, dsmDeleteObj, to delete storage objects in this scenario. If an arsadmin unload command is issued.
This command deletes a load from OnDemand in the same manner as the previous condition. A matching AFP resource is found in TSM when loading data.
OnDemand archives AFP resource and document data separately. If the load data contains AFP resources which match those that are already archived in TSM, OnDemand archives the most recent copy and deletes the previous matching AFP resource object in TSM. This ensures that the AFP resource and document data has the same archive date and retention period in TSM. The TSM Client API, dsmDeleteObj, is also used in this scenario.
If Data Retention Protection is enabled in TSM, OnDemand expires objects in the following situations:
Data Retention Protection On
Creation-based object expiration
OnDemand issues no commands to TSM. The objects are effectively orphaned by OnDemand and are expired by TSM based on their retention parameters. If the retention parameters are set to NOLIMIT, then the objects never expire.

Event-based object expiration
OnDemand issues an event trigger command through the TSM Client API. The event status of the objects that are affected are changed from PENDING to STARTED and the objects will be expired by TSM based on their retention parameters. If the retention parameters are set to NOLIMIT, then the objects never expire. If an application group is being deleted, a DELETE FILESPACE cannot be used with DRP enabled, so the operation is treated the same as if a delete were indicated. The status of all the affected objects is changed from PENDING to STARTED, and they will be expired by TSM based on their retention parameters. Because this leaves the filespace entries in TSM, you must manually delete these entries when the filespace is empty (even with DRP enabled).

Data Retention Protection Off
Creation-based object expiration
OnDemand issues a delete object command through the TSM Client API. Objects are deleted during the next inventory expiration. If an application group is being deleted, a DELETE FILESPACE command is issued instead and the objects are immediately deleted along with the file space.

Event-based object expiration
OnDemand issues an event trigger command through the TSM Client API. The status of the objects that are affected are changed from PENDING to STARTED, and the objects are expired by TSM based on their retention paramxieters. If the retention parameters are set to NOLIMIT, then the objects never expire. If an application group is being deleted, a DELETE FILESPACE command is issued instead, and the objects are immediately deleted along with the filespace.

To determine the object expiration policy of your TSM archive copy group, run the following command:

q copy t=a f=d

[pankaj.puranik]

I found that the object name could be found in message number 82 in System Log folder.

[pankaj.puranik]

In line with the posts above, I tried doing a load and an unload in our DEV environment.
Note that the app group is set to load data directly into SAN. The Retain Version on TSM is set to 0 (for our project specific requirement)
Here is what the TSM q act output was.

It says "Unknown command - arsadmin". Why would tsm use arsadmin, would it not have it's own deletion command?

03/05/13   05:00:38      ANR0406I Session 74 started for node SANNODE (ADMIN)
                          (Tcp/Ip servername(48510)). (SESSION: 74)
03/05/13   05:00:38      ANR8340I FILE volume /cna1/tsmfssan/00000006.BFS mounted.
                          (SESSION: 74)
03/05/13   05:00:38      ANR0511I Session 74 opened output volume
                          /cna1/tsmfssan/00000006.BFS. (SESSION: 74)
03/05/13   05:00:38      ANR0514I Session 74 closed volume
                          /cna1/tsmfssan/00000006.BFS. (SESSION: 74)
03/05/13   05:00:38      ANR0403I Session 74 ended for node SANNODE (ADMIN).
                          (SESSION: 74)
03/05/13   05:00:38      ANR0406I Session 75 started for node SANNODE (ADMIN)
                          (Tcp/Ip servername(48516)). (SESSION: 75)
03/05/13   05:00:38      ANR8340I FILE volume /cna1/tsmfssan/00000006.BFS mounted.
                          (SESSION: 75)
03/05/13   05:00:38      ANR0511I Session 75 opened output volume
                          /cna1/tsmfssan/00000006.BFS. (SESSION: 75)
03/05/13   05:00:38      ANR0514I Session 75 closed volume
                          /cna1/tsmfssan/00000006.BFS. (SESSION: 75)
03/05/13   05:00:38      ANR0403I Session 75 ended for node SANNODE (ADMIN).
                          (SESSION: 75)
03/05/13   05:00:41      ANR0406I Session 76 started for node SANNODE (ADMIN)
                          (Tcp/Ip servername(48521)). (SESSION: 76)
03/05/13   05:00:41      ANR8340I FILE volume /cna1/tsmfssan/00000006.BFS mounted.
                          (SESSION: 76)
03/05/13   05:00:41      ANR0511I Session 76 opened output volume
                          /cna1/tsmfssan/00000006.BFS. (SESSION: 76)
03/05/13   05:00:41      ANR0514I Session 76 closed volume
                          /cna1/tsmfssan/00000006.BFS. (SESSION: 76)
03/05/13   05:00:41      ANR0403I Session 76 ended for node SANNODE (ADMIN).
                          (SESSION: 76)
03/05/13   05:09:11      ANR0407I Session 77 started for administrator ADMIN (AIX)
                          (Tcp/Ip servername(48569)). (SESSION: 77)

03/05/13   05:10:14      ANR2017I Administrator ADMIN issued command: QUERY ACTLOG
                          (SESSION: 77)
03/05/13   05:12:37      ANR2017I Administrator ADMIN issued command: ARSADMIN
                          unload -h servername -u admin -p xxx -g CFSCearAGF
                          -L 5222-5-0-1FAA-14245-14245  (SESSION: 77)
03/05/13   05:12:37      ANR2000E Unknown command - ARSADMIN. (SESSION: 77)
03/05/13   05:12:37      ANR2017I Administrator ADMIN issued command: ROLLBACK
                          (SESSION: 77)
03/05/13   05:12:46      ANR0405I Session 77 ended for administrator ADMIN (AIX).
                          (SESSION: 77)
03/05/13   05:13:47      ANR0406I Session 78 started for node SANNODE (ADMIN)
                          (Tcp/Ip servername(48675)). (SESSION: 78)
03/05/13   05:13:47      ANR0403I Session 78 ended for node SANNODE (ADMIN).
                          (SESSION: 78)
03/05/13   05:13:57      ANR0407I Session 79 started for administrator ADMIN (AIX)
                          (Tcp/Ip servername(48679)). (SESSION: 79)

[Justin Derrick]

Someone very clearly typed a UNIX command into the TSM prompt.  arsadmin need to be run at a shell login prompt, not inside TSM.

-JD.

[satishkumar83]

Hello all,

May be too late to this post but never mind it can help someone in future

One more additional step needs to added here  to see the space reclamation i.e. TSM needs to restarted

below script can be used to unload all the doc id 's

Load id's can be retrieved from system log using msg num 127 and the same can be formatted in excel

 for i in `cat /tmp/unload.txt`
do
  arsadmin unload -u username -p pwd -h server name -g appgrp -L "$i"
done

[Alessandro Perucchi]

Hello satishkumar83,

Well at least if you answer try to give correct answer to the question asked!! :-)

The message number in the system log is not 127, but 87.

And additionally Pankaj was asking about how TSM was handling the retention and not how to remove the document, something he already did if you have read his e-mail.

Nevertheless thank you for trying to answer.

-
Alessandro