Changing retention for historical loads

[jsquizz]

I've asked a few questions about arsmaint. Here's where I am at. I'll use an example

AppGroup is linked to TSM storage set. AG settings say-

Cache Data for 60 days
Life and data of indexes - 365 days

So, I run arsmaint for the first time  - arsmaint -cd

My docs will be moved off cache to TSM. 295 days later I run arsmaint -cd again. Now my documents are expired, gone, "ca-put"

I'm told that "TSM is handling retention" meaning they are making deletions on the TSM side of the house rather than the CMOD side, which doesn't make any sense to me. In the past I've loaded ALL documents to one storage set (instead of a few year that are defined for 3 year, 5 year, forever..etc..) and then we've ran ARSMAINT..and I guess that's done the trick.

Making sure that documents that are current don't ever expire - is it as simple as changing the "life of data and indexes.." at the AG level?

[Justin Derrick]

The piece you're missing here is how TSM is configured.

If TSM is configured to expire data after 365 days, TSM will expire that data in 365 days after it's migrated in.  (Either by the Migrate-on-Load setting, or via nightly migration with CMOD's arsmaint -m option.)

If you change the retention in CMOD, without making a similar change in TSM -- TSM is going to delete that data after 365 days.

If TSM is configured to never expire data itself (that is, hold the data until IBM CMOD's arsmaint -d is run) then TSM will hold the data until OnDemand sends the 'delete' command to TSM to delete the objects that are ready for expiration.  There are exceptions to this rule, based on your expiration type.  If you don't get it right, then the data will live in TSM forever -- potentially a huge liability.

I imagine that the new Storage Set types (ICOS, S3, Filesystem) are going to be popular in the future for the reason that changes to retention won't be so confusing for filesystems or cloud storage.

-JD.

[jsquizz]

Thank you!

[Justin Derrick]

Oh, one more thing to make it even more complicated and ugly...

If you have a "1 year" policy set up in TSM, and multiple application groups use that "1 year" policy...  Then you change it to 7 years because the requirement for one Application Group has changed, then the retention in TSM will change for ALL of the Application Groups that use it.  It should go without saying that this is BAD.

This is why I recommend that every AG has it's own IBM CMOD storage set, and it's own policies and node defined in TSM.  It's a little extra work, but it is *so* worth it when you don't have to export / import an entire OnDemand Application Group.

-JD.

[jsquizz]

I don't mean to keep beating a dead horse with this one. But heres where more confusion lays. You mentioned one storage set for each AG..

One implementation I have seen, was several thousand application groups. We had one storage set defined in CMOD. I am not sure what the retention on the TSM side was at the time. We had so many initiatives going on within CMOD, that I didn't have time to learn anything about ANYTHING ELSE!

I wonder how that worked..we were strictly TSM, and anything that was cache was being migrated through a painfully slow process. We ran arsmaint weekly..using the -cdeimr flags. And app group life of data/indexes was various, 2555, 3650, 365, forever..etc.

So I wonder if we were just deleting the index and keeping the orphan document in TSM.

[Justin Derrick]

Yeah, even though there's a way to configure TSM to work that way...  It's a terrible idea.  It's very difficult to get right -- and even if you document how it works, you can't force your successors to read the docs and carry on in the same fashion.

-JD.

[jsquizz]

::bangs head off desk::

[jsquizz]

Not to bump this thread back from the dead.

Heres 2 scenarios.

Lets say I have an application group...

Load to cache for 5 days, then migrate over to TSM. Expire from TSM @ 30 days

Then we run our arsmaint -cvrmsid

30 days passes, and we run arsmaint as regularly scheduled. By now, the document has been expired from cache, and migrated to TSM. It's at its 30 day point, so ARSMAINT will remove the indexes from the db, and TSM will expire the data, because we have RETVER=30 as the setting.

Lets say I have another application group...

Load directly to TSM, never expire

Then we run our arsmaint -cmvrsid regularly, and our RETVER=9999 (Or whatever max aka forever is...)

The document will be expired in like 30 years 

Does ARSMAINT trigger or "TELL" TSM, hey, its time to delete this data? How does that work? is it proprietary?

[Justin Derrick]

If the expiration type is 'Load', then CMOD tells TSM to delete the object when it's eligible for deletion.  If the Expiration Type in the AG config is 'Segment' or 'Document', then the storage manager won't be instructed to delete objects, but will rely on it's own expiration.

-JD.

[jsquizz]

If the expiration type is 'Load', then CMOD tells TSM to delete the object when it's eligible for deletion.  If the Expiration Type in the AG config is 'Segment' or 'Document', then the storage manager won't be instructed to delete objects, but will rely on it's own expiration.

-JD.

Excellent, thanks

[jsquizz]

Bumping this back to life.

Does anyone know if the "-d" flag will handle expiration of documents from ICOS based on AG retention settings?

My initial two approaches-

ICOS vault for each application group -> retention settings mirror the application group
One ICOS vault -> Never expire, and let "-d" take care of it?

Does anyone have benefits/drawbacks to each approach?