Author Topic: Enhanced Retention Management and expiration  (Read 2485 times)

rjchavez

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Enhanced Retention Management and expiration
« on: December 16, 2020, 07:44:39 AM »
We're embarking on implementing ERM soon.  As I understand it, ERM retains documents by placing HOLDS and not by placing an expiration date on the document.  It is also my understanding that you must remove expiration processing currently in place (we currently use OAM for storing our reports).

So if t his is the case then how will reports that do not contain a HOLD, expire under notmal circumstances?  How do I control certain reports' expiration dates as I do using OAM today?  Will I lose this capability?


Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2228
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: Enhanced Retention Management and expiration
« Reply #1 on: December 16, 2020, 04:42:59 PM »
The thing to know about ERM is that it allows you to select documents to be put on hold (say, a regulatory issue, or lawsuit, or audit) or 'implied hold', which means 'hold everything I load until I say it can be deleted'.

To answer your first question...  In the first situation, yes.  Documents without 'holds' applied to them will expire as per your standard schedule.  Documents *with* holds will be held.  In the second situation (implied hold), you will have to remove the implied hold from documents in order to get them expired.

You're going to want to work with a specialist in CMOD on Z, because the implementation details are tricky, and going back to apply ERM on old documents will take some work - either low-level changes to database tables, or export and reload of data.



IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

Mehmet S Yersel

  • Jr. Member
  • **
  • Posts: 56
    • View Profile
    • LinkedIn Profile
Re: Enhanced Retention Management and expiration
« Reply #2 on: December 17, 2020, 06:51:48 AM »
1. ... how will reports that do not contain a HOLD, expire under normal circumstances? 
2. How do I control certain reports' expiration dates as I do using OAM today? 
3. Will I lose this capability?

1. You need to first document the "normal circumstances". Whether OAM does life cycle management via daily DFHSM processing or not in addition to ARSMAINT process. Who (CMOD/DFHSM) deletes what? Who is in control for each different scenario in your situation? You will need a storage engineer from OAM support for some details if you are not well versed in OAM side.
2. This depends on what you find out in #1. Let's say OAM expire objects in addition to and in coordination with CMOD ARSMAINT processing. You will then need to identify the storage groups related to the application groups, find their storage/management classes (life cycle policy) and make changes there... You can make global changes by asking storage engineers to change the life cycle rules on storage/management classes; or you can do a directory update of select OAM objects by manipulating the ODCREATDT/ODPENDDT ... I have done both depending on the requirement and my comfort level. Updating the OAM directory tables with SQL query is very risky if you are not well versed in OAM. You should test it thoroughly if you choose this option. Making changes to storage/management class life cycle management rules as well could be risky if you are not granular enough to impact only the intended objects.
3. Everything depends on what you find in #1 and what you do in #2 and how you do it.   

EDIT: I should have said OSMC instead of DFHSM; I realized that I started to mix these 2 acronyms sometimes. As much as I know to look for the OSMC keyword in the logs when checking what OAM house keeping tasks did, I mix these keywords when speaking about them. Thanks for correcting.
« Last Edit: December 18, 2020, 11:55:31 AM by myersel »
#zOS #Multiplatforms
#DB2 #OAM
#AFP #RiCOH AFP2PDF #SnowBound
#Finance #Telecom #Airlines
#ICN #IHS #WAS ND #Cert and Key Management
#Migrations #Data Modeling #RACF-2-CMOD Synch
#FileTek AMMO #ABI #RMDS #RADAR

rjchavez

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: Enhanced Retention Management and expiration
« Reply #3 on: December 18, 2020, 08:53:44 AM »
Thank you for your responses.

We assign storage nodes using the ARS.RSADUPDT exit which analyzes the number of Days coded in the "expire in" field of the AG definition.  Based on the number of days coded, there is an assigned storage node with the appropriate management class assigned to it.  We then run OSMC automatically to expire objects based on their assigned expiration (no DFHSM).  We also run nightly ARSMAINT to expire there as well.

Now we do not want to change that process and do not wish to change any of the current object's expiration dates.  All we want to use ERM is to retain documents "already loaded" to a different date.  We often receive requests from the users that they would like to retain reports that, for example, expire after 90-days, to now expire after 5-years.  I suppose that we could then place a HOLD on all of these reports but that would mean we would need to "remember" to remove the holds individually, after 5-years.  Does not seem like ERM is the right solution for this but rather some in-hour grown solution?

Your thoughts and comments are welcomed.

Mehmet S Yersel

  • Jr. Member
  • **
  • Posts: 56
    • View Profile
    • LinkedIn Profile
Re: Enhanced Retention Management and expiration
« Reply #4 on: December 18, 2020, 12:12:12 PM »
Thanks for describing the environment in good enough detail so we can understand the problem better.

As I understand, ARSMAINT takes care of the indexes and OAM built in life cycle management (OSMC) is used to expire objects. In this case, again, depending on the situation you have different options:

I would definitely go with updating the ODCRETDT/ODPENDDT attributes such that OSMC when inspects the objects avoids expiring them. Just remember to restore to normal.

for example ODSCRETDT is 2020-01-01 and you want this object to have a +6 months extension on top of the original. You can run a DB2 command on the object directory to modify ODCREATDT to ODCREATDT + 6 months and ODPENDT to CURRENT DATE; this will ensure OSMC processing looks at the object in next cycle and gives it new expiry.

You must test this first and observe different use cases.

I have done this a lot but it could be dangerous if you are not sure what you are doing.
#zOS #Multiplatforms
#DB2 #OAM
#AFP #RiCOH AFP2PDF #SnowBound
#Finance #Telecom #Airlines
#ICN #IHS #WAS ND #Cert and Key Management
#Migrations #Data Modeling #RACF-2-CMOD Synch
#FileTek AMMO #ABI #RMDS #RADAR

rjchavez

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: Enhanced Retention Management and expiration
« Reply #5 on: December 19, 2020, 08:41:12 AM »
Thanks for describing the environment in good enough detail so we can understand the problem better.

As I understand, ARSMAINT takes care of the indexes and OAM built in life cycle management (OSMC) is used to expire objects. In this case, again, depending on the situation you have different options:

I would definitely go with updating the ODCRETDT/ODPENDDT attributes such that OSMC when inspects the objects avoids expiring them. Just remember to restore to normal.

for example ODSCRETDT is 2020-01-01 and you want this object to have a +6 months extension on top of the original. You can run a DB2 command on the object directory to modify ODCREATDT to ODCREATDT + 6 months and ODPENDT to CURRENT DATE; this will ensure OSMC processing looks at the object in next cycle and gives it new expiry.

You must test this first and observe different use cases.

I have done this a lot but it could be dangerous if you are not sure what you are doing.

Thanks again.  I could not find the ODCREATDT (nor ODSCRETDT) column you're referring to.  I suppose you're referring to ODCREATS in the OSM_OBJ_DIR table?  I did find ODPENDDT in the same table though.  Another question is, if I can successfully update the dates as you specify and OAM does retain the objects for longer periods, how do I update the indexes to prevent ARSMAINT from expiring them as well since without "both" I would not be able to retrieve the documents from within CMOD?
« Last Edit: December 19, 2020, 08:57:53 AM by rjchavez »

Mehmet S Yersel

  • Jr. Member
  • **
  • Posts: 56
    • View Profile
    • LinkedIn Profile
Re: Enhanced Retention Management and expiration
« Reply #6 on: December 21, 2020, 06:41:39 AM »
Sorry for the wrong name of the Object Create Time Stamp... I tried to remember it without looking at the manuals and I was close enough, so it is ODCREATS actually, you are right.

In CMOD Admin client, under [Application Group -> Storage Management -> Life of Data and Indexes] you can update the retention days to a higher value so that it synchs up with OAM retention. However, any new archive will store the new OAM objects with previous management class properties -- original retention days. If purge hold is temporary and before standard retention is reached the hold processing is removed, you are good and nothing you need to do in OAM; maybe undo the update only for the range of objects you originally updated and nothing other than that. If HOLD processing extends beyond the life of the standard life cycle processing in OAM, then you may lose newly added objects when they are up for deletion. You need to update newly added objects as well periodically to avoid losing them.
#zOS #Multiplatforms
#DB2 #OAM
#AFP #RiCOH AFP2PDF #SnowBound
#Finance #Telecom #Airlines
#ICN #IHS #WAS ND #Cert and Key Management
#Migrations #Data Modeling #RACF-2-CMOD Synch
#FileTek AMMO #ABI #RMDS #RADAR

rjchavez

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: Enhanced Retention Management and expiration
« Reply #7 on: December 22, 2020, 01:32:45 PM »
I've run into a -803 when trying to update ODCREATS.  I created a global temporary table containing all the rows I wish to update in ODCREATS since there may be multiple objects I need to update.  When using an UPDATE statement and trying to SET ODCREATS = CURRENT TIMESTAMP + 5 YEARS, I receive an obvious +803 due to a UNIQUE index on the OSM_OBJ_DIR table with the ODCREATS column.  Since the UPDATE is going to process several rows from AND since the CURRENT TIMESTAMP + 5 YEARS value is going to be the same for every UPDATE, then hence the duplicate primary key on index OBJDIRX1.

Not sure how I can get around this unless updating a single row at a time(?)

Note I'm using DSNTEP2 to update the table and thus a single UPDATE statement execution.
« Last Edit: December 22, 2020, 01:36:03 PM by rjchavez »

Mehmet S Yersel

  • Jr. Member
  • **
  • Posts: 56
    • View Profile
    • LinkedIn Profile
Re: Enhanced Retention Management and expiration
« Reply #8 on: December 22, 2020, 02:40:52 PM »
I don't remember having this problem of duplicate index at all.

To avoid duplicate index, maybe you can try selecting the most recent range of OAM objects and updating their ODCREATS/ODPENDT first in order to avoid 5 years old objects colliding with them? For example, WHERE ODCREATS BETWEEN '2020-12-01' and '2020-12-21' for all objects created within the current month until end of day yesterday? Then you do the same for updating the month of November etc... make sure you also update ODPENDT = CURRENT DATE so that OSMC processing sets expiry dates according to these updated values during next cycle processing.

I remembered the reason I had to do it this way: multiple storage groups were using the same management class and retention change was not needed for all storage groups. I had collateral damage if I changed at management class level. In this case, if I changed the management class, all store groups were impacted. To avoid this collateral damage, this was the method I used to isolate the documents I needed to retain.

Maybe you don't need to make changes in object directory attributes if all store groups that use this management class are required to be retained exactly the same. If that's the case, you need to change the management class properties and that's it. And change it back when needed. Still you may need to manipulate the OSM Object Directory to force expire objects that will no longer needed to be retained when you undo this change because their ODPENDT will remain some future date no longer needed.

 
#zOS #Multiplatforms
#DB2 #OAM
#AFP #RiCOH AFP2PDF #SnowBound
#Finance #Telecom #Airlines
#ICN #IHS #WAS ND #Cert and Key Management
#Migrations #Data Modeling #RACF-2-CMOD Synch
#FileTek AMMO #ABI #RMDS #RADAR

rjchavez

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: Enhanced Retention Management and expiration
« Reply #9 on: December 23, 2020, 08:13:21 AM »
It would not matter if you use the BETWEEN clause since every row being updated would contain the same ODCREATS value.

I think a better solution for me would be to change the MANAGEMENT CLASS in the OSM_OBJ_DIR for each of the objects I"m interested in changing.  ODMCNUM can be matched to an entry in CBR_MGT_CLASS_TBL correspoinding to the new MANAGEMENT CLASS and thus new retention.  Your thoughts?

Mehmet S Yersel

  • Jr. Member
  • **
  • Posts: 56
    • View Profile
    • LinkedIn Profile
Re: Enhanced Retention Management and expiration
« Reply #10 on: December 31, 2020, 06:48:59 AM »
I think changing the management class for the impacted objects would work.

Once the ODPENDT is hit, OSMC will readjust the retention and a new ODPENDT will be set. 
#zOS #Multiplatforms
#DB2 #OAM
#AFP #RiCOH AFP2PDF #SnowBound
#Finance #Telecom #Airlines
#ICN #IHS #WAS ND #Cert and Key Management
#Migrations #Data Modeling #RACF-2-CMOD Synch
#FileTek AMMO #ABI #RMDS #RADAR