Reload AFP Data

[Michel de Kraker]

Hi All

Is there a way to reload AFP data? The reason is that at the moment AFP documents are loaded with an expiration time of 7 years. This should be 10 years. The TSM MGMTCLASS has a retention of 11 years.

I have tried it with arsdoc get. But that output cannot be reloaded in CMOD because the ind file is not correct.
It gives the message:
Invalid structured field header in the file markiemark.afp.2.P121.P121 - Coil Met Tests.ind
Loaded 0 rows into the database

I have noticed in de DB2 Tables there is a table called archive_date. This is the date which reflects the time of archiving the AFP document. Can this tables be updated with a new date? For instance change 08/12/2007 into 08/12/2010?

If this can be done then arsmaint will start deleting documents after 10 years instead of 7.

Or is there another way to reload AFP documents?

Thanx in advance for your help.

Kind regards,

Michel de Kraker

[pankaj.puranik]

Hi Michel

I guess you can hold the documents from being eligible for deletion instead of reloading them (which would be a bigger task). I am not much aware of this technique but you could probably get some help from the links below.
If you hold the documents from being eligible for deletion when arsmaint is run, you would need to make sure that you delete them after 10 years retention.

http://www.odusergroup.org/forums/index.php?topic=493.0
http://www.odusergroup.org/forums/index.php?topic=478.0

Cheers
Pankaj.

[Michel de Kraker]

Thx Pankaj for your quick response.

I have always understood that changing the life of data and indexex parameter was only for new to be loaded documents, not for the existing ones.

1) From a CMOD standpoint, you can use option of "Never Expire" under "Life of Data and Indexes" in the "Storage Management" settings of application group. This setting is retro-active. So, any data that was previous loaded will also be retained indefinitely.

But
if this is OK , then i am happy.

Greetings, Michel.

[Michel de Kraker]

Pankaj,

One final question about changing the life of data and indexes parameter.
At the moment for the AG this is set to: 7 YEARS. You say , when you change this parameter to never expired, this setting applies also to allready loaded documents. When i change 7 YEARS to 10 YEARS , does this have the same impact?

Kind regards,

Michel.

[pankaj.puranik]

I don't think this would work because when you talk about CMOD, it stores the document in c:/arscache/some folder. There will be multiple folders within this directory having names like 12345 (some number in arsdate format). This number in arsdate format represents the expiration date for the documents in that folder (12345). If you need I can provide more details on this.

So what I want to point out here is, the data that is already loaded would be in some folder (12345). Now since this data was loaded before you changed the 7 year retention to 10 years, this means all the previously loaded data will still be removed at the end of 7 years since the folder (12345) would already have been created considering 7 year retention.

Let me try to put it this way. Suppose you have put the retention as 2555 days (7 years).
When you load the data in cmod you get a load ID of the form 3425-1-0-1FAA-8765-10601.
Here 10601 is the date in arsdate format of the highest date in the report.
So CMOD stores the data in a folder with name (10601+2555) or 13156.

In your case several folders would have already been created considering 7 year retention.
Changing the retention period now to 10 years won't affect the previous data/folder names.

But this might not be true for the newer CMOD versions. I have heard that the new version takes into account the current settings for all previous loaded data also.

Hope this helps!

Thanks
Pankaj.

[Michel de Kraker]

Once again thx Pankaj.

Final question.

You talk about cache expiration. But what about the DB2 indexes and data. When is that deleted?
arsmaint does this job. what is the trigger for arsmaint to delete db2 information? Does arsmaint look at "Life of data and indexes" or does it looks at another parameter?

Thx

Michel.

[Justin Derrick]

Hi.  Sorry I'm late to the game here. 

Pankaj is right on both counts about cache expiration -- historically, cache expiration was set at load time -- to make maintenance fast and easy by simply deleting directories that contained data that should have expired.  Of course, CPU power and IO bandwidth have improved dramatically, making this method obsolete.  Today, and  for a few years now, CMOD has been inspecting the contents of the cache and determining what is due to be expired based on the CURRENT cache expiration settings in the OnDemand Client.

The same applies for DB2 index expiration.  When configured properly, CMOD will expire records based on the CURRENT settings of the Life Of Data And Indexes parameter.

Maybe we should make a guide of FAQ for this topic.     Does anyone else have any questions (or information to share?) about extending retention?

-JD.

[Michel de Kraker]

Ok Thx JD.

1 thing is not clear to me now 

You say: CMOD will expire records based on the CURRENT settings of the Life Of Data And Indexes parameter
Does arsmaint check the archiving date of the loaded document and compare it with the Life Of Data And Indexes parameter and based on that it will expire objects and indexes?

One other question: When i have a expiration type of segment, does it als unloads objects from the cache and TSM? According you my system log it only expires the segment in DB2.

Thx again,

Michel.

[sandeepveldi]

Michel,
"Cache expiration", "DB2 & TSM data expiration" are two independent items when it comes to setting the expiration period. The "Life of Data & Indexes" in Application group "Storage Management" settings deals with "DB2 & TSM Data expiration". Cache expiration depends on the "number of days" you are setting up for expiration.
Starting with v8.4, both "Cache expiration" & "DB2 & TSM data expiration" are retroactive. Note that the expiration will depend on the load date when you don't have a date field which is defined as "segment" in the application group. If there's a segmentation date field defined in the application group, the expiration depends on this date field.

Data in cache is maintained in multiple filesystems based on the configuration done. cache1 filesystem contains softlinks to the compressed data objects available in other cache filesystems. It also, has expr and migr directories under it. In the expr directory, there will be sub-directories which have the names of dates in OD format. And these date sub-directories contain softlinks to CMOD compressed data objects. These date based directories are used for data expiration in cache once that date is reached.

TSM expiration depends on multiple factors. Please refer to http://www.odusergroup.org/forums/index.php?topic=478.0

DB2 & TSM expiration happens when you run the arsmaint -d. CMOD will wait for a confirmation from TSM that data is deleted and then only delete the data in DB2.

Feel free to ask more questions on this.

Sandeep Veldi

[RHPharr]

For the original question:

Yes you can reload previously parsed data.

Extract the original load as generic:
   arsdoc get -h {od_inst} -u {user} -p {pass} -G {AppGroup} -X {loadid} -cgNa -o {myfilename.ARD}

Then you can do a "force generic load" into an existing application.

  arsload -h {od_inst} -u {user} -p {pass} -vf -X G -g {AppGroup} -a {app}  {myfilename.ARD}

Depending on your version, the extract may add this line to the .ind file :
IGNORE_DATA_EVAL:1

but the loader does not recognize it  (Invalid generic index file format: >IGNORE_DATA_EVAL:1<).
Just edit the .ind file and strip it out.

[Michel de Kraker]

Thanks!

I will try this and let you know.

Michel.