Ondemand - data restoration after removing app gr

[Maciej Mieczakowski]

Hi,

I have quite an interesting topic concerning data recovery due to the accidental removal of the application group.

As you are aware, removing an application group is not a common thing that admin does, but still it possible to remove application group quite easily after confirmation.
After that, all links to objects are removed from CMOD DB, and what's even worse, all objects are removed in a fly from cache/TSM in a fly. OnDemand sends delete filespace command to TSM. It removes the data on the primary and eventually the copy pools and there is no retention policies in effect at all at that point. The delete filespace command in TSM is under the understanding that you don't want the data anymore, so it is gone out of the TSM database.
It is still on the copy pool because the reuse delay on the data is a few days, but the only way to get it back it restoring the TSM database.
Restoring the TSM database to get a small piece of data back is not a good solution to this. I feel we shouldn't be restoring TSM databases for data retrieval unless it is a disaster recovery situation because it is a major undertaking to do it to another server.

I'm wondering why it is done like that. Why data is not marked for be removed after next run of arsmaint instead of removing all of it in a fly. I have a feeling that having it pushed through the general expiration policy would be a much more secure approach, letting you revoke the bad decision, but maybe there is something I'm not aware of.

What do you think about that? Have you ever deal with such issue before? Do you have any experience how to secure such scenario?

I'd appreciate getting some feedbacks on how is that solved on your end. I see no good approach here. I started a talk with IBM to understand what options we have and got a reply that it would require enhancement to the product. I'm not sure if this is in scope.

What is your point of view? Any comments highly appreciated 

[Stephen McNulty]

My feelings on the subject, are that deleting an application group through an admin interface is not part of normal maintenance.

as you stated " it possible to remove application group quite easily after confirmation"  and as an admin I want it to be easy at that point.  To make such a mistake a person had to confirm more than once...  logging into admin client, delete app group, provide admin id and password, then choose okay on panel stating how many objects are stored.

I am not sure how it can be accomplished to delete afterwards through the normal expiration policy (when you are deleting objects that do not fall into the expiration policy).

In the few cases i have seen this scenario, we would either, restore from the backups (assuming there was no activity after the deletion), or if no backup to restore from export the app group definitions  from UAT or backup servers and then spool up objects from backup and reload into production.





 

[Maciej Mieczakowski]

Hi,

We restored objects, it's not a point here. I'm rather talking about the general approach for removing data this way.
The expiration policy uses the number of days that passed after the segment date that is set for the most common LOAD type. For segment type, it's a different approach, but in both cases removing the application group cause that tables have to be removed from DB and all stored objects from TSM/cache.
So normal expiration policy could expire documents and remove the table later on, the same thing you can accomplish by typing the future date (that exceed a total amount of days to keep data that is set in AG) in arsmaint command for certain AG for instance, -t parameter. I don't see a problem here. For me, this mechanism is already in place.

Of course, it's a human mistake, some cleaning actions due to GDPR, after migrating some data to another archive system. One extra AG was removed. It could be restored more easily I think if data was not removed in a fly.
 It's not a common thing, but still, I think it's worth to discuss if there is any benefit from removing data in fly as it is done now.

I'm simply curious.

[Justin Derrick]

I'd have to say I agree with Stephen.  There's only so much the developers can do to protect you from yourself.  Requiring an admin login & password as confirmation of deletion was added after it was deemed 'too easy' to delete AGs.

Otherwise, this becomes a lesson in not granting admin access to people who don't have the appropriate skillset, or putting extra policy requirements in place when doing deletions (two admins present, and one overseeing the other).

-JD.

[Maciej Mieczakowski]

You are right that nothing can protect you from yourself, for me it was a good lesson & chance to validate our disaster recovery policies. I found out that it would be much faster when objects would not be removed in a fly, but it was in that only scenario when you remove AG. But technically the approach I described seems to be logic for me, regardless if there is a sense of implementing it into the product or not. This is what I wanted to ask you about because I don't see any more benefits for this instant removal, except that it is fast and definitive.

[Justin Derrick]

Consider adding it as an Enhancement Request. 

[Maciej Mieczakowski]

Good point, thanks for your all replies! 

[jsquizz]

Consider adding it as an Enhancement Request. 

Sounds like adding a recycling bin to cmod

Actually, a pretty good idea depending on the situation.