31
z/OS Server / Re: How to trouble shoot CMOD OAM reports when they do not transition from OAM DASD
« on: July 01, 2020, 05:34:19 AM »
Hi Lizette -
Before you start, verify that the OAM storage/management class is defined to handle life-cycle as you expect (30 day DASD then 83 months on TAPE, etc). If not, this might be where the problem is. Assuming the storage/management class is defined correctly and assigned to the objects in question, then you should expect DFHSM to act accordingly.
To verify DFHSM works properly, you can run a query on the OAM Object Directory table and identify the objects (or object count) due for DFHSM processing. DFHSM started task will have 2 type of activities during its processing window:
1. Create backups of newly added objects or any objects so far not backed up yet (due to DFHSM window closing without going through all storage groups)
2. Migration (or expiration) of objects with ODPENDT =< CURRENT DATE
If there are no objects in any storage group that meet any of the 2 criteria above, there will be no action taken on that particular OAM Storage Group during DFHSM processing.
If any of the objects has not been backed up yet (secondary copy attributes blank (TAPE and VOLUME), can't remember the attribute names but if you pull the object directory structure, you can tell which one they are), or there are objects in need of DFHSM processing (ODPENDT <= today which means there is something in need of expiration or migration), then you should expect activity on that storage group.
If there are no errors, maybe there are other things such as a small processing window not sufficient to go through all the pending activities. Or, when CPU utilization is very high, DFHSM might not be able to get enough processing cycles and it may fall behind. Make sure to see DFHSM task visiting every storage group before it ends. If it ends and there are still objects to process, you may see a lag until you can catch up.
One final thing to check: If DFHSM procesisng is good and so far you have not seen any irregularities, check DB2 internals. Many years ago we had a DB2 table problem where deleted rows were not being physically deleted and the table size was constantly growing and adding new extents despite the fact that there were same number of deleted rows as newly added ones. It turned out to be a DB2 problem where space from deleted rows were not being reclaimed for use. OAM DASD sits on DB2 AUX tables. Make sure your DB2 tech team verifies that there is no DB2 problem and that if there are deletions, these deletions finally get deleted physically as well and don't stay there forever.
Good luck with your investigation of the root cause.
I hope this helps,
Mehmet
Before you start, verify that the OAM storage/management class is defined to handle life-cycle as you expect (30 day DASD then 83 months on TAPE, etc). If not, this might be where the problem is. Assuming the storage/management class is defined correctly and assigned to the objects in question, then you should expect DFHSM to act accordingly.
To verify DFHSM works properly, you can run a query on the OAM Object Directory table and identify the objects (or object count) due for DFHSM processing. DFHSM started task will have 2 type of activities during its processing window:
1. Create backups of newly added objects or any objects so far not backed up yet (due to DFHSM window closing without going through all storage groups)
2. Migration (or expiration) of objects with ODPENDT =< CURRENT DATE
If there are no objects in any storage group that meet any of the 2 criteria above, there will be no action taken on that particular OAM Storage Group during DFHSM processing.
If any of the objects has not been backed up yet (secondary copy attributes blank (TAPE and VOLUME), can't remember the attribute names but if you pull the object directory structure, you can tell which one they are), or there are objects in need of DFHSM processing (ODPENDT <= today which means there is something in need of expiration or migration), then you should expect activity on that storage group.
If there are no errors, maybe there are other things such as a small processing window not sufficient to go through all the pending activities. Or, when CPU utilization is very high, DFHSM might not be able to get enough processing cycles and it may fall behind. Make sure to see DFHSM task visiting every storage group before it ends. If it ends and there are still objects to process, you may see a lag until you can catch up.
One final thing to check: If DFHSM procesisng is good and so far you have not seen any irregularities, check DB2 internals. Many years ago we had a DB2 table problem where deleted rows were not being physically deleted and the table size was constantly growing and adding new extents despite the fact that there were same number of deleted rows as newly added ones. It turned out to be a DB2 problem where space from deleted rows were not being reclaimed for use. OAM DASD sits on DB2 AUX tables. Make sure your DB2 tech team verifies that there is no DB2 problem and that if there are deletions, these deletions finally get deleted physically as well and don't stay there forever.
Good luck with your investigation of the root cause.
I hope this helps,
Mehmet