OAM database error

[Greg Ira]

Anybody run into the following issue with their OAM database?  Sadly, due to circumstances out of our control, we are way out of support on both CMOD and DB2 so we have no help from IBM forthcoming.  Shot in the dark that someone else has run into this.
  We have an instance that is loading a lot of PDFs through a windows loader into our CMOD instance on z/OS.  Everything is going fine until, after only a few weeks and only a few million documents, we hit this:

ARS0430E LOAD40PW OAM Error: ARGMVSST: 0000000b(0000000c-8802fc78) OND.  426
COLL.FOREVER1.SGD4099B.MCDY99(CAA.R7294) OAM OSREQ MSGAREA:  DSNT408I
SQLCODE = -904, ERROR:  UNSUCCESSFUL EXECUTION CAUSED BY AN
UNAVAILABLE RESOURCE. REASON 00C900A4, TYPE OF RESOURCE
00000200, AND RESOURCE NAME OBD4099B.OSMOTS32   

DSNT501I  +ONDDODP DSNISGPI RESOURCE UNAVAILABLE  709
           CORRELATION-ID=OND40P                     
           CONNECTION-ID=DB2CALL                     
           LUW-ID=DODP.DODPLU1.DBEDD5A785C4=172377   
           REASON 00C900A4                           
           TYPE 00000200                             
           NAME OBD4099B.OSMOTS32                     

Our DBAs are very inexperienced and have been unable to diagnose the issue.  Our solution was to create a new OAM database which, as mentioned, only lasted a couple weeks and now we find ourselves with the same issue.  Available storage is way more than needed so were having a hard time figuring out why OAM cannot find enough available segment space.  We have a half dozen other instances on our system that have been loading this way for years with no issues.  If anybody has run into something similar I would appreciate it if you could share your findings.

CMOD on z/OS V10.1
DB2 11
z/OS V2.4

[Ed_Arnold]

This has worked for others:

You can try to reclaim space by either REORGing or perhaps allocating the tablespace with COMPRESS YES.

Ed

[Greg Ira]

Thanks Ed,
We did try that,  the DBAs failed to get a reorg to work after multiple attempts.  The docs are kept forever so I don't think there would be much space to reclaim anyways.
  I had to roll up my sleeves and restore my DB2 knowledge from my brain archives to dig down and find the issue.  What it looks like happened is when the DBA created the OAM database he created a relatively small simple database, particular the 32K tablespace.  When the users started hammering us with PDFs it filled the 32K table quickly and we hit the 32 dataset limit for the 32K tablespace OSMOTS32.  He repeated the same definition for our second OAM database so that filled up just as fast.
  I ended up having him define the next database using a partitioned tablespace for OSMOTS32 using VSAM extended in the back end and allowing 128 partitions.  Based on my estimates that should get us through a little more than half the year before we need another one.  We can live with that.
Figured I'd update just in case someone else runs into this.

[Mehmet S Yersel]

Do you have Virtual Tape (or TAPE-less TAPE)? You can make changes to your management classes assigned to the OAM collections such that data stays on DASD for ## days and migrates to TAPE (which happens to be TAPEless TAPE) after that. By Migration of OAM objects to TAPE, you are suddenly infinitely scalable; yet, you will need to take care of the directory tables in DB2. They need to be partitioned to not get stuck with 64 GB limit there too if you have that much growth.

It has been a few years I have not worked on OAM, but when I worked, I had even written utilities to merge OAM instances live while users were accessing them; I hope there has not been too many new things to invalidate what I said above. If so, I am sorry; I tried to help.