Loads Failing

[tbe2856]

I am wondering if someone might be able to help with these load failures. We are seeing load failures with the messages below.  We are seeing this on multiple jobs.  This one example This is an 8.5 server

ARS0024E $LFWJD Object >442679FAAB< in Application Group >RBA< not  888   
found in node >ODMP.LL01.SGLL01.MLL01<  Srvr->  non-SSL<-                                                             

ARS0020E $LFWJD SM Error: ARSMVSRE: 0000000b(0000000c-6c030910) ODMP.  889
LL01.SGLL01.MLL01(RBA.L442679.FAAB), Return Code=11, Reason=0, File=     
arssmsms.cpp, Line=1586  Srvr->Server IPAddress non-SSL<-       



ARS0024E $LFWJD Object >444689FAAB< in Application Group >RBA< not  907   
found in node >ODMP.LL01.SGLL01.MLL01<  Srvr->Server IPAddress non-SSL<-                                                             

ARS0020E $LFWJD SM Error: ARSMVSRE: 0000000b(0000000c-6c030910) ODMP.  909
LL01.SGLL01.MLL01(RBA.L444689.FAAB), Return Code=11, Reason=0, File=     
arssmsms.cpp, Line=1586  Srvr->Server IPAddress non-SSL<-       



ARS0024E $LFWJD Object >442679FAAB< in Application Group >RBA< not  092   
found in node >ODMP.LL01.SGLL01.MLL01<  Srvr->Server IPAddress non-SSL<-                                                             

ARS0020E $LFWJD SM Error: ARSMVSRE: 0000000b(0000000c-6c030910) ODMP.  093
LL01.SGLL01.MLL01(RBA.L442679.FAAB), Return Code=11, Reason=0, File=     
arssmsms.cpp, Line=1586  Srvr->Server IPAddress non-SSL<-       



ARS0024E ARSOND Object >9< in Application Group >RBA< not found in  644   
node >ODMP.LL01.SGLL01.MLL01<  Srvr->Server IPAddress non-       
SSL<-                                                                     

ARS0020E ARSOND SM Error: ARSMVSRE: 0000000b(0000000c-6c030910) ODMP.  645
LL01.SGLL01.MLL01(RBA.R9), Return Code=11, Reason=0, File=arssmsms.       
cpp, Line=1586  Srvr->Server IPAddress non-SSL<-                 

ARS0088E ARSOND Application Group Failed Load: Name(LL01) LoadId()  649   
File(//ars4/tmp4/LL010306.3083.LBSW500.SYS00010.LL01.191007.             
1716060000008495.ARD)                                                     

[tbe2856]

One thing I forgot to add was that this particular job has been loading successfully prior to this weekend.  The job runs every weekend.

[Nolan]

Check the Z/OS logs at load time for an SMS message explaining why it doesn't like the collection name.   Was there any change to SMS routines over the weekend?  You sure your report specification exit is putting the correct APP/AG and collection node to use?

You should also move to a new version of OnDemand to get IBM support.

[Justin Derrick]

Original post was edited (by me) to remove identifying information.

-JD.

[tbe2856]

We were thinking that the issue could possibly have been space related.  We did do a reorg of the databases.  Some of the messages related to RBA not found are no longer coming out.  The only one that remains is the one for ODMP.LL01.SGLL01.MLL01(RBA.R9).  I seem to recall an issue a while back about a resource that was not available or had been deleted and something about OnDemand would reload it but I do not remember the specifics.  I do see an entry in the DLL01.OSM_OBJ_DIR database for ODNAME=RBA.R9 but I don't see an entry in either of the DLL01.OSM_04K_OBJ_TBL or DLL01.OSM_32K_OBJ_TBL databases with a matching entry for OTNAME=RBA.R9.  Could this be the issue.

We are working on upgrading from 8.5.

Thanks
Tom



ARS0024E ARSOND Object >9< in Application Group >RBA< not found in  152   
node >ODMP.LL01.SGLL01.MLL01<  Srvr->Server IPAddress non-       
SSL<-                                                                     

ARS0020E ARSOND SM Error: ARSMVSRE: 0000000b(0000000c-6c030910) ODMP.  153
LL01.SGLL01.MLL01(RBA.R9), Return Code=11, Reason=0, File=arssmsms.       
cpp, Line=1586  Srvr->Server IPAddress non-SSL<-                 

ARS0088E ARSOND Application Group Failed Load: Name(LL01) LoadId()  154   
File(//ars4/tmp4/LL010306.3083.LBSW500.SYS00010.LL01.191008.             
1239310000008495.ARD)                                                     

[Justin Derrick]

Post modified AGAIN to remove identifying information.

tbe2856:  Please anonymize the information you cut-and-paste into the forums BEFORE you post it!  We're trying to keep you from leaking information about your installation.

[tbe2856]

Now we are seeing errors on prior reports that were loaded . . . .

Different RBA not found

ARS0020E $XTBE SM Error: ARSMVSRE: 0000000b(0000000c-6c030910) ODMP.  282
LL01.SGLL01.MLL01(RBA.L448816.FAAA), Return Code=11, Reason=0, File=     
arssmsms.cpp, Line=1586  Srvr->Server IPAddress non-SSL<-       

Seems to be isolated to Application Group   LL01.  And the old reports back prior to 8/17/2019 seem to be getting the error.

Thanks
Tom

[Ed_Arnold]

Hmmm -

Section 12.4 OAM Macro Return and Reason Codes
 
Table 93. OAM Macro Reason Codes
........................................................................
 
Return                Reason
Code     Error Code   Code     Description
........................................................................
X'0C'    Hardware     X'0910'  Specific request for unreadable volume. 
         resource
         unavailable

Suggest you open a PMR with OAM L2.

Ed

[jmimi]

Hello Ed and CMOD users on z/OS.

We have the same problem for 2 days ago…

ARS0430E M345909 OAM Error: ARSMVSRE: 0000000b(0000000c-6c030910)
OAMPROD.OAMPR06(GGA.L187.FAAE), Return Code=11, Reason=0, File= 
arssmsms.cpp, Line=1893  Srvr->IpAddress non-SSL<-     

Did you have any experience return for this issue ?
A PMR has been opened but for now, no solution returned yet….

Thanks for your help,
Best regards,
Jean-Michel

[Justin Derrick]

So, this is the THIRD time I've had to edit posts to remove identifying information from logs.

PLEASE REMOVE IP Addresses and Hostnames from the logs you post! 

-JD.

[Ed_Arnold]

I shamelessly stole this writeup from a colleague on the OAM reason code:

PROBLEM DESCRIPTION:
ARS0020E NDCTALF SM Error: ARSMVSRE:
0000000b(0000000c-6c030910)
 
 
 
 
The 0000000B indicates that on an attempt to retrieve the document in
question, OAM is returning an error as indicated by the return and
reason codes (0000000C-6C030910).
 
The meaning of those return and reason code can be found in Chapter 12
OAM Diagnostic Aids, of the DFPSMSdfp Diagnosis guide located in the
z/OS 1.10 Library Center at the following url(this is the older one,
the 1.12 one is not accessible):
 
http://publibz.boulder.ibm.com/bookmgr_OS390/libraryserver/zosv1r10/
 
And summarized here:
 
Section 12.2 OSREQ Return and Reason Codes
 
 Table 89. OSREQ Return and Reason Codes
 
........................................................................
 Return        Reason Code           Description
 Code          (Bytes)
               0      1    2    3
.......................................................................
 12 (X'0C')    X'6C'  x    y    z    OAM address space detected failure.
                                     yz   LCS reason code. See
                                     "OAM Macro Return and Reason Codes"
                                     in topic 12.4 and look for the
                                     associated OAM macro reason code.
                                     The rreturn code 12 is an OSREQ
                                     return code, and may not match the
                                     return code for the OAM macro.
........................................................................
 
 
Section 12.4 OAM Macro Return and Reason Codes
 
Table 93. OAM Macro Reason Codes
........................................................................
 
Return                Reason
Code     Error Code   Code     Description
........................................................................
X'0C'    Hardware     X'0910'  Specific request for unreadable volume.
 
         resource
         unavailable
 
 
To confirm that the failure can be isolated to what is occurring in OAM,
please issue the following requests in TSO:
 
TSO OSREQ QUERY <collection name> <object name>
 
TSO OSREQ RETRIEVE <collection name> <object name>
 
The collection and object names to specify in the above commands can be
found in the ARS0020E message itself. For example, the following
complete ARS0020E message was issued for another customer's similar
issue:
 
ARS0020E O09402 SM ERROR: ARSMVSRE: 0000000B(0000000C-6C030910)
IODC.R90D0442.SGROUP10.OD10YEAR(JQA.L53.FAAA), RC=11, REASON=0, FILE
=ARSSMSMS.CPP, LINE=917  SRVR->MY.SRVR.FOO n.n.n.n<-
 
In that case, IODC.R90D0442.SGROUP10.OD10YEAR was the collection name
and JQA.L53.FAAA was the object name.
 
 
I also anticipate that the OSREQ RETRIEVE request will fail with the
same return and reason codes as when the retrieve request is issued
through OnDemand.
 
This may actually be the result of just one unreadable tape rather than
the backup tapes/volume not being available in general. Here is what the
OAM support team recommended it further diagnose the failure in the
other customer's case:
 
........................................................................
  You can issue the D SMS,VOL(volser) command to check for the volume
status or do a SPUFI on the volume to see if it has been set to readable
= NO.  In any case, OAM provides an UPDATE VOLUME command that should be
used to update the VOLUME or TAPEVOL DB2 Tables.  Here is an example of
the OAM Update volume command:
 
       F OAM, UPDATE,VOL,volser,READABLE,Y
.
  This command can be used to change other setting such as writable,
etc.   Please review the OAM PISA for Object support on more information
about the OAM update volume command.
........................................................................
 
 
In that case, the customer was able to set the volume to readable, but
found that they actually needed to do a complete volume recovery on the
primary tape volume.   When attempting to recover the tape, they further
found that some of the backup tapes were also bad and not readable.
They cleared the errors on the backup tapes and were then able to
restart the volume recovery on the primary volume.
 
In your case, please issue the indicated OSREQ and D SMS commands and
report the results. If it is the anticipated backup volume that is
triggering the failure, try the F OAM command to set the volume to
readable. If that does not work, the volume may need a
complete recovery.

If you still need help you need to talk to OAM support.

Ed

[Justin Derrick]

Ed, you're a superstar. 

-JD.

[Ed_Arnold]

Ed, you're a superstar. 

-JD.

Ha!  I'd love to take credit for that write up but credit has to go to my very thorough colleague, Trina.

Ed