OnDemand User Group
Support Forums => z/OS Server => Topic started by: Ed_Arnold on June 19, 2015, 09:46:09 AM
-
I was asked for a brief checklist for items in addition to the migration instructions, sharing it here:
- If migrating to V8.5 or above for the first time, are you comfortable with the ARS_ORIGINAL_CODEPAGE setting you'll be using?
- do you understand why you see this message ARS0220I Server code page is 1200 ? Note that your ARS_ORIGINAL_CODEPAGE setting is probably not 1200.
- are you using the new ARSSOCKD parms of "-S -I [instance] -v"?
Reference:
http://www.odusergroup.org/forums/index.php?topic=762.0 (http://www.odusergroup.org/forums/index.php?topic=762.0)
- Is this CMOD instance on the same LPAR or on a different LPAR than your test system? If on a different LPAR, ensure that your OMVS settings are the same or greater on the new LPAR. (operator commands D OMVS,L and D OMVS,O to check)
- If not already in production, have all DB2 PTFs from the following web pages at least received into your DB2 SMP/E:
http://www.odusergroup.org/forums/index.php?topic=1525.0 (http://www.odusergroup.org/forums/index.php?topic=1525.0)
http://www.odusergroup.org/forums/index.php?topic=1318.0 (http://www.odusergroup.org/forums/index.php?topic=1318.0)
- Have all CMOD PTFs received into your CMOD SMP/E. Ensure you have received the CMOD PTF for APAR PI42688 which corrects an error in an error message
- the most common error reported to Level 2 after an upgrade from V8.4.1 or before to V8.5 or above is the restriction that the OS/390 Indexer will now report when an index value is not found and no default value is specified.
The error message you receive when encountering this will be, for example "8 fields submitted by the indexer, 12 expected"
Ed
-
Thanks Ed!
-
- PI10044 contained two different improvements:
(a) better storage management. no action required to exploit.
(b) delays due to lock contention. The following ACTION must be taken
to improve performance:
If installing on CMOD 8.5, the SARSINST(ARSUSECX) must be
re-assembled and re-linked after the APPLY.).
- PI15871 The ARSLOAD executable is larger than necessary. This results
in increased CPU and EXCPs when ARSLOAD is fetched into memory.
Action required: None. Just APPLY the PTF
- PI15930 High CPU ARSYSPIN and ARSLOAD is being fetched into storage
every time it was ATTACHed.
Action required: None. Just APPLY the PTF
- PI22584 ODF performance improvement, reduce the number of program
fetches for cobol exits.
Action required: None. Just APPLY the PTF
- PI20473 which is a DB2 APAR whose symptoms have been unearthed by CMOD
- (Added 11/2/2015) PI50742 390 INDEXER PERFORMANCE IMPROVEMENT.
Action required: None. Just APPLY the PTF
- (Added 12/8/2015) 9.5.0.3 rollup PTF includes exploitation of hardware compression/decompression
Action required: None. Just APPLY the PTF. However, it does require a certain level of CPU.
(Updated 2/25/2016) 9.5.0.4 changes the action required extensively. You must read the following technote in order to implement h/w compression:
http://www-01.ibm.com/support/docview.wss?uid=swg27046215 (http://www-01.ibm.com/support/docview.wss?uid=swg27046215)
-
Modifying a post doesn't bump this topic up on the age list.
So this is a bump as I feel it's worth raising the visibility of the List of Performance Related Fixes.
Ed
-
Bump for the h/w compression update rolled out in 9.5.0.4.
Ed
-
One of the changes starting in CMOD V9.0.0 is that the ARSANN table now contains LOBs which can possibly waste a lot of space.
For more consult the following...
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W7c1e3103820b_4d9e_add6_b5ade3280dcb/page/Inline+LOBs+%28Large+Objects%29
-
If you haven't already, consider moving to Universal Table Spaces (UTS). See the DB2 Knowledge Center for more information.
http://www.ibm.com/support/knowledgecenter/SSEPEK_10.0.0/intro/src/tpc/db2z_universaltablespaces.html (http://www.ibm.com/support/knowledgecenter/SSEPEK_10.0.0/intro/src/tpc/db2z_universaltablespaces.html)
The essential information from the Knowledge Center is:
You can alter existing table spaces to universal table spaces by using the ALTER TABLESPACE statement. If your database contains any simple table spaces, you should alter them to universal table spaces as soon as possible.
Related to this is a recommendation received:
IF you have a high number of applications per application group (we've seen this work particularly well when it's a few hundred)
AND IF you see a high number of DB2 getpages after moving to 9.5 or above
AND IF you see large increase in CPU consumption in the DB2 address space after moving to 9.5 or above
THEN moving the ARSAPP table to a Universal Table Space (UTS) and implementing inline LOBs is recommended.
http://www.ibm.com/support/knowledgecenter/SSEPEK_10.0.0/perf/src/tpc/db2z_improveperformanceoflobdata.html
-
At its heart, CMOD is a DB2 application.
There are two operating system parms that I'm aware of to benefit DB2.
I'm not going to go into detail on them as they're easily google-able.
1. USEZOSV1R9RULES(NO)
Set via your DIAGxx.
Can also be implemented dynamically with a SET DIAG=xx command.
To see if you have this enabled use the operator command D DIAG
D DIAG
IGV007I 12.36.01 DIAG DISPLAY
...
VSM USEZOSV1R9RULES(NO)
...
2. MEMDSENQMGMT: ENABLE
Set via your ALLOCxx
Can also be implemented dynamically using SETALLOC SYSTEM,MEMDSENQMGMT=ENABLE
To see if you have it enabled use the operator command D ALLOC,OPTIONS
D ALLOC,OPTIONS
IEFA003I 13.19.40 ALLOC OPTIONS
SPACE foo
UNIT foo
TIOT foo
SDSN_WAIT foo
VOLUME_ENQ foo
VOLUME_MNT foo
SPEC_WAIT foo
ALLC_OFFLN foo
CATLG_ERR foo
2DGT_EXPDT foo
VERIFY_VOL foo
SYSTEM foo
MEMDSENQMGMT: ENABLE
Note: if implementing dynamically I believe you need to bring up DB2 after the parms are set.
I have no benchmarks, sorry.
Ed
-
CMOD has its own defaults for LE storage settings such as HEAP and HEAPPOOLS.
If you feel you must override the default settings be sure to do the LE storage tuning exercise.
Note: the defaults are bigger in V10.1 than V9.5 which are bigger than the defaults in V8.4.1.
Use the LE option RPTOPTS(ON) to determine what your settings are and where they're set. RPTOPTS(ON) costs nothing in performance, just writes the options in use to SYSOUT.
One way to specify RPTOPTS(ON):
//ARSSOC95 EXEC PGM=ARSSOCKD,REGION=0M,TIME=NOLIMIT,
// PARM='RPTOPTS(ON)/-S -I ARCH950 -v'
Ed
03/10/2020 Update: Do not override CMOD V10 LE options unless you know exactly why. Testing has shown that with ARSSOCKD now being 64 bit, there should be no reason to override the default LE parms. If you're overriding LE in V9.5 or below, remove those overrides.
-
(Reference my previous post)
And just what are the LE settings set by the CMOD product?
Content Manager OnDemand for z/OS - Initial Recommended LE Settings
http://www-01.ibm.com/support/docview.wss?uid=swg22011284 (http://www-01.ibm.com/support/docview.wss?uid=swg22011284)
Note that bigger is usually better. I would be concerned only if I had overridden the default LE values to less than their defaults, especially for HEAP.
There is no substitute for doing the LE tuning exercise.
Ed
03/10/2020 Update: Do not override CMOD V10 LE options unless you know exactly why. Testing has shown that with ARSSOCKD now being 64 bit, there should be no reason to override the default LE parms. If you're overriding LE in V9.5 or below, remove those overrides.
-
What other performance enhancements should a shop be investigating?
1. For running in a SYSPLEX, the auto-tuning of the JES2 checkpoint first available in z/OS V2.2 is recommended:
https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/has2v5_automatic_approach.htm (https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/has2v5_automatic_approach.htm)
2. Ensure your COFVLFxx, at a minimum, has the following specified:
CLASS NAME(CSVLLA) /* Class name for Library Lookaside @P2C*/
EMAJ(LLA) /* Major name for Library Lookaside @P2C*/
CLASS NAME(IRRSMAP) /* OpenMVS_RACF SMAP Table */
EMAJ(SMAP) /* Major name = SMAP */
CLASS NAME(IRRGMAP) /* OpenMVS_RACF GMAP Table */
EMAJ(GMAP) /* Major name = GMAP */
CLASS NAME(IRRUMAP) /* OpenMVS_RACF UMAP Table */
EMAJ(UMAP) /* Major name = UMAP */
CLASS NAME(IRRGTS) /* RACF GTS Table */
EMAJ(GTS) /* Major name = GTS */
CLASS NAME(IRRACEE) /* RACF saved ACEEs */
EMAJ(ACEE) /* Major name = ACEE */
Ed
-
At z/OS V2.1 a new version or release was made for zFS V1.5 or V5 depending on who you're talking to.
I talked about this here:
www.odusergroup.org/forums/index.php?topic=1710 (http://www.odusergroup.org/forums/index.php?topic=1710)
There's a great testimonial on the web here:
https://groups.google.com/forum/#!topic/bit.listserv.ibm-main/CLBgzhjJPQk (https://groups.google.com/forum/#!topic/bit.listserv.ibm-main/CLBgzhjJPQk)
Ed
-
PH02490 / UI59445- V9.5 only, reduces internal ODF lock contention between ODF and load processing by reducing the length of time ARSLOAD is holding the internal ODF lock.
-
PH09636 / UI62202
No action required, just install the PTF.
Ed
-
OA55720 - HIGH CPU AND UNEXPECTEDLY LONG DURATION OF OSMC PROCESSING FOR
AN OBJECT STORAGE GROUP
ERROR DESCRIPTION:
The customer noticed that when they had an Object Storage Group
with a few hundred thousand objects eligible for transition
processing and the transition processing resulted in assigning a
management class whose transition criteria set the new pending
action date (ODPENDDT) to the current date or earlier, that the
processing caused high CPU and took hours to complete. When they
modified their ACS routines to instead assign a new managment
class whose transition criteria set the new pending action date
to the future, the CPU usage decreased dramatically, as well as
the overall processing time.
This issue was incorrectly addressed by FIN APAR OA44651 with
pending action date behavior changed within base V2R2 release.
OA44651 pending action behavior was then reverted by APAR
OA55718. Please see OA44651 and OA55718 for additional details.
Additional Keywords: OSMC CPU OAM OBJECT
LOCAL FIX:
Ensure that the management class being assigned during
transition processing sets the pending action date to a future
date, via either the transition criteria or the expiration
date in that management class.
OA57287 - OSMC PROCESSING SLOWDOWN FOR STORAGE GROUPS WITH LARGE AMOUNT
OF OBJECTS (APPRX. HALF A BILLION) AFTER OA55720.
ERROR DESCRIPTION:
After the application of APAR OA55720, OSMC processing slows
down when a particular storage group has a large number of
objects. Objects are processed successfully, but the processing
takes significantly longer. The customer noticed the difference
in processing time was significant for storage groups that had
approximately half a billion objects.
LOCAL FIX:
Do not start OSMC processing for storage groups with a large
amount of objects.
-
ARSYSPIN and ARSLOAD when loading directly from spool use the SAPI interface.
A CPU savings and performance improvement was made available starting in z/OS V2.1.
In z/OS 2.1 the code was updated to improve overall performance for SAPI
devices. Two new tree structures are being used to improve the
performance of processing SAPI requests. One tree is used to organize the
JOEs based on various characteristics. This new tree is stored in the JES2
checkpoint for use by all members (2.1 and later). The other tree organizes
the SAPI devices waiting for work to process. That tree is maintained in
private JES2 storage. The creation and maintenance of each tree is
controlled by new keywords on the OUTDEF statement. These can be set
at initialization or via operator command. Turning these options on and off
will create and destroy the tree.
The trees are organized based on fixed characteristics that have traditionally
been used to process output groups. These are queue, route code and
OUTDISP. If you are not selecting on these characteristics (or any of the
combinations above), then these tree structures may not be for you. There
is a cost to maintain the trees that could be greater than the benefits
provided by the tree structure. If this is the case in your environment, you
may want to consider restructuring how you select output to take advantage
of the benefits of the tree structure.
Note that if you are selecting held output, the tree structure helps organize
the held output on your system. This may alter the order that output is
processed but it provides significant improvement over the current selection
process. It is still not recommended to select held output due to
performance cost, this change goes a long way to improve that environment.”
Ref: Starting at page 29 of the following SHARE presentation:
https://share.confex.com/share/120/webprogram/Handout/Session13026/JES2%20Performance%20Considerations.pdf (https://share.confex.com/share/120/webprogram/Handout/Session13026/JES2%20Performance%20Considerations.pdf)
Ref: JES2 pub
https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/sapipost.htm (https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/sapipost.htm)
Ref: JES2 pub
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/sapienable.htm (https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/sapienable.htm)
A few things of note:
1. You can turn this on and off with the $TOUTDEF operator command.
$DOUTDEF
$HASP836 OUTDEF 107
$HASP836 OUTDEF COPIES=255,DMNDSET=NO,JOENUM=8000,JOEFREE=7088,
$HASP836 JOEWARN=80,OUTTIME=CREATE,PRTYLOW=0,
$HASP836 PRTYHIGH=255,PRTYOUT=NO,PRYORATE=0,SEGLIM=100,
$HASP836 STDFORM=STD,USERSET=NO,JOERBLDQ=NONE,
$HASP836 DSLIMIT=10M,LDEV_OPT=NO,SAPI_OPT=YES,WS_OPT=YES
2. “This can alter the order in which output is presented to the SAPI application”. So when it's enabled arsload/arsyspin might pick files in a different order.
3. "What is even more significant, is the benefit when a device reaches the end of the element to process. In this case, it generally takes longer to discover that there is no work than to discover that there is work."
CMOD development does not have their own benchmark figures, but also doesn't doubt that this works exactly as advertised.
Ed
-
Symptom: Slow loading from USS. Throughput not what was expected.
Cause: large V1.4 format zfs at /foo/retr/SL/DOC/ . (Define large: 64 meg in size)
This file was getting searched sequentially for each load.
Resolution: Stopped all loading. Ran zfsadm convert –path /foo/retr/SL/DOC to convert that to a V1.5 format zFS. Then resumed loading.
That zFS is now indexed. Searches are no longer sequential.
Ref: https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.ioea700/zfsconvert.htm
The conversion of that file took less than 1 minute.
Result: Magic. Loads ran several times faster instantly.
Lesson learned: Convert all zFS files to V1.5.
Ed
-
https://www-01.ibm.com/support/docview.wss?uid=isg1OA56836 (https://www-01.ibm.com/support/docview.wss?uid=isg1OA56836)
****************************************************************
* USERS AFFECTED: *
* OAM objects users utilizing the IBM TS7700 *
* Virtualization Engine D/T3957 that see high *
* SYSZTIOT contention. Though this references *
* the TS7700, it may also help with other *
* tape vendors where SYSZTIOT contention is *
* observed. *
****************************************************************
* PROBLEM DESCRIPTION: *
* During heavy OAM workload processing *
* (tape retrieves), SYSZTIOT contention *
* observed slowing down retrieval rate. *
****************************************************************
* RECOMMENDATION: *
****************************************************************
{Note: OAM was not intended to primarily store objects with
high expected retrieval rates (or in other words a "hot" object)
exclusively to tape without first going to DB2 DASD tier.
DB2 DASD should be used as the preferred resting place for a hot
object for x amount of days, where x is the number of days this
object is intended to be in a hot state.
This is recommended
since DASD will give substantially faster retrieval performance
for that object with a lot less overhead than tape library
hardware.
Even if the tape library is backed by faster storage
(i.e. Flash), the tape library logic still has a lot of
overhead and is serialized by a SYSZTIOT resource whereas the
DB2 DASD tier has minimal logic overhead and allows for
multi-threaded processing enabling parallelism that
dramatically increases retrieval rates.
Once the object
retrieval rate for this object starts becoming more spread out
over a longer period of time (or in other words, the object
becomes "cold/colder") then it can be moved to a colder OAM
tier such as tape.
Ed
-
Symptom - switched CMOD workload to a new processor, saw unexplained increased use of CPU by CMOD
Cause - hardware. CSF accelerator card not coming up in time to be recognized as expected
How to tell if you have the problem:
Look at the CSF started task log
You should see this message:
CSFM111I CRYPTOGRAPHIC FEATURE IS ACTIVE. coprocessor-name cii, SERIAL NUMBER nnnnnnn.
before this message:
CSFM015I FIPS 140 SELF CHECKS FOR PKCS11 SERVICES SUCCESSFUL.
______________
If those messages are out of order then software emulation of the hardware accelerator is likely being performed.
No fix yet.
If you have this symptom you'll want to contact CSF support.
-
****************************************************************
* USERS AFFECTED: All Content Manager OnDemand for z/OS 10.1 *
* and above. *
****************************************************************
* PROBLEM DESCRIPTION: 1.Excessive CPU noticed when running *
* arsload in a non-SSL environment. A *
* performance monitor will show CPU being *
* consumed in module GSKC64F. GSKC64F *
* is a DLL for z/OS Cryptographic *
* Services System SSL. *
Improvement is via PH28123 for both 10.1 and 10.5
More detail at:
https://www.ibm.com/support/pages/node/6262509?mhsrc=ibmsearch_a&mhq=ph28123 (https://www.ibm.com/support/pages/node/6262509?mhsrc=ibmsearch_a&mhq=ph28123)
Ed
-
The ARSRPT function is very useful for taking a look at what is going on within ARSSOCKD.
White Paper here:
https://www.ibm.com/support/pages/arsrpt-%E2%80%93-content-manager-ondemand-enhanced-reporting-utility (https://www.ibm.com/support/pages/arsrpt-%E2%80%93-content-manager-ondemand-enhanced-reporting-utility)
Note: you have to be logging certain messages (mentioned in the White Paper) for ARSRPT to be able report on.
Pro tip: If ARSRPT says that ODWEK is doing a lot of logon/logoffs, perhaps the ODWEK application is not using connection pooling.
Connection pooling is very much recommended for efficient processing.
Ed
-
Noting that although this thread hasn't been updated in over a year, it's still being maintained.
Just nothing new under the sun.
Make sure you have all of those z/OS system parameters that improve Db2 performance integrated when rolling out the next release of z/OS.
Ed
-
Starting in 10.1 a new sample arsusec is now available in 64-bit mode which should provide better efficiency. This is in the base code in V10.5.
The reason for the better efficiency is that the flow was previously:
Content Manager OnDemand -> ARSUSEC -> ARSUSECX -> dynamic exit facility -> ARSUSECZ
With the new 64-bit exit, extra code is no longer needed to translate 64 -> 31-bit instructions. The path now is simply:
Content Manager OnDemand -> ARSUSEC4
For further information:
http://www.odusergroup.org/forums/index.php/topic,2252.msg10902.html#msg10902
https://www.ibm.com/support/pages/node/1165276
Ed