Author Topic: List of Performance Related Fixes plus a Brief Checklist, Upgrading CMOD on z  (Read 8076 times)

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1140
    • View Profile
JES2 Performance Improvement Available Starting In z/OS V2.1
« Reply #15 on: February 05, 2020, 01:54:39 PM »
ARSYSPIN and ARSLOAD when loading directly from spool use the SAPI interface.

A CPU savings and performance improvement was made available starting in z/OS V2.1.

Quote
In z/OS 2.1 the code was updated to improve overall performance for SAPI
devices. Two new tree structures are being used to improve the
performance of processing SAPI requests. One tree is used to organize the
JOEs based on various characteristics. This new tree is stored in the JES2
checkpoint for use by all members (2.1 and later). The other tree organizes
the SAPI devices waiting for work to process. That tree is maintained in
private JES2 storage. The creation and maintenance of each tree is
controlled by new keywords on the OUTDEF statement. These can be set
at initialization or via operator command. Turning these options on and off
will create and destroy the tree.

The trees are organized based on fixed characteristics that have traditionally
been used to process output groups. These are queue, route code and
OUTDISP. If you are not selecting on these characteristics (or any of the
combinations above), then these tree structures may not be for you. There
is a cost to maintain the trees that could be greater than the benefits
provided by the tree structure. If this is the case in your environment, you
may want to consider restructuring how you select output to take advantage
of the benefits of the tree structure.

Note that if you are selecting held output, the tree structure helps organize
the held output on your system. This may alter the order that output is
processed but it provides significant improvement over the current selection
process. It is still not recommended to select held output due to
performance cost, this change goes a long way to improve that environment.”


Ref:   Starting at page 29 of the following SHARE presentation:

https://share.confex.com/share/120/webprogram/Handout/Session13026/JES2%20Performance%20Considerations.pdf

Ref: JES2 pub

https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/sapipost.htm

Ref:  JES2 pub

https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.hasa300/sapienable.htm


A few things of note:

1. You can turn this on and off with the $TOUTDEF operator command.

$DOUTDEF                                                         
$HASP836 OUTDEF 107                                             
$HASP836 OUTDEF  COPIES=255,DMNDSET=NO,JOENUM=8000,JOEFREE=7088,
$HASP836         JOEWARN=80,OUTTIME=CREATE,PRTYLOW=0,           
$HASP836         PRTYHIGH=255,PRTYOUT=NO,PRYORATE=0,SEGLIM=100, 
$HASP836         STDFORM=STD,USERSET=NO,JOERBLDQ=NONE,           
$HASP836         DSLIMIT=10M,LDEV_OPT=NO,SAPI_OPT=YES,WS_OPT=YES


2. “This can alter the order in which output is presented to the SAPI application”.   So when it's enabled arsload/arsyspin might pick files in a different order.

3. "What is even more significant, is the benefit when a device reaches the end of the element to process.  In this case, it generally takes longer to discover that there is no work than to discover that there is work."

CMOD development does not have their own benchmark figures, but also doesn't doubt that this works exactly as advertised.

Ed







#zOS #ODF

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1140
    • View Profile
Don't Overlook The Benefit of V1.5 zFS's
« Reply #16 on: February 12, 2020, 02:26:11 PM »
Symptom: Slow loading from USS.  Throughput not what was expected.

Cause:  large V1.4 format zfs at /foo/retr/SL/DOC/ .  (Define large:  64 meg in size)

This file was getting searched sequentially for each load.

Resolution:  Stopped all loading.  Ran zfsadm convert –path /foo/retr/SL/DOC to convert that to a V1.5 format zFS.  Then resumed loading.

That zFS is now indexed.  Searches are no longer sequential.

Ref:  https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.ioea700/zfsconvert.htm

The conversion of that file took less than 1 minute.

Result:  Magic.  Loads ran several times faster instantly.

Lesson learned:  Convert all zFS files to V1.5. 

Ed
« Last Edit: February 13, 2020, 07:51:49 AM by Ed_Arnold »
#zOS #ODF

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1140
    • View Profile
OA56836: MINOR IMPROVEMENT TO SYSZTIOT CONTENTION ON TAPE REQUESTS
« Reply #17 on: February 25, 2020, 08:23:23 AM »
https://www-01.ibm.com/support/docview.wss?uid=isg1OA56836

****************************************************************
* USERS AFFECTED:                                              *
*  OAM objects users utilizing the IBM TS7700                  *
*                 Virtualization Engine D/T3957 that see high  *
*                 SYSZTIOT contention. Though this references  *
*                 the TS7700, it may also help with other      *
*                 tape vendors where SYSZTIOT contention is    *
*                 observed.                                    *
****************************************************************
* PROBLEM DESCRIPTION:                                         *
*  During heavy OAM workload processing                        *
*                      (tape retrieves), SYSZTIOT contention   *
*                      observed slowing down retrieval rate.   *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************


Quote
{Note: OAM was not intended to primarily store objects with
high expected retrieval rates (or in other words a "hot" object)
exclusively to tape without first going to DB2 DASD tier.

DB2 DASD should be used as the preferred resting place for a hot
object for x amount of days, where x is the number of days this
object is intended to be in a hot state.

This is recommended
since DASD will give substantially faster retrieval performance
for that object with a lot less overhead than tape library
hardware.

Even if the tape library is backed by faster storage
(i.e. Flash), the tape library logic still has a lot of
overhead and is serialized by a SYSZTIOT resource whereas the
DB2 DASD tier has minimal logic overhead and allows for
multi-threaded processing enabling parallelism that
dramatically increases retrieval rates.

Once the object
retrieval rate for this object starts becoming more spread out
over a longer period of time (or in other words, the object
becomes "cold/colder") then it can be moved to a colder OAM
tier such as tape.

Ed

#zOS #ODF

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1140
    • View Profile
ICSF Issue
« Reply #18 on: July 27, 2020, 01:54:06 PM »
Symptom - switched CMOD workload to a new processor, saw unexplained increased use of CPU by CMOD

Cause - hardware. CSF accelerator card not coming up in time to be recognized as expected

How to tell if you have the problem:

Look at the CSF started task log

You should see this message:

CSFM111I   CRYPTOGRAPHIC FEATURE IS ACTIVE. coprocessor-name cii, SERIAL NUMBER nnnnnnn.

before this message:

CSFM015I   FIPS 140 SELF CHECKS FOR PKCS11 SERVICES SUCCESSFUL.
______________

If those messages are out of order then software emulation of the hardware accelerator is likely being performed.

No fix yet.

If you have this symptom you'll want to contact CSF support.
#zOS #ODF

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1140
    • View Profile
ARSLOAD CPU savings - non-SSL environment
« Reply #19 on: August 27, 2020, 09:28:24 AM »
****************************************************************
* USERS AFFECTED: All Content Manager OnDemand for z/OS 10.1   *
*                 and above.                                   *
****************************************************************
* PROBLEM DESCRIPTION: 1.Excessive CPU noticed when running    *
*                      arsload in a non-SSL environment.  A    *
*                      performance monitor will show CPU being *
*                      consumed in module GSKC64F.  GSKC64F    *
*                      is a DLL for z/OS Cryptographic         *
*                      Services System SSL.                    *


Improvement is via PH28123 for both 10.1 and 10.5

More detail at:

https://www.ibm.com/support/pages/node/6262509?mhsrc=ibmsearch_a&mhq=ph28123

Ed
#zOS #ODF

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1140
    • View Profile
The ARSRPT function is very useful for taking a look at what is going on within ARSSOCKD.

White Paper here:

https://www.ibm.com/support/pages/arsrpt-%E2%80%93-content-manager-ondemand-enhanced-reporting-utility

Note:  you have to be logging certain messages (mentioned in the White Paper) for ARSRPT to be able report on.


Pro tip:  If ARSRPT says that ODWEK is doing a lot of logon/logoffs, perhaps the ODWEK application is not using connection pooling.

Connection pooling is very much recommended for efficient processing.

Ed
#zOS #ODF

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1140
    • View Profile
Noting that although this thread hasn't been updated in over a year, it's still being maintained.

Just nothing new under the sun.

Make sure you have all of those z/OS system parameters that improve Db2 performance integrated when rolling out the next release of z/OS.

Ed
#zOS #ODF