arssockd failing to start with SSL port defined.

[R2D2]

Hi:
I am trying to implement SSL on an AIX CMOD server at release level 9.5.0.2.
The server works fine when not using SSL.
When using a SSL port defined in the ars.ini file the server fails to initialize and become active.
I have painstakingly followed instructions to create the key db, the stash file and a self-signed certificate.
The ring parameters in the ars.ini file are correct. Yet the server will not start.
I collected a detailed trace and the last few lines show the following:
13107364:1 05/01/2018 11:53:49:916911 FLOW arssrvr.c(5380)ArcSERVP_Srvr:Enter
13107364:1 05/01/2018 11:53:49:916998 FLOW arssock.c(3819)ArcSOCKET_ServerInit:Enter
13107364:1 05/01/2018 11:53:49:917009 INFO arssock.c(3830)ArcSOCKET_ServerInit:SOMAXCONN so_max=1024
13107364:1 05/01/2018 11:53:49:917017 FLOW arssock.c(3065)ArcSOCKETP_AllocSocketHandle:Enter
13107364:1 05/01/2018 11:53:49:917025 FLOW arssock.c(2488)ArcSOCKETP_Startup:Enter
13107364:1 05/01/2018 11:53:49:917047 FLOW arssock.c(2945)ArcSOCKETP_Startup:Return arccs return code=0,ARCCS_OKAY
13107364:1 05/01/2018 11:53:49:917056 FLOW arssock.c(3083)ArcSOCKETP_AllocSocketHandle:Return arccs return code=0,ARCCS_OKAY
13107364:1 05/01/2018 11:53:49:917065 INFO arssock.c(3882)ArcSOCKET_ServerInit:Setting up socket port_ptr=1456 use_ssl=1
13107364:1 05/01/2018 11:53:49:917073 FLOW arssock.c(2488)ArcSOCKETP_Startup:Enter
13107364:1 05/01/2018 11:53:49:922183 FLOW arssock.c(1715)ArcSOCKETP_GSKitAttributes:Enter
13107364:1 05/01/2018 11:53:49:922310 INFO arssock.c(1737)ArcSOCKETP_GSKitAttributes:GSKit Version version=8.0.14.43
13107364:1 05/01/2018 11:53:49:922323 INFO arssock.c(1806)ArcSOCKETP_GSKitAttributes:SSL SID Cache cache_timeout=86400 cache_size=512
13107364:1 05/01/2018 11:53:49:922331 INFO arssock.c(1822)ArcSOCKETP_GSKitAttributes:Keyring Info KeyRing File=/opt/IBM/ondemand/V9.5/config/ondemand.kdb KeyRing Stash=/opt/IBM/ondemand/V9.5/config/ondemand.sth KeyRing Label=CMODselfsigned
13107364:1 05/01/2018 11:53:49:922351 FLOW arssock.c(2032)ArcSOCKETP_GSKitAttributes:Return ssl_rc=0

The forum community here seems very knowledgeable so I thought it would be an excellent place to see if others have encountered this issue and could suggest debugging ideas.
BTW I have opened a PMR with IBM but have yet to receive a timely response. 

Thanks...........

[Justin Derrick]

Hi there!

Yes, SSL is tricky, *especially* with self-signed certificates.

The first suggestion I'd make is to update your version of IBM CMOD and the IBM Global Security Kit.  There are links FixPacks to CMOD & the GSKit on the CMOD wiki:  https://cmod.wiki/index.php?title=Main_Page#IBM_CMOD_Fixpacks_.26_Security_Bulletins ... or use the shorlink I've created http://cmod.co/fixpack .

The weird thing is, the return code from the SSL Library appears to be zero -- indicating that there wasn't an obvious error.  Can you outline the process you followed?

-JD.

[Ed_Arnold]

First off, I agree with Justin it's time to get current.

9.5.0.2 is a little old, 9.5.0.11 is current.

You've seen the steps I followed for z at http://www.odusergroup.org/forums/index.php?topic=1938 ?

What's in your ars.ini?

Ed

[R2D2]

Hi:
I followed instruction in the document authored by Greg Felderman.
https://cmod.wiki/dox/CMODv8.5/UsingSSLwithCMOD.pdf

The selfsigned certificate seems to valid as:
1: The RC for SSl is 0.
2. And the following GSK command work as expected.
gsk8capicmd_64 -cert -list -db ondemand.kdb
gsk8capicmd_64 -cert -details -db ondemand.kdb  -label "CMODselfsigned"

Here is ars.ini contents. I am using the ARCHIVE2 instance.
[@SRV@_ARCHIVE]
HOST=10.20.1.213
PROTOCOL=2
PORT=0
SRVR_INSTANCE=ARCHIVE
SRVR_INSTANCE_OWNER=root
SRVR_OD_CFG=/opt/IBM/ondemand/V9.5/config/ars.cfg
SRVR_DB_CFG=/opt/IBM/ondemand/V9.5/config/ars.dbfs
SRVR_SM_CFG=/opt/IBM/ondemand/V9.5/config/ars.cache
[@SRV@_ARCHIVE2]
HOST=10.20.1.213
PROTOCOL=2
PORT=1455
SSL_PORT=1456
SRVR_INSTANCE=ARCHIVE2
SRVR_INSTANCE_OWNER=root
SRVR_OD_CFG=/opt/IBM/ondemand/V9.5/config/ars.2.cfg
SRVR_DB_CFG=/opt/IBM/ondemand/V9.5/config/ars.2.dbfs
SRVR_SM_CFG=/opt/IBM/ondemand/V9.5/config/ars.2.cache
SSL_KEYRING_FILE=/opt/IBM/ondemand/V9.5/config/ondemand.kdb
SSL_KEYRING_STASH=/opt/IBM/ondemand/V9.5/config/ondemand.sth
SSL_KEYRING_LABEL=CMODselfsigned
SSL_CLNT_USE_SSL=0
[@SRV@_DD]
PROTOCOL=1

I will check into upgrading.

Thanks.............................

[jsquizz]

The forum community here seems very knowledgeable so I thought it would be an excellent place to see if others have encountered this issue and could suggest debugging ideas.
BTW I have opened a PMR with IBM but have yet to receive a timely response. 

Thanks...........

Concerning, but this happens to me all the time. I usually always open PMR's as SEV2 bug instead of usage. I haven't had any issues except I wasn't really allowed to do that when we were IBM gold partners...or something like that.

[R2D2]

For future reference I wanted to follow up.
Changing the ulimits to unlimited allowed the server to complete initialization.
It now comes up and listens on a SSL port and nonSSL port.
Thanks for the suggestions.

[Justin Derrick]

Thanks for the update, and I'm glad to hear you got it figured out.

-JD.

[jsquizz]

For future reference I wanted to follow up.
Changing the ulimits to unlimited allowed the server to complete initialization.
It now comes up and listens on a SSL port and nonSSL port.
Thanks for the suggestions.

This bit me in the butt a few months ago!

[Justin Derrick]

The ulimit issue pops up so frequently that I'm going to make a note of it in the IBM CMOD troubleshooting guide on the wiki:  https://cmod.wiki/index.php?title=Troubleshooting_Content_Manager_OnDemand  ... or use the shortlink I've created:  http://cmod.co/troubleshooting

-JD.

[jsquizz]

The ulimit issue pops up so frequently that I'm going to make a note of it in the IBM CMOD troubleshooting guide on the wiki:  http://cmod.co/troubleshooting

-JD.

I was told by John @ IBM to pretty much max them out. I've had grumpy SA's in the past say no. We are maxed out now and running fine.

[Justin Derrick]

Yeah, the number of times I've been given fresh, brand new servers with anemic ulimits...  It gives me a headache just trying to count them... 

-JD.