Messages

106

OD/WEK & JAVA API / Connection between ODWEK and Library Server.

« on: September 01, 2021, 08:36:32 AM »

Apologize for the ignorance with the topic, I am not an ODWEK person, at all.

Our ODWEK folks came to us with a question- they want to know what protocol is used by ODWek to communicate with the CMOD servers, and most importantly, whether that protocol is secure.

It is my understanding that TCP/IP is used, and that the connection is secure because ODWEK and the library server, will use the IBM Global Security kit to secure that connection.

Is that accurate? Or am I in left field.

Thanks!

107

MP Server / Re: arssockd not starting on New RHEL Server

« on: August 30, 2021, 10:55:46 AM »

I am installing a new OnDemand v.10.5 server on RHEL v.7.9 with an Oracle v.19 database. There is no TSM or SSL requirements. We plan to use LDAP but do not have this configured yet.

The software seems to be successfully installed and the arsdb command ran successfully to create an instance called ondq (this will be the QA Server).  I have been able to run the commands to create the System Log and System Load files.  I think this next step is to try to start the server job.  When I run either of these commands I get NO response.  I just get returned quickly with a command prompt:

./arssockd ondq arssockd
or
arssockd -S -I ondq

I can not find any new log files being created with any information to go on.

I feel like I'm missing something simple??  Any ideas?

Thanks,
Joe

One thing I do, and I am not sure if its the "IBM suggested" approach, is just use trace. that has worked most if not all of the time on numerous rhel/aix systems-

1) Edit /opt/ibm/ondemand/V10.5/config/trace.settings, I usually turn it all on, TRACE_LEVELS=ALL=15,   TRACE_SETTINGS=/somewhereWithSpace/ondqa.trace
2) Edit /opt/ibm/ondemand/V10.5/config/ars.cfg = TRACE_SETTINGS=trace.settings (Make sure you comment this out when done, requires arssockd restart)
3) Start arssockd, let it fail.
4) Execute /opt/ibm/ondemand/V10.5/bin/arstfmt -i /somewhereWithSpace/ondqa.trace -o /somewhereWithSpace/readable_trace.out
5) execute more /somewhereWithSpace/readable_trace.out | grep ERR (Or whatever way to search the file..)

Most of the time its because I forgot to make a setting change or something. Such as,

setting ars.cfg to = ARS_STORAGE_MANAGER=CACHE_ONLY (i think default is TSM, or- If using new api's for azure/aws/etc.. ARS_STORAGE_MANAGER=NO_TSM)
if you're using cache, you didnt define a file system in /opt/ibm/ondemand/V10.5/config/ars.cache, or - odqa/archive user can't read/write that file system

Curious as to what the culprit was.

108

MP Server / Re: quick daemon question

« on: July 20, 2021, 02:24:09 PM »

Return code 6 is a pretty generic "abort" message.  Can you run the command in the foreground without issue?

-JD.

I forgot to reply. I added -c / -d and it started working.

109

MP Server / quick daemon question

« on: July 19, 2021, 08:04:05 AM »

I have a requirement where I "Think" a daemon might be our best bet to load a bunch of files.

I am trying to fire it up, not so much working. It has actually been about 5-6 years since ive had to do this. Here's what I have.

Code: [Select]

arsload -nvf -u jeff -p jeff.stash -B IGN.APP.AG.IGN.IGN.IGN.EXT -t 60 -h archive &
The file name is-

Code: [Select]

9898541175.TESTAPP.TESTAG.IGNOREHTHIS.20210712.010447405.ARD
I go to fire it up-

Code: [Select]

[1] +  Done(6)                 arsload -nvf -u jeff -p jeff.stash -B IGN.APP.AG.IGN.IGN.IGN.EXT -t 60 -h archive &
I have a feeling this is a very simple issue. OF course, I could easily script this as well, but I want to test something for a daemon.

110

MP Server / recreated extracted PDF File?

« on: July 16, 2021, 12:43:51 PM »

I am working with a file that was loaded using PPD's, trying to figure out some slowness.

Once the file is loaded, is there any way to extract the file as a whole, as it was loaded originally into CMOD? i am having issues getting it from the folks that send the files.

Basically, is it possible to some how recreate the file using the .out and the .res file?

Thanks in advance

111

Report Indexing / Re: Generic Indexing : Batch Vs Single

« on: June 10, 2021, 09:03:42 AM »

thank you for this.

I checked out our app group settings, and i noticed that one of our "heavy hitters" that comes from DocOrigin i believe.. Is using OD77, so its trying to compress a generic index PDF file and giving us ZERO savings, costing us resources.

Tested this in dev, much quicker as far as loading goes.

As far as the arsload table savings, space savings sounds good! I assume that this will also make arsmaint run much quicker as well.

112

MP Server / Re: Upgrade from Redhat 6-7, CMOD 9.5-10.5, Latency Issues

« on: June 09, 2021, 03:41:15 PM »

CMOD Performance Tuning is a bit of an art since there are so many moving parts.  I'd turn on maximum logging for that AG, and then check all the records in the System Log to see which steps are taking the longest.  It seems like you've done that and storage is a good place to start, especially because cloud storage is way slower than cache / TSM. 

You might be able to test your storage backend with a tool like 'curl' to access the S3 API, and that will help you narrow it down.

-JD.

Thats exactly what we did.

We turned on query logging, and were able to identify a gap in the CMOD system log where there was ZERO activity for like 40 seconds., some kind of lag/delay/etc.

So far, our disk changes worked and loads are working much better. Maybe I can sleep this evening.

113

Report Indexing / Generic Indexing : Batch Vs Single

« on: June 09, 2021, 11:44:21 AM »

I am noticing that our one Line of Business will send tens of thousands of files to CMOD. All with one-two documents, that uses the generic indexer (User Defined PDF / HTML)

Obviously I've followed best practices with AFP files and the like, send down a large batch, let cmod compress'em.

I am actually trying to convince the business to send down PDF's with PPD's, not sure if they can change the HTML files we receive though.

However, until then..Trying to build a case.

Is there a very large benefit in load times when doing a generic indexed file- To do lets say, 1000 docs in a file, compared to 1-5? I am going to assume so, because arsload is only called once, one set of inserts, one logon/logoff, etc.

thanks in advance

114

MP Server / Re: Upgrade from Redhat 6-7, CMOD 9.5-10.5, Latency Issues

« on: June 09, 2021, 11:37:37 AM »

So we worked on this quite a bit over the past few days and we've linked this back to an issue with our storage, slow response times. The disks are being upgraded and mirrored as we speak. hopefully that works well. Last time I supported something like this years ago. here's where the first indication of this was.

Code: [Select]

00:46:45.195162: ARS4342I Storage Node >CACHE_OBJEC
 00:46:45.195212: ARS4312I Loading started, --UNKNOW
 00:46:46.705322: ARS1144I OnDemand Load Id = >5041-
 00:47:01.442199: ARS1146I Loaded 2 rows into the database
 00:47:03.847714: ARS1175I Document compression type used = OD77
 00:47:03.847789: ARS4310I Loading completed
23 Seconds for a 241KB document to load, on a relatively new RHEL box with 64GB of memory. It's a generic indexed PDF. For comparison, Another document of the same size, .39 seconds.

Need to optimize that as well.

Will update this when and if our disk changes work

115

MP Server / Re: Upgrade from Redhat 6-7, CMOD 9.5-10.5, Latency Issues

« on: June 08, 2021, 08:34:51 AM »

> One of our SA's suggested rolling back the RHEL kernel to 7.8, due to some major features added in 7.9. We did that overnight, and sure enough when I logged on this morning, no bueno.

That means that didn't help?  That you're seeing the same symptoms?

Ed

Yeah. They seemed adamant that would resolve our issues.

i'm wondering if there was some kind of setting changed at an OS level that's being missed, something that is causing us tons of grief thats being overlooked.

116

MP Server / Upgrade from Redhat 6-7, CMOD 9.5-10.5, Latency Issues

« on: June 08, 2021, 05:00:42 AM »

Little background of our environment, 1Lib Server, 1 Obj Serv, RHEL + DB2, DB2 storage is on SAN Veritas Cluster, CMOD Cache is also on Veritas Cluster

We just upgraded from Redhat 6.8 to Redhat 7.9 Maipo. After a few days we noticed slow response times from our API's calling via ODWEK (Weblogic). These requests would timeout after around 20 seconds, causing impact to our customers. We also see instances where loading is slow, example- 12.8 Seconds to load a file with 11 Docs..500kb size.. using generic indexer (user defined html file).

We opened up a case with IBM, and the first suggestion was to upgrade CMOD to 10.5, so we did that, the CMOD upgrade went perfectly fine - and the issue persisted. We checked all of our DB2 (V11.1fp4) logs, and we don't see anything. CMOD says they cant see anything wrong, and I agree with them. the only thing they are seeing is a pause/delay in the system log..There was a 40-50 second delay where there was absolutely no activity, whereas for the whole day there is usually stuff going on. After that pause, there's some 6second long queries in the 226 records.

We also pulled in DB2 Support, and we are giving them updates/db2mon/db2logs - but i dont feel like we are getting anywhere.

Veritas is looking at our system, along with Redhat support.

We've engaged several of our DBA's, Storage Engineers, and System Admins to address the issues.

One of our SA's suggested rolling back the RHEL kernel to 7.8, due to some major features added in 7.9. We did that overnight, and sure enough when I logged on this morning, no bueno.

The biggest issue, is that the problem is intermittent. Our next step is to run on one library server instead of lib/obj to rule out any kind of network latency or anything like that. I also think it might be a shot moving the DB2 volumes off veritas to local disk, or something.

I am open to any possible feedback, if anyone has encountered issues going from RHEL6-7.

117

MP Server / Re: db2 cleanup against system log

« on: May 28, 2021, 08:50:51 AM »

Got it, I figured. that's what our dba suggested.

I'd like to take a backup of all those tables either way to a flat file, compress it compress it compress it.

Have you seen it as a practice to run arsmaint against the system log? I have not even with 7-8 clients.

I reached out to the business folks and told them all we are logging. And they were shocked. haha

118

OD/WEK & JAVA API / Re: Installing ODWEK 10.1 on Linux

« on: May 27, 2021, 10:11:58 AM »

Is this happening when you try to add the CMOD repository in ICN?

119

MP Server / db2 cleanup against system log

« on: May 27, 2021, 08:08:00 AM »

Just wondering if its a common practice for folks to run arsmaint against the system log.

Ours is fairly large I guess, 1B+ rows. I am not sure in the scheme of things if that is "large" but it's taking up alot of space.

It's difficult for our client to get space which is.. strange.

We're considering doing a cleanup against the system log, unused storing, 84/85/83/82/81..226/65/66 (Need to check with compliance), arsmaint, etc.

Wondering if anyone does this? Is it common? we'd probably make a backup of all the tables, and compress it, then store it in TSM or our sharepoint.

Thanks gang

120

MP Server / Re: arslog vs system trace for troubleshooting.

« on: May 24, 2021, 12:30:39 PM »

So arslog is usually a shell script that contains some really basic things like detecting failed loads or other errors and either writing them to a file, or sending notifications.  There are only performance implications in the largest systems, when you start getting into things like deadlocks in the filesystem because there are too many writes to the same log file.

Your database errors look like you yanked DB2 out from under CMOD.  Make sure your shutdown scripts are doing things in the correct order, and are double-checking their results.  (Don't just ask CMOD to stop -- wait a few seconds, then see if it's actually down.)

-JD.

Whoops, my bad again.

The errors i am showing, were just me testing something on our QA box.

I purposely did that. I turned tracing on, and I "yanked" db2. I didn't know we were using arslog exit, i just noticed syslog.log file being generated, and sure enough it had some info in it.

We just are trying to "catch" errors with unexpected CMOD as they happen. And trying to pinpoint as to what causes these failures. We think we have the culprit, but I'm just testing a few scenarios. I don't want to do tracing obviously and pray the thing crashes. Management is wondering why CMOD isn't logging why CMOD isnt crashing, and I tried to explain to them why from a technical standpoint.

This post is also extremely helpful, I think arslog with some tweaking might work.

http://www.odusergroup.org/forums/index.php?PHPSESSID=qr6bcru5vrdasncbi7icjqpkl1&topic=2965.msg11897#msg11897