Messages

76

MP Server / Re: Need help for SSL setup and Configuration on CMOD 9.5 Linux server

« on: February 08, 2017, 07:22:21 AM »

Hi Jaydeep,

Not long ago, I read through the entire documentation on how to activate SSL. It was for a Windows-based OnDemand, but I expect the principal differences to be minimal or none.

The step you describe above seems to cover what I documented as "Step 1" in my 3-step plan.

So although I haven't executed the plan yet (still waiting for a "Go"), I'd say what you need to do is this:

Step 2: Verify the GSKit level on the "client machines", IE where you have ODWEK for ICN, as well as from your thick client (I assume that is what you meant by "think client" ) machine.

Step 3: Activate Secure Sockets layer on the client (software) you will use. For the OD32 thick client, this would be "Check the checkbox" in the "Update a server" window at login.
Here's info on how you add your certificate to your client:
http://www.ibm.com/support/knowledgecenter/SSEPCD_9.5.0/com.ibm.ondemand.installmp.doc/dodnl027.htm

Good luck!

77

MP Server / Re: Delay during unload in 8.5

« on: February 08, 2017, 02:51:27 AM »

Thank you Justin!

78

MP Server / Delay during unload in 8.5

« on: February 07, 2017, 08:37:08 AM »

Hi all,

I was wondering if anyone out there with a deeper knowledge of the nuts and bolts inside / behind the curtains of the arsadmin Unload command could inform me on a matter.

We have a script which is run regularly to unload some data from LoadIds which we do not have the exact date range for. Basically the command looks like this:

> arsadmin unload -h ARCHIVE -g XYZ2 -u Username -p password -Q -L 4527-13-0-179721FAA-0-0

Even though the AppGroup contains a lot of documents, even small loads (LoadIds with maybe 100 docs in them) render a 6 minute delay.
This happens only in those Application Groups where we have Enhanced Retention Management activated. In other Application Groups with a similar amount of documents, the Unload is without delay.
DO NOTE HOWEVER that all Holds have been successfully removed BEFORE the command above is run. Even for large batches, this Hold removal (arsdoc hold_release ...) takes no more than 2-4 seconds. The problem does not lie in the removal of the holds, neither named or implied.

This snippet below is taken from the System Log. Please let me know if that Msg 65 that looks truncated and the Msg 283 is normal. I cannot find those when I do unload in version 9.0 on Windows, but this is an AIX-based CMOD system 8.5.0.9 with a DB2 10.5 database.

09:30:38  30  Login: ondemand.xyzcorp.net 10.111.192.11 non-SSL (AIX) (ARSADMIN) (8.5.0.9)
Note the 423 second delay before the next message.
09:37:41  65  Application Group Query: Name(XYZ2) Agid(4527) Sql(1 <-- Note that this message looks truncated
09:37:41 283 Application Group Check Unload: ApplGroupName(XYZ2) Agid(4527) Aid(4530) LoadId(4527-13-0-179721FAA-16245-16245)  Total Load Rows(101)  Total Queried Rows(101)  Total Held(0)  Hold(0)  Non-Implied(0)  Implied(0)  CFS-OD RM(0)  CFS-OD Fed(0)  Pct M
09:37:44  85  Application Group Unload Storage Manager: Name(XYZ2) Agid(4527) NodeName(EKSTG) Nid(13) Server(-LOCAL-) LoadId(179721FAA) Objects Deleted(2)
09:37:44  85  Application Group Unload Storage Manager: Name(XYZ2) Agid(4527) NodeName(-CACHE-) Nid(0) Server(-LOCAL-) LoadId(179721FAA) Objects Deleted(0)
09:37:44  84  Application Group Unload Database: Name(XYZ2) Agid(4527) LoadId(179721FAA-16245-16245) Rows Deleted(101) SM UnLoad Ready(1)

I am unable to find this message "283" in any V9.0 system i Search. Are they actually related to CFSOD? (We do not have CFSOD installed)
I suspect the delay is caused by either the "truncated" MsgNum 65, or by the (unnecessary?) check unload MsgNum 283.
Can I add an index to some table and field to speed this procedure up? If so, which ones?

By the way, the DB2 seems to be well tuned, it does not appear to suffer from any memory allocation storms or similar resource problems. It merely sat there during these delays.

I'd be grateful for any knowledge you may wish to share with me and other CMOD Admins.

All the best,

Lars

PS: MsgNum 283 also occurs during arsmaint.

79

MP Server / Re: arsdoc get not returning all hits?

« on: December 02, 2016, 05:28:49 AM »

Anyways, due to this anomaly above, we cannot use "-X" with arsdoc if we want to export all documents inside a particular LoadId.
It is a pity that there is no "-L" parameter which works in the same manner as during "arsadmin unload ... -L <LOADID>" - that command appears to always find the exact right amount of records to delete.

Also, we have earlier run tests that show that we cannot use "-x" (lower case x) either, since this OnDemand system contains records that are so old, that "-x" won't produce the correct hits. (If I recall correctly, it cannot find the Load at all, this produces zero hits/exports zero documents)

So, how DO we get all records in a LoadId exported then?
Good question. We are currently experimenting with SQL commands like "WHERE DOC_NAME='<The LoadId plus an extra "A">' " and similar, but if one of you guys (JD?) have already solved this dilemma, please share!

All the best,

Lars

80

MP Server / Re: arsdoc get not returning all hits?

« on: December 02, 2016, 05:21:55 AM »

Edited to remove passwords.

Heh, thanks - although the ones I put there were of course not the real ones.

(Sensitive information changed for security reasons)

81

MP Server / arsdoc get not returning all hits?

« on: November 29, 2016, 04:31:10 AM »

EDITED: I just found THIS issue, which probably covers this question. Will need to do some testing though to verify it.
http://www.odusergroup.org/forums/index.php?topic=996.0

Hi,

This is a long shot, something for all you OnDemand-lovers out there!

We currently have a customer, on OD 8.5.0.9 running on AIX, where we recently experienced something very weird. A PMR is opened, and we are working with IBM on this, but we have been unable to efficiently reproduce the problem on purpose - thus far, at least.

We made a script which had the purpose of doing this (plus a little extra, but we can pinpoint the error to these two steps):
* Export all documents within a named LoadId
* Unload all documents in that same LoadId - although here the two last parts of the LoadId are replaced with "-0-0" for practical reasons.

The result is, somewhat surprisingly, that on step 1, X number of documents are exported.
While in step 2, a number GREATER than X is Unloaded!!!

This shouldn't be possible, no.
The number in step two, for example "34", matches the number reported in the System Log MsgNum 87 (arsload) for the same load. But when we exported that LoadId, we only got 33 documents.

Can you guys please think of any situation where this may come true?
YES: we use Enhanced Retention Management on this system. Between step 1 and step 2, we remove ALL Holds on all documents within the LoadId. Never any problem with this, all documents in the LoadId ALWAYS get deleted. IF there was a problem with this step, the resulting number of deleted documents should be LESS THAN the number that got exported, but it is the other way around.

Here are the exact commands that we used in the script. Let us know if they are incorrect in some way, which may cause the behaviour above.
Also, let us know if you in any other setting have experienced similar problems, please describe when and how.

The exact scripted arsdoc get command looked like this:
${OD_BASE_CATALOG}/arsdoc get -a -d ${WORKDIR} -G ${APPGRP} -X ${LOAD_ID} -g -h ${OD_INSTANCE} -o ${PREFIX} -u ${USERNAME} -p ${PASSWORD} -A 7 -N -c > tmp.txt 2>&1

Which instantiated would translate to something like this (Sensitive information changed for security reasons):
arsdoc get -a -d /opt/ondemand/arsdownload/erm/working -G OURAPPGROUP -X 6099-13-0-2129FAA-16683-16683 -g -h ARCHIVE -o FILE2129  -u ODUSER -p xxxxxxxx -A 7 -N -c -v >> FILE2129.txt 2>&1

The exact scripted arsadmin unload looked like this:
${OD_BASE_CATALOG}/arsadmin unload -h ${OD_INSTANCE} -g ${APPGRP} -u ${USERNAME} -p ${PASSWORD} -Q -L ${LOAD_ID} > tmp.txt 2>&1

Instantiated, something like this:
arsadmin unload -h ARCHIVE -g OURAPPGROUP -u ODUSER -p xxxxxxxx -Q -L 6099-13-0-2129FAA-0-0 > tmp.txt 2>&1

82

MP Server / Re: WARNING - ARSDOC GET -X <loadid> issues

« on: November 29, 2016, 04:29:28 AM »

Oh my gosh!
This fits a problem we currently have.
I gives me an idea on how to reproduce it too - if I understand correctly, we need to edit the metadata of one or more of the records, and then arsdoc get -X will fail to return all documents?
Holy crap, that's scary. And yes, we have probably lost some real data due to this.

83

Tips and Tricks / Re: Script to show doc counts for a list of app groups

« on: November 02, 2016, 06:28:22 AM »

I'd like to up the challenge level a bit, and change the question a bit. Hopefully someone already knows the answer to this.

When you send AFP documents to OnDemand, it is possible to create more than one DB record per physical document. I.e. let's assume that you create an AFP file with 10 letters in it (10 documents), but each of these are related to two different persons. So you add two metadata records per document, i.e. 20 records/rows get added to the DB, but only 10 documents.

So the scripts above don't really count the number of DOCUMENTS in OnDemand, but rather the number of document RECORDS (or "rows") in the database.

Technically, it would be possible to run a SQL query against each table, and then group the result on the field that we know is duplicated. (In the example above, it could be "CustName" or "SocSecNo" or something similar.)

But assume that we do not always know which field is the one that is varied in non-unique records, how can we count the actual number of DOCUMENTS in OnDemand? Across quite a few Application Groups, which have a fairly large number of tables each, and different sets of metadata.

In the example above, let's say that Alessandros excellent scripts tell us that there are 1.342.917 records in one particular application group, and we happen to know that there have only been 642.387 documents loaded. What we want to find is that number - "642.387". How can we do that count?
 

(Yes, I have a real-life request for this particular info.)

EDIT: For the record, I already know of and make use of this info:
http://www.odusergroup.org/forums/index.php?topic=2086.msg7928#msg7928

84

MP Server / Decrease the number of arscaches

« on: October 25, 2016, 06:37:44 AM »

Hi guys,

One of our customers has a high number of small arscache directories. They want to change that into between 1 and 5 larger ones, because it is easier for them to maintain (plus a few other reasons).
Solaris platform. They are currently not interested in/happy about using a TSM. (But I will try to convince them, see option 3 below)

What would YOUR approach be to this?

1. Export all data, delete it, rebuild arscache structure, import/reload all data?
(Will probably not help, since System Log and System Load data is likely to be spread all over the arscaches)

2. Parse through all files, update the links (file paths) in the first arscache directory and move the actual files+folders from the other arscache directories?
(Obviously we would write a program or a script that does this! )

3. Install a TSM, create suitable Storage Sets on TSM and CMOD, create copies of their existing AppGroups but set storage = TSM Storage Set "x", set "cache = no", THEN export all data in an Appgroup and import it to the new AppGroup-copy, then delete the old AppGroup. (Repeat for all AppGroups).
(Probably still needs to move SL and SA data from higher arscache directories to lower ones, but a lot less to do.)
When done, stop CMOD and "undefine" excess arscaches, restart. (Obviously run arsmaint to verify arscaches afterwards)

4. Other? Please describe.

85

Tips and Tricks / Re: Calling a java program during indexing.

« on: August 15, 2016, 12:11:45 AM »

Pankaj, do you have an example of a working Postprocessor command which you can include here?
Do you just type (for example) C:\myjavaappfolder\myjavaapp.jar in the field, and that's it?

86

OD/WEK & JAVA API / Re: ODWEK Authentication Mechanisms

« on: March 11, 2016, 04:38:45 AM »

Alex has given a superb description of how to do user authentication via ODWEK using SSO.
Now;
Has anyone implemented user authentication, via ODWEK Java API, where you DON'T have SSO?
(This request is for a Windows-based CMOD system, so "odServer.unifiedLogon" would be quite possible.)

What we want to do is that users in integrated systems send a request to a web service on a middleware. We want them to use their own UserId to be authorized in CMOD, not a CMOD System account (which the MW team would like).

I checked with experts in here and with IBM Tech Support - they confirm that the unifiedLogon method cannot be used other than from the machine where it is called. Thus, with a middleware in place, the user that will be logged in when unifiedLogin is called will be the one who is "logged in", i.e. "is running the middleware service" on the middleware server.

So it is confirmed, odServer.unifiedLogon cannot be used to automatically log on web users. We'll have to go for the security exit solution instead.

87

Report Indexing / Re: Loading with PDF indexer, but getting invalid generic index error

« on: March 08, 2016, 07:40:36 AM »

Hi Jason,

Are you by any chance using the "graphical indexer" feature of PDF Indexer?
If so, it is possible that sometimes you get data that is partially outside of the indexing "rubber rectangle" you have specified.
I believe the PDF Indexer builds a "generic" indexer file based on the fields it finds in the PDF file. (Other forum experts will probably laugh at me and let you know if I'm wrong here - that's OK, I can take it.  )

If not, I would guess that this may be due to invalid data being snuck into the document, such as a linefeed or other non-printable character in the middle of a field. This would invalidate the format of the generic indexer file that is auto-created and would cause such an error message.

When you have this message, have you looked in the arstmp folder, or whatever folder is used during indexing? You should be able to find the faulty *.ind file there, and you can manually look through it to check for errors. (You can find the valid format for indexing files in the CMOD docs online)

Hope this helps!

/Lars

88

OD/WEK & JAVA API / Re: ODWEK Authentication Mechanisms

« on: March 07, 2016, 07:22:17 AM »

Alex has given a superb description of how to do user authentication via ODWEK using SSO.
Now;
Has anyone implemented user authentication, via ODWEK Java API, where you DON'T have SSO?
(This request is for a Windows-based CMOD system, so "odServer.unifiedLogon" would be quite possible.)

What we want to do is that users in integrated systems send a request to a web service on a middleware. We want them to use their own UserId to be authorized in CMOD, not a CMOD System account (which the MW team would like).

89

Report Indexing / Re: Steps of PDF Indexing with PPD

« on: March 07, 2016, 05:09:07 AM »

Hi Teera,

Happy to give you some feedback.

1: You can also use CMOD 9.0, with PDF Indexer, and THEN install patch 9.0.0.3. But 9.5 is perfect.
2, 3, 4 & 5: Well, the normal way to go would probably be to tell the person who creates the PDF with PPD in it which index fields to include... but your method is not wrong, it will work.
6: DEFINITELY "PDF" and not User defined...
Indexer information; don't forget to set Indexer = "PDF"...  Your parameters are correct.
Load Information; remember to set "Data Compression", you have loads of data to gain by using compression now.
Our tests show that sizes went from 15-40% compression without the PDF Indexer to 70-80 % compression with. We use "OD77".

7: Yes.

Your note; Bud Paton can tell you, but I am about 90% sure that I read somewhere that this can NOT be done with password-protected PDF files.
If your files are password protected, I would suggest you go with another indexing method.

90

MP Server / Re: User Exit call to SOAP web service

« on: November 08, 2015, 11:44:11 PM »

Hi Alex,

My programmer colleague reported that we almost immediately ran into problems. To being with, the axis software was no longer available for download - the ftp folder was empty. But after Googling a bit, we were able to find the downloads on a hidden web page. However, when we looked closer at the axis 2 modules, they were very old. Version 1.6.0 (windows binary) was dated 2009. That is not a problem per se, but the problem is that the code appears to be 32-bit, and it is also dependent on a whole array of other 32-bit modules (DLLs), which therefore (according to my colleague) cannot be compiled into a 64-bit DLL, which is what CMOD requires.
If you know of a way around the 32-bit problem, please advice.

I just read an email from my colleague that he has found another way to solve our problem. Apparently it works by using Microsoft's "winhttp" API. I will get back here later to tell about that solution.

Excerpt from the build file, with axis 2:
#############################################################################
### Dependant Binary Locations (Required) ###
#############################################################################
#
# libxml2 binary location ( axis2c is built with libxml2 )
LIBXML2_BIN_DIR = E:\libxml2-2.6.30.win32
#
# iconv binary location
ICONV_BIN_DIR = E:\iconv-1.9.2.win32
#
# zlib binary location
ZLIB_BIN_DIR= E:\zlib-1.2.3.win32
...