Topics

1

OD/WEK & JAVA API / Stable permanent DocId links

« on: December 22, 2023, 07:04:26 AM »

So, I have read existing threads here that touch upon the usage of the CMOD Unique DocID feature .

• Topic 1
• Topic 2
• Topic 3

I'd like to catch up with the community here as what can be done to create permanent document links. As far as I understand, even if we activate unique docids in CMOD, we are still required to use odfolder.recreateHit in combination with either getOpenDocid or getDocid - which IBM warns produces non-persistent links (in case of reloads etc).

IBM also states (in the same ODApi docs):

The docid format is subject to change. If your application requires a unique document identifier to persist, consider using the Application Group field option "Unique ID".

What for, mate? How would that help? I can find no retrieve doc function that exclusively uses the unique docid (and perhaps a folder name or its Application Group or AGID) to retrieve a document.

So what's the use then, if we cannot create persistent doc links? (like the ones they have in that not-as-nice system "FileNet". This is a big commercial issue as the lack of links will entice companies into using or migrating to that system, instead of CMOD.)

Here is my current idea on how unique docids might be used to create persistent/permanent links.

1. Make sure the field you use for the Unique DocId has the same name in all Application Groups.
2. Search using a combination of folder name, application group name and the unique docid. For example:
URL Link = ODServer/search/folder_name/agcode+unique_docid
(such as www.mycomp.com/ODWEKServer/search/generalfolder/B6EA49-2ff02876-ffa5-4564-b0fb-966361dcf4f0 where "B6EA49" is an encoded Application Group Name and the rest is the actual, CMOD-generated Unique ID)

Code: [Select]

appgrpname = decodeAGname(agcode); // decodes "B6EA49" from the URL into some existing Application Group name
ODFolder odf = ods.openFolder(foldername); // "generalfolder" from the URL
odf.setApplGroupForSearchWithSQL(appgrpname);
ODHit uniqueHit = odf.search("WHERE $uniqidfieldname = $unique_docid); // get the unique hit
uniqueHit.retrieve('',C:\Temp\uniquefile);  // retrieve the unique file
//...Do something with the file...
(Note: this is untested code with the syntax of no real programming language - it is merely an idea)

Depending on how the rest of the web app is built, the user with the link will somehow need to be authenticated and have authorization to view the file.

Do you have any better ideas? Please share!
Or insights from IBM workshops etc? Sorry if I am currently not up to date.
Or improvements or helpful thoughts on the matter.

2

Other / The famous but illusive 33 petabyte system

« on: November 03, 2022, 02:43:50 AM »

Hi all,

I believe I once saw a technical description of the famous OnDemand system, which allegedly holds 33 petabyte of document data on a standalone server.
But I cannot find it anymore. Does anyone have a link to where I can find detailed info on the setup of that system?

3

MP Server / MASK field in ARSSEG - bit meaning

« on: October 13, 2021, 07:17:55 AM »

Does anyone have a list (full or partial) of the what the different bits in the MASK field of ARSSEG are for?
A guy asked us, the theory is that one or more of the bits determine which date style is used in the Segment date field of the table?

4

Windows Client / Data update validation

« on: August 31, 2021, 07:39:58 AM »

NOTE TO ALL READERS: This article is NOT orthodox usage of OnDemand. Be warned.

Hey, we have an unfortunate situation where some documents residing in CMOD needs to have their metadata corrected. This is done (by a few privileged users) using the built-in "Update Fields..." functionality in the OnDemand Client.

This of course has multiple impact, one being the discrepancy incurred between CMOD-stored metadata and DB-stored metadata.
The one I wish to ask about however is the risk of PEBKAC ("Problem Exists Between Keyboard And Chair" ) i.e. "human error" during data entry.

Has anyone implemented a well-functioning solution to validate the new data entered by these individuals? Like with a simple data mask, length check etc.

We currently do not have IBM Content Navigator in place, and I am reluctant to suggest a web service using the ODWEK Java API for handling this instead, due to that the size of such a project might be tough to squeeze in before the end of this year.
All suggestions welcome!

PS: Oh, by the way, we are NOT looking for a DDE/OLE solution.

5

MP Server / How to get the Application Group from arsdoc queries?

« on: July 09, 2021, 06:05:40 AM »

Hey guys!

I think I'm having a mind lapse - but how do you get the AGID or the Application Group name in the output when you run an arsdoc query?
I tried the -D parameter, but the AGID is not included there, just the DOC_NAME.
I also tried creating a Folder Field named "appgrp", but arsdoc aggressively refuses to print it. ESPECIALLY SO when I request it using "-N "(appgrp)" -->
ARS6020E The database field 'appgrp' was not found in the application group or it was not mapped to a folder field.

So - how?
Setup:
I have a search folder which contains more than one Application Group, and I don't want to create multiple different single-application-group folders instead and run the query multiple times (about 50 times).
After I've done the arsdoc query, I need to run arsdoc update on parts of the result. But arsdoc update refuses to run without either:
a) -G <application group>, or
b) a search folder which only contains a single Application Group

Any ideas?

6

Tips and Tricks / How to let user know they are viewing an extra sensitive doc?

« on: October 20, 2020, 07:32:49 AM »

Hi!

We have a solution in place where we use one of the OD user exits. If a user is not authorized to view a particular customer's document(s), we won't let them. This solution is based on one of the metadata field values, and a lookup in an external DB/web service.
NOTE: Whether a customer is "extra sensitive" or not is most oftenly set LONG time after we load the documents into CMOD. There is no reasonable way of setting this in a metadata field at load time.

However, some users ARE authenticated to view these documents. Now;
How would you suggest to inform those users that this is an extra sensitive document?
Oh, and we are talking about the thick OD Client, as well as anyone accessing via Content Navigator.

If it was merely AFP documents, we could probably add a watermark with this info, but there are several different file formats, including PDF, a standard which is prone to change over time. AFP, TIFF and PDF would be a minimum to cover for the solution.
Also, we don't want to make any permanent changes to the documents, since the "extra sensitive" status is usually temporary (years).

Post your suggestions, I'm all ears/eyes!

7

Content Navigator / ICN programming/customization for CMOD help needed

« on: November 22, 2019, 06:43:38 AM »

(The last part of this post borders another topic header, Job opp's)

We need to get a new developer quickly up to speed on customizing Content Navigator.
On IBM "My Learning", the training materials referred to (to get a badge) are:
* Recommended Reading: IBM Content Navigator V3.0 Knowledge center,
* Optional: Review the "Customizing and Extending IBM Content Navigator" Redbook.
(Plus taking the ICN Intro training)

The RedBook is huge (566 pages) and the examples in the Knowledge Center are mostly cryptic/not well enough documented.

Does anyone know of any place online with short, logical examples with well-documented sample code, where one can learn one thing at a time, to quickly get the geist of it?
So they get up to speed, to be able to then continue with the Redbook, and the more complex and unstructured examples in the Knowledge Base.
(Oh - and preferably for ICN 3.0, not just old 2.x stuff.)

Also, if you have the skills to customize ICN 3.0 and would be willing to hold a paid training onsite or online, please send me a PM!

8

Other / OAIS support (ISO 14721)?

« on: September 11, 2019, 01:55:20 AM »

Hey, does anyone know if it is possible to make OnDemand OAIS compliant? (Or if it already is - where can I find proof of this?)
How much work would it take? Is IBM already looking into it?

Reference: https://en.wikipedia.org/wiki/Open_Archival_Information_System
(The OAIS/ISO 14721 costs half an arm and a quarter leg, plus hours of reading for those who are interested)

9

OD/WEK & JAVA API / Unload call

« on: October 26, 2018, 02:46:42 AM »

Hi all, maybe it's just me, but I'd like to have an ODAPI call that does Unload.
A swift way to remove botched loads without needing to have the thick OD client installed, nor need to file a case and send it to the OnDemand Administrator.

Has anyone found another smooth solution for this?

10

MP Server / How to index fully composed AFP with ACIF

« on: January 16, 2018, 08:32:46 AM »

Warning: incoming "stupid" question.
Although having worked with CMOD for many years, I have not worked a huge lot with AFP files.
I have currently received an - allegedly - fully composed AFP file, with all its resources inline.
Now, how the eitch do I set up the Indexer information to properly process and index this file?
It seems that no matter what parameters I use, I end up with either APK459S or with APK210S+APK420S.
Which one seems to boil down to whether I use "CONVERT="NO"or YES.
Error message type one - you can see my settings as a part of this:
: ARS4302I Indexing started, 2174218 bytes to process
: APK415I CC=YES
: APK415I CCTYPE=A
: APK415I TRC=YES
: APK415I DCFPAGENAMES=YES
: APK415I UNIQUEBNGS=YES
: APK415I IMAGEOUT=ASIS
: APK415I INDEXOBJ=ALL
: APK415I FORMDEF=DUMMY
: APK415I RESTYPE=ALL
: APK415I INPUTDD=D:\OnDemandDirectories\arsload\ACME.VPAK_X.kivratestfil3.kivratest3.20171214.131155.ARD
: APK415I OUTPUTDD=E:\arstmp\ACME.VPAK_X.kivratestfil3.kivratest3.20171214.131155.ARD.out
: APK415I INDEXDD=E:\arstmp\ACME.VPAK_X.kivratestfil3.kivratest3.20171214.131155.ARD.ind
: APK415I RESOBJDD=E:\arstmp\ACME.VPAK_X.kivratestfil3.kivratest3.20171214.131155.ARD.res
: APK210S DATA IN AN INPUT RECORD OR RESOURCE IS INVALID: A REQUIRED TRIPLET OR SELF-DEFINING PARAMETER WITH ID '85'X WAS MISSING FROM A MFC STRUCTURED FIELD.
: APK420S AN ERROR OCCURRED WHILE ATTEMPTING TO OPEN T1001143 RETURN CODE 28.
: APK412I MODULE APKSRIAX HAS RETURNED WITH RETURN CODE 255.
: APK104S DATA IN AN INPUT RECORD OR RESOURCE IS INVALID: ECF STRUCTURED FIELD IS NOT ALLOWED OR FORMS AN INVALID SEQUENCE.
: APK105I THE ERROR REPORTED ABOVE OCCURRED IN LOGICAL RECORD NUMBER 75, WHOSE SEQUENCE NUMBER IS 75, AND RESOURCE NAME IS F1FAL.
: APK441I ACIF AT IPTR923 HAS COMPLETED ABNORMALLY WITH RETURN CODE 16.
: ARS4309E Indexing failed

Error message type two:
: ARS4302I Indexing started, 2174218 bytes to process
: APK415I CC=YES
: APK415I CCTYPE=A
: APK415I TRC=YES
: APK415I CONVERT=NO
: APK415I DCFPAGENAMES=YES
: APK415I UNIQUEBNGS=YES
: APK415I IMAGEOUT=ASIS
: APK415I INDEXOBJ=ALL
: APK415I FORMDEF=DUMMY
: APK415I RESTYPE=ALL
: APK415I INPUTDD=D:\OnDemandDirectories\arsload\ACME.VPAK_X.kivratestfil3.kivratest3.20171214.131155.ARD
: APK415I OUTPUTDD=NUL
: APK415I INDEXDD=E:\arstmp\ACME.VPAK_X.kivratestfil3.kivratest3.20171214.131155.ARD.ind
: APK415I RESOBJDD=NUL
: APK459S INDEX NEEDED FOR THE GROUPNAME WAS NOT FOUND.
: APK441I ACIF AT IPTR923 HAS COMPLETED ABNORMALLY WITH RETURN CODE 16.
: ARS4309E Indexing failed
: ARS4324E File >D:\OnDemandDirectories\arsload\ACME.VPAK_X.kivratestfil3.kivratest3.20171214.131155.ARD<

So what am I doing wrong here? Or is the file indeed corrupted some way?
The TLE tags part looks OK though:
  796 BDT Begin Document ACME_VPAK                           0012 D3A8A8
  797   BNG Begin Named Page Group G0000001                 0010 D3A8AD
  798     TLE Tag Logical Element                           0019 D3A090
          TLE Fully Qualified Name Triplet (02)
          TLE  0B Attribute Name
          TLE  Name = 'KANALVAL'
          TLE Attribute Value Triplet (36)
          TLE  Value = '1'
  799     TLE Tag Logical Element                           0017 D3A090
          TLE Fully Qualified Name Triplet (02)
          TLE  0B Attribute Name
          TLE  Name = 'DIGBREV'
          TLE Attribute Value Triplet (36)
          TLE  Value = ''
  800     TLE Tag Logical Element                           002F D3A090
          TLE Fully Qualified Name Triplet (02)
          TLE  0B Attribute Name
          TLE  Name = 'KALLA'
          TLE Attribute Value Triplet (36)
          TLE  Value = 'B20171214FECDB113113800012'
  801     TLE Tag Logical Element                           001F D3A090
          TLE Fully Qualified Name Triplet (02)
          TLE  0B Attribute Name
          TLE  Name = 'PNR'
          TLE Attribute Value Triplet (36)
          TLE  Value = '197503271234'
  802     IMM Invoke Medium Map A4_S_B1                     0010 D3ABCC
  803     BPG Begin Page P0000001                           0010 D3A8AF
  804       NOP No Operation                                0011 D3EEEE
            NOP 'ISISTEST'
...
  848     EPG End Page P0000001                             0010 D3A9AF
  879   ENG End Named Group G0000001                        0010 D3A9AD
  880 EDT End Document ACME_VPAK                             0010 D3A9A8
...(more pages)

11

MP Server / Alternative delete methods

« on: January 02, 2018, 02:07:37 AM »

With the new GDPR regulation coming into effect by May 2018, many companies need better ways to delete their documents, especially from OnDemand.
Most regulatory agencies don't accept the "lazy delete" done with arsdoc delete and by the ODWEK API, where only the database record pointing to the document is removed, while the document data itself is left intact on disk (and other storage).
(Most of you guys here on the forum would succeed in restoring such a "deleted" document, if you had access to the database and the files on disk.)

Are there any other options?
If we do an export + reload (after removing the document that is to be deleted) and then unload the original load, a new (minor) problem appears: the segmentation order is disrupted. Example: Say that you for a given Application Group have 100 or more segment tables, and they have been created sequentially by/with daily printouts.
Then you unload and reload a batch, which for this example is 5 years old. When you reload it, it ends up in the CURRENT segment table, and the START_DT (Start date) column for the current table will be set to a much earlier date. If you repeat this, the segmentation will eventually become really messed up and searches will be slower, since a lot of (unnecessary) tables will be searched.

Are there any better methods out there to delete data from the FAA* files?
Has anyone been bold/crazy enough to investigate a solution which overwrites part of the data file on disk? (NOT recommended!)
Can you "re-open" a segment table for writing, temporarily? (As far as I know, you can only close a table and that automatically creates a new one. Of course, you could close the current table, reload the old data into a new table, and then close that table too. But that would create a whole lot of new tables over time.)
Can you forcefully move a batch of documents from one segment table to another?
Any other solution?

Please share your thoughts and solutions here. Also if you happen to know that IBM has a solution for this up the sleeve, I'd like to know.

12

MP Server / Restoring an IMPLIED_HOLD?

« on: December 01, 2017, 02:50:22 AM »

Hi guys,

To delete a document held by Enhanced Retention Management, you must first release all Holds on that document. Fine.
You can release any hold using for example arsdoc hold_release. This includes removing an Implied Hold, using "-l IMPLIED_HOLD".
However, what if you accidentally remove the implied hold for the wrong document? How do you reset it properly?
arsdoc hold_add does not seem to accept "-l IMPLIED_HOLD":

2017-12-01 10:20:26.804052: ARS6822I Attempting login for userid 'odadmin' on server 'ARCHIVE' ...
2017-12-01 10:20:26.834828: ARS6080I Login successful
2017-12-01 10:20:26.834960: ARS6131I Searching for hold 'IMPLIED_HOLD' ...
2017-12-01 10:20:26.913433: ARS6085E Search unsuccessful
2017-12-01 10:20:26.913715: ARS6133E Unable to get hold information.  The hold does not exist or the user does not have permission to access the hold.
2017-12-01 10:20:26.914632: ARS6026I arsdoc completed.

So. We don't want to export the entire batch and reload it again.
Can we "cheat" in a "half nice" way and run:
arsdoc update ... -i "<suitable SQL to find the document to update" -n Lockdown=<current value of Lockdown +16384>
?
Or how else do we restore the Implied Hold to the document?
(As far as I know, Implied Holds are only stored in the "Lockdown" field, as the value 16384, not in any other ARSHOLD* table. Let me know if this is wrong - if so, the suggested action above will obviously not work.)

13

Windows Client / Tracing the Windows Client

« on: November 20, 2017, 04:39:48 AM »

Is there a "secret" way to activate trace logging for the Windows Client? Or at least more extensive logging to the Windows Event Log?

14

MP Server / Delay during unload in 8.5

« on: February 07, 2017, 08:37:08 AM »

Hi all,

I was wondering if anyone out there with a deeper knowledge of the nuts and bolts inside / behind the curtains of the arsadmin Unload command could inform me on a matter.

We have a script which is run regularly to unload some data from LoadIds which we do not have the exact date range for. Basically the command looks like this:

> arsadmin unload -h ARCHIVE -g XYZ2 -u Username -p password -Q -L 4527-13-0-179721FAA-0-0

Even though the AppGroup contains a lot of documents, even small loads (LoadIds with maybe 100 docs in them) render a 6 minute delay.
This happens only in those Application Groups where we have Enhanced Retention Management activated. In other Application Groups with a similar amount of documents, the Unload is without delay.
DO NOTE HOWEVER that all Holds have been successfully removed BEFORE the command above is run. Even for large batches, this Hold removal (arsdoc hold_release ...) takes no more than 2-4 seconds. The problem does not lie in the removal of the holds, neither named or implied.

This snippet below is taken from the System Log. Please let me know if that Msg 65 that looks truncated and the Msg 283 is normal. I cannot find those when I do unload in version 9.0 on Windows, but this is an AIX-based CMOD system 8.5.0.9 with a DB2 10.5 database.

09:30:38  30  Login: ondemand.xyzcorp.net 10.111.192.11 non-SSL (AIX) (ARSADMIN) (8.5.0.9)
Note the 423 second delay before the next message.
09:37:41  65  Application Group Query: Name(XYZ2) Agid(4527) Sql(1 <-- Note that this message looks truncated
09:37:41 283 Application Group Check Unload: ApplGroupName(XYZ2) Agid(4527) Aid(4530) LoadId(4527-13-0-179721FAA-16245-16245)  Total Load Rows(101)  Total Queried Rows(101)  Total Held(0)  Hold(0)  Non-Implied(0)  Implied(0)  CFS-OD RM(0)  CFS-OD Fed(0)  Pct M
09:37:44  85  Application Group Unload Storage Manager: Name(XYZ2) Agid(4527) NodeName(EKSTG) Nid(13) Server(-LOCAL-) LoadId(179721FAA) Objects Deleted(2)
09:37:44  85  Application Group Unload Storage Manager: Name(XYZ2) Agid(4527) NodeName(-CACHE-) Nid(0) Server(-LOCAL-) LoadId(179721FAA) Objects Deleted(0)
09:37:44  84  Application Group Unload Database: Name(XYZ2) Agid(4527) LoadId(179721FAA-16245-16245) Rows Deleted(101) SM UnLoad Ready(1)

I am unable to find this message "283" in any V9.0 system i Search. Are they actually related to CFSOD? (We do not have CFSOD installed)
I suspect the delay is caused by either the "truncated" MsgNum 65, or by the (unnecessary?) check unload MsgNum 283.
Can I add an index to some table and field to speed this procedure up? If so, which ones?

By the way, the DB2 seems to be well tuned, it does not appear to suffer from any memory allocation storms or similar resource problems. It merely sat there during these delays.

I'd be grateful for any knowledge you may wish to share with me and other CMOD Admins.

All the best,

Lars

PS: MsgNum 283 also occurs during arsmaint.

15

MP Server / arsdoc get not returning all hits?

« on: November 29, 2016, 04:31:10 AM »

EDITED: I just found THIS issue, which probably covers this question. Will need to do some testing though to verify it.
http://www.odusergroup.org/forums/index.php?topic=996.0

Hi,

This is a long shot, something for all you OnDemand-lovers out there!

We currently have a customer, on OD 8.5.0.9 running on AIX, where we recently experienced something very weird. A PMR is opened, and we are working with IBM on this, but we have been unable to efficiently reproduce the problem on purpose - thus far, at least.

We made a script which had the purpose of doing this (plus a little extra, but we can pinpoint the error to these two steps):
* Export all documents within a named LoadId
* Unload all documents in that same LoadId - although here the two last parts of the LoadId are replaced with "-0-0" for practical reasons.

The result is, somewhat surprisingly, that on step 1, X number of documents are exported.
While in step 2, a number GREATER than X is Unloaded!!!

This shouldn't be possible, no.
The number in step two, for example "34", matches the number reported in the System Log MsgNum 87 (arsload) for the same load. But when we exported that LoadId, we only got 33 documents.

Can you guys please think of any situation where this may come true?
YES: we use Enhanced Retention Management on this system. Between step 1 and step 2, we remove ALL Holds on all documents within the LoadId. Never any problem with this, all documents in the LoadId ALWAYS get deleted. IF there was a problem with this step, the resulting number of deleted documents should be LESS THAN the number that got exported, but it is the other way around.

Here are the exact commands that we used in the script. Let us know if they are incorrect in some way, which may cause the behaviour above.
Also, let us know if you in any other setting have experienced similar problems, please describe when and how.

The exact scripted arsdoc get command looked like this:
${OD_BASE_CATALOG}/arsdoc get -a -d ${WORKDIR} -G ${APPGRP} -X ${LOAD_ID} -g -h ${OD_INSTANCE} -o ${PREFIX} -u ${USERNAME} -p ${PASSWORD} -A 7 -N -c > tmp.txt 2>&1

Which instantiated would translate to something like this (Sensitive information changed for security reasons):
arsdoc get -a -d /opt/ondemand/arsdownload/erm/working -G OURAPPGROUP -X 6099-13-0-2129FAA-16683-16683 -g -h ARCHIVE -o FILE2129  -u ODUSER -p xxxxxxxx -A 7 -N -c -v >> FILE2129.txt 2>&1

The exact scripted arsadmin unload looked like this:
${OD_BASE_CATALOG}/arsadmin unload -h ${OD_INSTANCE} -g ${APPGRP} -u ${USERNAME} -p ${PASSWORD} -Q -L ${LOAD_ID} > tmp.txt 2>&1

Instantiated, something like this:
arsadmin unload -h ARCHIVE -g OURAPPGROUP -u ODUSER -p xxxxxxxx -Q -L 6099-13-0-2129FAA-0-0 > tmp.txt 2>&1