Messages

16

MP Server / Re: Arsdoc get — best optimized way to extract and reload

« on: August 05, 2021, 04:25:56 AM »

There's also a #4 possibility - depending on the format of your data (see especially JD's comment previously).
If you keep the original input data files that you receive, for example in a (compressed) folder on disk for 1 month, then re-load it into the new long-term appgroup when the time has come, and simply do an unload of those LoadIDs in the first/short-term Application Group.

This method does take up extra disk space, but disk is really cheap nowadays. It would save you the hassle of exporting, reloading and unloading in a batch.

17

MP Server / Re: recreated extracted PDF File?

« on: August 05, 2021, 04:20:10 AM »

Also, I believe that the PPD info that was originally there is not kept when the data is loaded into OnDemand. So the exported files will be without the PPD data - as far as I recall.
(Correct me if I'm wrong, anyone)

18

MP Server / Re: ARSUSEC password requirements

« on: July 12, 2021, 01:51:11 AM »

Hey @karmiller & @pankaj9 - if you still need help with customizing the ARSUSEC exit, send me a PM. I have done it several times before.
PS: Another good-to-have is a list of "unacceptable passwords", such as "12345678" and others. I've done that too with ARSUSEC.

19

MP Server / Re: How to get the Application Group from arsdoc queries?

« on: July 09, 2021, 06:11:06 AM »

Ah yes -
The solution is of course to use "-I p" to get the full LoadID in the result list, and then use the first part of the LoadID, which is the AGID, to find the Application Group name.

Other solutions are still welcome!

20

MP Server / How to get the Application Group from arsdoc queries?

« on: July 09, 2021, 06:05:40 AM »

Hey guys!

I think I'm having a mind lapse - but how do you get the AGID or the Application Group name in the output when you run an arsdoc query?
I tried the -D parameter, but the AGID is not included there, just the DOC_NAME.
I also tried creating a Folder Field named "appgrp", but arsdoc aggressively refuses to print it. ESPECIALLY SO when I request it using "-N "(appgrp)" -->
ARS6020E The database field 'appgrp' was not found in the application group or it was not mapped to a folder field.

So - how?
Setup:
I have a search folder which contains more than one Application Group, and I don't want to create multiple different single-application-group folders instead and run the query multiple times (about 50 times).
After I've done the arsdoc query, I need to run arsdoc update on parts of the result. But arsdoc update refuses to run without either:
a) -G <application group>, or
b) a search folder which only contains a single Application Group

Any ideas?

21

Windows Client / Re: System Load

« on: May 18, 2021, 09:46:35 AM »

Hi,

You are correct that the Doc begin date, Doc end date, start date and stop dates are all taken from the index data that goes with the file. For an AFP file, it would be embedded in tags in the file, and for generic indexed docs it would come from within the .ind file.
In both cases, the date field that is used is defined in the Application Group and the Application that you have defined. For example, you have a field called "MyDate" (type Date) in the Application Group, which you have tagged as "Segment" on the Field Information tab, using the corresponding check box. And you have also defined in the Application (for example) that it should be picked from page 1 of every new "Group" (aka each independent doc in the input file), row X column Y, and Z characters long. And/or you have defined the date Format to match the date in your input documents, so it gets interpreted correctly, of course.
The highest and lowest date(s) found inside the loaded file is used to determine these dates in the System Load log, and you can see it in the MsgNum 87 in the System Log too.

Hope this helps! 

22

MP Server / Re: 1300: Incorrect AFPDS data was found

« on: May 18, 2021, 09:31:14 AM »

Hi all, I know this is an old topic, but we just received this message for an entirely different reason.

We have an OnDemand-using company that wishes to load AFP documents with PDF "pages" embedded inside the AFP file. This is all in compliance with the MO:DCA standard/definition of AFP, and it works (of course!) in the application that creates the AFP file.

Also, the resulting AFP files (out, ind and res) load perfectly fine into OnDemand, but when we try to view those documents in the OnDemand Client, the following happens:
1. First page is shown (AFP only) without problems.
2. When we scroll to page 2 (the PDF embedded page), this error message is show (as in the topic title)
3. After that, a warning message about code page is shown: "1213: The code page 4 is not supported in the Unicode converter..." - or with code page 119.
4. Finally, the page is loaded in the OnDemand Client Viewer as text only, starting with "%PDF-1.4%.....", showing that it is indeed a PDF.
5. Pages 3 and 4 (AFP only) display normally, no error.

So this "1300:..." seems to be a catch-all for a diverse set of "errors or deviations in the AFP format as expected by the OnDemand Client".

I have reported this to IBM, asking if this is due to some misconfiguration in the AFP file, in the *.fnt files of the OnDemand Client (i.e. do we need to edit/add anything there), or if this type of embedding simply is not supported by the OnDemand Client (V10.1.0.5/64-bit by the way)

I'll keep you posted on their reply.

23

Tips and Tricks / Re: How to let user know they are viewing an extra sensitive doc?

« on: November 24, 2020, 02:55:18 AM »

OK, continued;

So if we try to go for a solution where we insert a preface page, OR a watermark across all pages, this seems doable. Reviewing the three file types which we will be working with, programmatically:

• Inserting a page into a PDF is easy. Plenty of APIs are available.
• Manipulating TIFFs is not too bad either.
• But so far, we are unable to find a suitable API to do the same to AFP documents.

What API (java strongly preferred) would you guys suggest to do this operation with AFP documents?

24

Tips and Tricks / Re: How to let user know they are viewing an extra sensitive doc?

« on: November 12, 2020, 09:02:27 AM »

Sounds like you need the same thing insurance companies have...  You receive a 'master file' from a source system, and you process it to update fields in CMOD with arsdoc update.

You'll need to catch both processes -- once someone gets onto the list, it marks their documents as sensitive, once someone leaves the list, it removes the 'sensitive' label.

You could discriminate between the two with a query restriction - normal access is "NOT like 'sensitive'", and higher levels of access don't have that restriction at all.

-JD.

I think we slipped slightly off the subject here.
The issue is not hiding these sensitive documents from unauthorized access. We have a system in place that handles that part already.

The issue is: how do we inform the AUTHORIZED users that "This document that you are currently viewing, is of a sensitive kind. Do not share it outside of the corporation.".
(We know if it is a sensitive doc at the time of the retrieval, but not necessarily when we search.)
Unauthorized users will not be able to do bad stuff with the doc anyway.

25

Tips and Tricks / Re: How to let user know they are viewing an extra sensitive doc?

« on: November 10, 2020, 08:56:25 AM »

We might do that... but then we would need each integrated system to publish/handle the data in that field and build one solution each.

26

Tips and Tricks / Re: How to let user know they are viewing an extra sensitive doc?

« on: November 02, 2020, 04:19:44 AM »

Thanks for the suggestions.

There is however nothing in the document itself that determines this, it is a "property" of the person the document pertains to.
I.e. if someone starts working for the CIA, all documents belonging to this person suddenly becomes "sensitive" (or "classified" if you wish).
This info is in a different database, so we WILL know if this is the case. But how to display it to the few authorized viewers of such documents? OD and CN clients won't care.

27

Tips and Tricks / How to let user know they are viewing an extra sensitive doc?

« on: October 20, 2020, 07:32:49 AM »

Hi!

We have a solution in place where we use one of the OD user exits. If a user is not authorized to view a particular customer's document(s), we won't let them. This solution is based on one of the metadata field values, and a lookup in an external DB/web service.
NOTE: Whether a customer is "extra sensitive" or not is most oftenly set LONG time after we load the documents into CMOD. There is no reasonable way of setting this in a metadata field at load time.

However, some users ARE authenticated to view these documents. Now;
How would you suggest to inform those users that this is an extra sensitive document?
Oh, and we are talking about the thick OD Client, as well as anyone accessing via Content Navigator.

If it was merely AFP documents, we could probably add a watermark with this info, but there are several different file formats, including PDF, a standard which is prone to change over time. AFP, TIFF and PDF would be a minimum to cover for the solution.
Also, we don't want to make any permanent changes to the documents, since the "extra sensitive" status is usually temporary (years).

Post your suggestions, I'm all ears/eyes!

28

MP Server / Re: How to delete single statements from a CMOD Load File

« on: January 27, 2020, 02:57:56 AM »

Hi Lars,
Thankyou for the quick reponse let me see the guideline which you have shared .
Thankyou

This type of query should get you all of the above. Replace anything starting with "my" with your own, real object names:

arsdoc query -f "MyFolder" -h ARCHIVE -i "WHERE MyID='MYID_12345'" -u MyUsername -p ~/myStashfile.stash -v -G MyApplicationGroup -H -D -I p

29

OD/WEK & JAVA API / Re: Is anyone using the load calls in ODWEK as their primary method for loading?

« on: January 27, 2020, 02:50:11 AM »

Hey again,
Of course you can look inside the 87's too.
In this environment, we did indeed only load one doc at a time. But loading bigger batches is simply a matter of "looping" the LoadAddDoc step, once per document, until you finally run LoadCommit only once.

That also answers your third question - to the best of my knowledge, the only way to enter index data is by using the LoadAddDoc step. That means you probably have to parse index data in your intended input files in order to index the documents properly. I currently do not know of any way to index the documents locally on the OD Server using ACIF or PDF indexer.

As per your question whether I expect a performance penalty: it may actually be the other way around. Sure, there is a penalty for transferring the file to the OD Server via ODWEK, but it is probably not a lot greater than transferring the file any other way. And when it comes to the indexing - there is none! Since you have already entered the index data programmatically during the LoadAddDoc step, on the ODWEK server. When I look in the System Load table, the column "Index elapsed time (seconds)" is always zero for these documents. So, you may actually have a performance bonus instead of a penalty - but I suggest you test this properly in the actual environment you will be using. (For example if you run ODWEK on the same server as your Load/Library server, you will probably not decrease the OD Server load very much, if at all.)
 

2018-03-14 12:12:20.325756: ARS4315I Processing file >/opt/ondemand/arscache/tmp/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111220153101.139912273372928.ARD<
2018-03-14 12:12:20.325860: ARS4334I Load Version <9.5.0.9>  Operating System <Linux> <#1 SMP Fri Sep 22 12:32:14 EDT 2017.2.6.32-696.13.2.el6.x86_64>  OS Userid <ODADMIN>  Install Location </home/odadmin/ibm/ondemand/V
2018-03-14 12:12:20.325890: ARS4335I Server Version <9.5.0.9>  Operating System <Linux> <#1 SMP Fri Sep 22 12:32:14 EDT 2017.2.6.32-696.13.2.el6.x86_64>  Database <DB2> <10.01.0006>
2018-03-14 12:12:20.345012: ARS4339I Application Group >BANKDOC-NY<
2018-03-14 12:12:20.345039: ARS4340I Application >BANKDOC4PDF<
2018-03-14 12:12:20.345048: ARS4341I Storage Set >Cache Only - Library Server<
2018-03-14 12:12:20.345056: ARS4342I Storage Node >Cache Only - Library Server<
2018-03-14 12:12:20.345098: ARS4312I Loading started, 3146122 bytes to process
2018-03-14 12:12:20.399537: ARS1144I OnDemand Load Id = >5083-1-0-1198FAA-20180202000000-20180202000000-5089<
2018-03-14 12:12:20.416971: ARS1146I Loaded 1 rows into the database
2018-03-14 12:12:20.424766: ARS1175I Document compression type used - OD77.  Bytes Stored = >3268< Rows = >1<
2018-03-14 12:12:20.424830: ARS4310I Loading completed
2018-03-14 12:12:20.427613: ARS4317I Processing successful for file >/opt/ondemand/arscache/tmp/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111220153101.139912273372928.ARD<

30

OD/WEK & JAVA API / Re: Is anyone using the load calls in ODWEK as their primary method for loading?

« on: January 20, 2020, 09:39:05 AM »

Hi Justin!

We have been doing this for a few customers, as well as in lab & test environments of course. Although there are limits of course, we have had no issues with this method of loading data whatsoever.

First you need to set up a "landing pad" on the OD server. This is where files transferred with the Load API will "land" before they are loaded into OnDemand.
You set these two parameters in the ars.cfg file:

ARS_DOWNLOAD_DIR=/opt/ondemand/arsload2
ARS_DOWNLOAD_TMP_DIR=/opt/ondemand/arsload2/tmp

Failure to do so properly (folder security for instance) may render something like MsgNum 119, "Unable to write to <path>. The error number is 28.", or similar errors.

But once you get past that, you will start to see MsgNum 441 in the log for successful transfers:
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.081) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937617.139912288081664.ARD)
These will then (hopefully) be followed by an ordinary MsgNum 87, or an 88. (and their usual accompanying messages, like 86)

I included some manipulated System Log output for you to look at, I included some initial errors too:

Error   No      119   Unable to write to file >/opt/ondemand/arsload2/ARS.4352.00007F2C57FFF700.BULKLOG<.  The error number is 28  Srvr->localhost 127.0.0.1 non-SSL<-
Info   Yes       87   Application Group Load: Name(ARTURTEST) LoadId(5123-1-0-12FAA-20181022000000-20181022000000-5124) File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.133525823777.139918805210880.ARD) InputSize(1060) OutputSize(927) Rows(1) Time(0.0408) Appl(ARTURPDF) InputFileSize(1534)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1534) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.133525823777.139918805210880.ARD)
Info   Yes       87   Application Group Load: Name(ARTURTEST) LoadId(5123-1-0-11FAA-20181022000000-20181022000000-5124) File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.093116676754.139918805210880.ARD) InputSize(1060) OutputSize(927) Rows(1) Time(0.0416) Appl(ARTURPDF) InputFileSize(1534)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1534) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.093116676754.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(ARTURTEST) LoadId() File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.091817013480.139918805210880.ARD)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1622) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.091817013480.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(ARTURTEST) LoadId() File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.090522683240.139918805210880.ARD)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1622) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.090522683240.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(ARTURTEST) LoadId() File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181019.130548870132.139918805210880.ARD)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1622) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181019.130548870132.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(BANKDOC-NY) LoadId() File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDFs.ODADMIN.20180326.125734051301.139912253396736.ARD)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDFs.ODADMIN) Bytes(1444) Time(0.000) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDFs.ODADMIN.20180326.125734051301.139912253396736.ARD)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1583FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180320.122223189469.139912253396736.ARD) InputSize(1060) OutputSize(939) Rows(1) Time(0.1476) Appl(BANKDOC4PDF) InputFileSize(1456)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(1456) Time(0.000) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180320.122223189469.139912253396736.ARD)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1580FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937704.139912285980416.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.1764) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1581FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937617.139912288081664.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.1370) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1582FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937614.139912273372928.ARD) InputSize(3145728) OutputSize(3267) Rows(1) Time(0.1631) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.081) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937617.139912288081664.ARD)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.084) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937704.139912285980416.ARD)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.084) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937614.139912273372928.ARD)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1573FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111312572140.139912300951296.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.2457) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1574FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111312673416.139912288081664.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.1244) Appl(BANKDOC4PDF) InputFileSize(3146122)