Messages

1

MP Server / Re: Manually retrieving a document?

« on: March 15, 2024, 04:21:19 AM »

Still curious about the 2nd and 3rd parameters: 148456,36678.

They are the offset and length of the decompressed document within the compressed document block.
The 4th and 5th parameter (976552,243727) are the offset and length of that compressed block within the 801FAAB object (DOC) file.

So, first decompress the 243727 bytes starting at byte 976552 of 801FAAB, and then take the 36678 bytes starting at 148456 of that decompressed output, for the actual document.

2

MP Server / Re: Help needed: reverse-engineer on-disk cache format in MP V9.5

« on: March 11, 2024, 03:47:54 AM »

Okay...

I think I figured it out now. The second pair of offset (106262) and length (104751) is the location in the uncompressed data file of the compressed "block" where the first pair of offset (0) and length (21894) of the actual document:

1136FAAA       0     21894       106262      104751   U   O   0   1   0

I fooled myself when running arsadmin decompress without offset and length and got the first section decompressed without issues.
Didn't realize that these pdf documents could be compressed so efficiently that 211013 bytes would be decompressed into two files of 638685 + 637376 byte.

Once again, CMOD and it's storage structures impress me! There are indeed some truth to the "things were better back in the day" phrases.
A more recently built solution would probably have been very hard to recover from in a situation where the database with the metadata was lost. Placing the small metadata on disk in the cache file systems is a real life saver!

3

MP Server / Re: CMOD Migration to new DC

« on: March 11, 2024, 03:20:24 AM »

What I once did, (on a cache-only system) was;

Nightly cron job on the "current" (old/source) server doing DB2 exports into files with DB2MOVE + DB2LOOK,
then rsync of the database exports and cache file systems,
On the receiving (new/target) server, another cron job that re-created the DB2 database from the export files, and then performed a CMOD DB version upgrade (because why not )

Then we had time to test the new installation with current contents, and we even did the all new loads/ingestions in both systems for a while (with the older system being mandatory, since the new one was being rebuilt each night).

When we were satisfied that the new system was functional and reliable, we stopped syncing and killed the old server.

I'm not saying that this is how it should be done, but it worked very well for us. The rsync was very useful since we had a lot of data (being a cache-only setup with everything online on disk), and didn't want a very long downtime to make the switch.

4

MP Server / Help needed: reverse-engineer on-disk cache format in MP V9.5

« on: March 11, 2024, 02:08:31 AM »

Background: cache-only setup, database backup failure, cache filesystem backups (with TSM) complete.
Documents stored "as is" with the generic indexer only, so no separation of resources and document data.

Need to recover some documents from AG where arsmaint -c and arsmaint -d have been run where the segment date incorrectly was set to 1970-01-01 on some documents...

I've restored the affected DOC files from backup (into another folder), both the 1136FAA1 (OD77-compressed metadata) and 1136FAAA (OD77-compressed documents).
I've run arsadmin decompress on the restored files, and have noticed that the ...FAA1 files contain the document metadata, and the ...FAAA etc files contain the documents themselves.

This far, I've noticed that the first line (sometimes lines) begin with "<" and end with ">" and in between contain tab-separated AG field names.
Then follows lines with the metadata, one line for each document. First comes the values of the AG field names, in the same order as in the <>-enclosed header, and then some CMOD-specific fields that could look like this:
1136FAAA   0   21945   0   106262   U   O   0   1   0

The first is obviously the name of the file containing the documents, and the second and third one (0 + 21945) is the byte offset and length of the document, after decompression, in the document data file.

But, what are the other fields? 0, 106262, U, O, 0, 1, 0 ?

In this example file, the first 29 documents make perfect sense. Here is the CMOD-data for documents 27 - 32 in the decompressed 1136FAA1 file:

1136FAAA  572939     21888            0      106262   U   O   0   1   0
1136FAAA  594827     21887            0      106262   U   O   0   1   0
1136FAAA  616714     21971            0      106262   U   O   0   1   0
1136FAAA       0     21894       106262      104751   U   O   0   1   0
1136FAAA   21894     22109       106262      104751   U   O   0   1   0
1136FAAA   44003     22005       106262      104751   U   O   0   1   0


The 1136FAAA file is exactly 616714 + 21971 byte after decompression of the entire file, so, at the same time the offset counter drops back to zero, and the second pair of "counters" increase, I don't understand where to find these remaining documents.

Does anyone here know what the 0 / 106262 / 104751 in the columns after the first offset+length pairs mean?
Do you think there could be a way to salvage the remaining documents from the cache backups, without using the database?

Thanks!
Matt

5

MP Server / Re: Manually retrieving a document?

« on: October 02, 2019, 05:27:07 AM »

Hi again :-)

I did some more digging, and the PDF i got matched the metadata of another document.

The document I was hoping to find had these attributes:
801FAAB,148456,36678,976552,243727,U,O,0,1,0,2423-1-0-801FAA-15416-15416-2424

And the document I found had these attributes:
801FAAB,222153,37134,976552,243727,U,O,0,1,0,2423-1-0-801FAA-15428-15428-2424

So I guess I don't fully understand how the data in the cache filesystem is stored.
Any ideas?

6

MP Server / Re: Manually retrieving a document?

« on: October 02, 2019, 05:16:37 AM »

Thank you so much, Lars!

In the -D information from arsdoc query, in my case: ,801FAAB,148456,36678,976552,243727,U,O,0,1,0

It looks like the 4th value (976552) is indeed a valid document offset in the 801FAAB file. When giving it as the "begin" parameter to arsadmin decompress, I do get a valid PDF document output (if I adjust it to another nearby value I get an error message). If I pass the 5th value (243727) as the length I get the same results. If I supply a smaller length, it appears to just hang, so I figure it must be the compressed length or something like that.

However, in my case, the resulting PDF Document doesn't look like it belongs to the document metadata, so I'm still confused about what the h*** is going on here. Unfortunately. 

Still curious about the 2nd and 3rd parameters: 148456,36678.

But, arsadmin decompress was very valuable to learn about, nonetheless.
Thanks again!

7

MP Server / Manually retrieving a document?

« on: July 31, 2019, 05:19:54 AM »

Hello!

We have been running CMOD for Multiplatform in a cache-only mode for a very long time, and are currently on v9.5.
While doing a large amount of exports with arsdoc get, I came across some documents that appear to have been corrupted somehow, that could not be extracted. They can also not be opened with the thick client nor with the Content Navigator web gui.

Using arsdoc get with option -n makes it fail on the second document in my selection, and when doing it without the -n flag, it fails on the eleventh. That is probably just an ordering issue, though. I don't think that's worth investigating. I've done single-document queries so I know exactly which documents are damaged.

The messages from arsdoc while retrieving are:

2019-07-24 12:08:26.270740: (2): ARS6073I Retrieving document for userid 'ADMIN' ...
2019-07-24 12:08:26.294203: ARS6096E Retrieve unsuccessful
2019-07-24 12:08:26.294313: ARS2094E The server failed while retrieving a document

There is nothing in the system log when this happens. Using the native windows client or the web client (Content Navigator) on the corrupted documents also logs nothing into the System Log.

Using arsdoc query with the -D flag, adds this information to the output: ,801FAAB,148456,36678,976552,243727,U,O,0,1,0.
Am I correct in thinking that the compressed document should be somewhere in /arscache/cache1/retr/HDB/DOC/801FAAB ?

Example:

lrwxrwxrwx    1 archive  db2adm           38 Apr 29 2016  /arscache/cache1/retr/HDB/DOC/801FAAB -> /arscache/cache8/19130/HDB/DOC/801FAAB
-r--------    1 archive  db2adm      7798612 Apr 01 2012  /arscache/cache8/19130/HDB/DOC/801FAAB

ARSMAINT has no complaints about the links, and the permissions and ownership on the retr link and the actual file it points to are correct, and the link target can be read with tools like wc or strings, so there is no underlying I/O error.

So, I guess what I'm wondering now is how to find the offset and length in the DOC file. Could it be the second (148456) and third (36678) parameters in the "Document handle" that the -D flag provides to the arsdoc query output?
If so, I guess the only thing I need to figure out is how to do an OD77 decompress "by hand"  - Is there a command line tool for that?

Obviously the document retrieval in the clients should work if the above works, but I still would like to give it a final try to just to be absolutely certain that these documents are lost forever.

Any further ideas? Anyone knows what _all_ the parts of the "document handle" are?

Thanks!
Matt

8

Content Navigator / Re: Pass CMOD Folder search parameters in URL to Content Navigator 2.0.3?

« on: March 14, 2018, 01:15:21 AM »

I found the answer here: https://www.ibm.com/support/knowledgecenter/en/SSEUEX_3.0.3/com.ibm.installingeuc.doc/eucco107.htm
Just submit an array with the two date values:

{"property":"Fakturadatum","value":["2017-09-04","2017-09-04"]}

9

Content Navigator / Re: Pass CMOD Folder search parameters in URL to Content Navigator 2.0.3?

« on: March 14, 2018, 12:25:18 AM »

I have a follow-up question regarding Content Navigator in front of Content Manager OnDemand (for multiplatforms).

How do I pass a Date range, or an exact date, to folders that default to Between (but also allow Equals)?
If not passing the date, the search will be limited to the last month, which is not what I want.

Anyone have experience with this?

Thanks!
Matt

10

Content Navigator / Re: Pass CMOD Folder search parameters in URL to Content Navigator 2.0.3?

« on: October 03, 2017, 01:36:19 AM »

Thank you Lisa!
It was indeed the "docid" and "template_name" parameters that was confusing.

I can confirm that it works excellent in ICN 2.0.3 with a CMOD 9.5 backend.

Opening of the search dialog: works just as you describe (calling bookmark.jsp with the folder name as docid as well as template_name).
Populating it with search parameters: works just as you describe (JSON format, but I didn't need to urlencode it when testing from my browser).
Immediately starting the search: works just as you describe (passing autoRunSearch=true).

Kind regards
Matt

11

Content Navigator / Re: Pass CMOD Folder search parameters in URL to Content Navigator 2.0.3?

« on: October 02, 2017, 07:53:04 AM »

Thank you so much, lgartside, for taking the time to respond!
I will look into this shortly and share my results.

12

Content Navigator / Pass CMOD Folder search parameters in URL to Content Navigator 2.0.3?

« on: September 14, 2017, 05:24:20 AM »

Hello!

I'm having trouble finding documentation about how to pass search parameters to an OnDemand folder in the URL to Content Navigator. I run ICN 2.0.3 (located behind a separate http(s) proxy (running apache), don't ask me why).

What I would like to be able to do is to from another web application redirect the user to a search/result in a folder. I assume it can be done, at least under normal circumstances (if the user is already logged in to Content Navigator)?
So far, I've found https://www.ibm.com/support/knowledgecenter/en/SSEUEX_2.0.2/com.ibm.developingeuc.doc/eucbd001.html which doesn't tell me enough about Content Manager OnDemand folders.

Has anyone else done this? How?

Kind regards
Matt

13

WEBI interface / Re: IE 8 Blank Page -

« on: August 29, 2014, 08:15:49 AM »

I had the problem with a completely blank WEBi login page in IE 11.

The problem is that IE cannot load binary data with ajax if it is not encoded properly, and WEBi uses ajax to load some big parts of the dojo framework asynchronously.

I "solved" it (explained at http://serverfault.com/q/625004/50874 ) by decompressing the javascripts and css files that were gzipped, and actually got it to work in IE11 too!

Why it stopped working in IE11 is probably because Microsoft chose to change the UserAgent (removing "MSIE" from it), which makes old frameworks fail the guess-the-browser game on IE11, which in turn causes the code to miss the extra hoops that must be jumped through on microsofts browsers, since IE11 appears more like Mozilla in the user agent string.

14

MP Server / Re: CMOD Warning after Successful Loads

« on: July 23, 2012, 12:15:42 AM »

Hi!

Just wanted to tell you that omitting

Code: [Select]

arssyscr -a (thereby not creating the "System Load" tables) helped me to get rid of the error.


/Mattias

15

MP Server / Re: CMOD Warning after Successful Loads

« on: July 20, 2012, 03:07:50 AM »

Document compression type used - OD77. Bytes Stored = >6952< Rows = >2<
arsload: 12/09/11 07:03:22 Loading completed
arsload: Processing successful for file >/prod/msp/data/ei_data/eip/cmod/File.AFP<
An error occurred. Contact your system administrator and/or consult the System Log. File=arsadmin.c, Line=2587
arsload: Unable to log load information
arsload: Processing has stopped. The remaining files will NOT be processed.

I am able to open the loaded documents from the OD Client after the load.

I have the same problem, on Multiplatform:

Code: [Select]

Load Version <8.5.0.3> Operating System <AIX> <6.1>
Server Version <8.5.0.3> Operating System <AIX> <6.1> Database <DB2> <09.07.0003>
07/20/12 10:32:02 -- Loading started, --UNKNOWN-- bytes to process
OnDemand Load Id = >2601-1-0-2080FAA-15542-15542<
Loaded 2 rows into the database
Document compression type used - OD77.  Bytes Stored = >43108< Rows = >2<
07/20/12 10:32:04 Loading completed
Processing successful for file > .... <

An error occurred.  Contact your system administrator and/or consult the System Log.  File=arsadmin.c, Line=2587
Unable to log load information
Processing has stopped.  The remaining files will NOT be processed.

Except for this error, everything seems fine.

I'm about to migrate to this system from 7.1.1.1, so I suspect I've done something wrong in the upgrade step.
I did on the old system:

Code: [Select]

db2move archive export
db2look -d archive -a -c -e -l -x -z root -o lookfile.dat

and on the new all sorts of stuff:
dropping the database,
creating it,

Code: [Select]

db2 -tvf lookfile.dat
db2move archive load -lo replace
arsdb -mv
arssyscr -l
arstblsp -a 1 -g "System Log"
and then

Code: [Select]

arssyscr -a
and for tables ARSUSRGRP ARSLOAD ARSNODE ARSAPP:

Code: [Select]

arsdb -rv
and then another

Code: [Select]

arsdb -mv

Tonight I'm going to try omitting the

Code: [Select]

arssyscr -a.
Could that help, you think?

Thanks!
Mattias