Topics

1

MP Server / Help needed: reverse-engineer on-disk cache format in MP V9.5

« on: March 11, 2024, 02:08:31 AM »

Background: cache-only setup, database backup failure, cache filesystem backups (with TSM) complete.
Documents stored "as is" with the generic indexer only, so no separation of resources and document data.

Need to recover some documents from AG where arsmaint -c and arsmaint -d have been run where the segment date incorrectly was set to 1970-01-01 on some documents...

I've restored the affected DOC files from backup (into another folder), both the 1136FAA1 (OD77-compressed metadata) and 1136FAAA (OD77-compressed documents).
I've run arsadmin decompress on the restored files, and have noticed that the ...FAA1 files contain the document metadata, and the ...FAAA etc files contain the documents themselves.

This far, I've noticed that the first line (sometimes lines) begin with "<" and end with ">" and in between contain tab-separated AG field names.
Then follows lines with the metadata, one line for each document. First comes the values of the AG field names, in the same order as in the <>-enclosed header, and then some CMOD-specific fields that could look like this:
1136FAAA   0   21945   0   106262   U   O   0   1   0

The first is obviously the name of the file containing the documents, and the second and third one (0 + 21945) is the byte offset and length of the document, after decompression, in the document data file.

But, what are the other fields? 0, 106262, U, O, 0, 1, 0 ?

In this example file, the first 29 documents make perfect sense. Here is the CMOD-data for documents 27 - 32 in the decompressed 1136FAA1 file:

1136FAAA  572939     21888            0      106262   U   O   0   1   0
1136FAAA  594827     21887            0      106262   U   O   0   1   0
1136FAAA  616714     21971            0      106262   U   O   0   1   0
1136FAAA       0     21894       106262      104751   U   O   0   1   0
1136FAAA   21894     22109       106262      104751   U   O   0   1   0
1136FAAA   44003     22005       106262      104751   U   O   0   1   0


The 1136FAAA file is exactly 616714 + 21971 byte after decompression of the entire file, so, at the same time the offset counter drops back to zero, and the second pair of "counters" increase, I don't understand where to find these remaining documents.

Does anyone here know what the 0 / 106262 / 104751 in the columns after the first offset+length pairs mean?
Do you think there could be a way to salvage the remaining documents from the cache backups, without using the database?

Thanks!
Matt

2

MP Server / Manually retrieving a document?

« on: July 31, 2019, 05:19:54 AM »

Hello!

We have been running CMOD for Multiplatform in a cache-only mode for a very long time, and are currently on v9.5.
While doing a large amount of exports with arsdoc get, I came across some documents that appear to have been corrupted somehow, that could not be extracted. They can also not be opened with the thick client nor with the Content Navigator web gui.

Using arsdoc get with option -n makes it fail on the second document in my selection, and when doing it without the -n flag, it fails on the eleventh. That is probably just an ordering issue, though. I don't think that's worth investigating. I've done single-document queries so I know exactly which documents are damaged.

The messages from arsdoc while retrieving are:

2019-07-24 12:08:26.270740: (2): ARS6073I Retrieving document for userid 'ADMIN' ...
2019-07-24 12:08:26.294203: ARS6096E Retrieve unsuccessful
2019-07-24 12:08:26.294313: ARS2094E The server failed while retrieving a document

There is nothing in the system log when this happens. Using the native windows client or the web client (Content Navigator) on the corrupted documents also logs nothing into the System Log.

Using arsdoc query with the -D flag, adds this information to the output: ,801FAAB,148456,36678,976552,243727,U,O,0,1,0.
Am I correct in thinking that the compressed document should be somewhere in /arscache/cache1/retr/HDB/DOC/801FAAB ?

Example:

lrwxrwxrwx    1 archive  db2adm           38 Apr 29 2016  /arscache/cache1/retr/HDB/DOC/801FAAB -> /arscache/cache8/19130/HDB/DOC/801FAAB
-r--------    1 archive  db2adm      7798612 Apr 01 2012  /arscache/cache8/19130/HDB/DOC/801FAAB

ARSMAINT has no complaints about the links, and the permissions and ownership on the retr link and the actual file it points to are correct, and the link target can be read with tools like wc or strings, so there is no underlying I/O error.

So, I guess what I'm wondering now is how to find the offset and length in the DOC file. Could it be the second (148456) and third (36678) parameters in the "Document handle" that the -D flag provides to the arsdoc query output?
If so, I guess the only thing I need to figure out is how to do an OD77 decompress "by hand"  - Is there a command line tool for that?

Obviously the document retrieval in the clients should work if the above works, but I still would like to give it a final try to just to be absolutely certain that these documents are lost forever.

Any further ideas? Anyone knows what _all_ the parts of the "document handle" are?

Thanks!
Matt

3

Content Navigator / Pass CMOD Folder search parameters in URL to Content Navigator 2.0.3?

« on: September 14, 2017, 05:24:20 AM »

Hello!

I'm having trouble finding documentation about how to pass search parameters to an OnDemand folder in the URL to Content Navigator. I run ICN 2.0.3 (located behind a separate http(s) proxy (running apache), don't ask me why).

What I would like to be able to do is to from another web application redirect the user to a search/result in a folder. I assume it can be done, at least under normal circumstances (if the user is already logged in to Content Navigator)?
So far, I've found https://www.ibm.com/support/knowledgecenter/en/SSEUEX_2.0.2/com.ibm.developingeuc.doc/eucbd001.html which doesn't tell me enough about Content Manager OnDemand folders.

Has anyone else done this? How?

Kind regards
Matt