Author Topic: arsdoc get not returning all hits?  (Read 3284 times)

Lars Bencze

  • Full Member
  • ***
  • Posts: 116
  • CMOD Expert at Skandia
    • View Profile
    • INACTIVE - Bezland Consulting
arsdoc get not returning all hits?
« on: November 29, 2016, 04:31:10 AM »
EDITED: I just found THIS issue, which probably covers this question. Will need to do some testing though to verify it.
http://www.odusergroup.org/forums/index.php?topic=996.0

Hi,

This is a long shot, something for all you OnDemand-lovers out there!

We currently have a customer, on OD 8.5.0.9 running on AIX, where we recently experienced something very weird. A PMR is opened, and we are working with IBM on this, but we have been unable to efficiently reproduce the problem on purpose - thus far, at least.

We made a script which had the purpose of doing this (plus a little extra, but we can pinpoint the error to these two steps):
* Export all documents within a named LoadId
* Unload all documents in that same LoadId - although here the two last parts of the LoadId are replaced with "-0-0" for practical reasons.

The result is, somewhat surprisingly, that on step 1, X number of documents are exported.
While in step 2, a number GREATER than X is Unloaded!!!

This shouldn't be possible, no.
The number in step two, for example "34", matches the number reported in the System Log MsgNum 87 (arsload) for the same load. But when we exported that LoadId, we only got 33 documents.

Can you guys please think of any situation where this may come true?
YES: we use Enhanced Retention Management on this system. Between step 1 and step 2, we remove ALL Holds on all documents within the LoadId. Never any problem with this, all documents in the LoadId ALWAYS get deleted. IF there was a problem with this step, the resulting number of deleted documents should be LESS THAN the number that got exported, but it is the other way around.

Here are the exact commands that we used in the script. Let us know if they are incorrect in some way, which may cause the behaviour above.
Also, let us know if you in any other setting have experienced similar problems, please describe when and how.

The exact scripted arsdoc get command looked like this:
${OD_BASE_CATALOG}/arsdoc get -a -d ${WORKDIR} -G ${APPGRP} -X ${LOAD_ID} -g -h ${OD_INSTANCE} -o ${PREFIX} -u ${USERNAME} -p ${PASSWORD} -A 7 -N -c > tmp.txt 2>&1

Which instantiated would translate to something like this (Sensitive information changed for security reasons):
arsdoc get -a -d /opt/ondemand/arsdownload/erm/working -G OURAPPGROUP -X 6099-13-0-2129FAA-16683-16683 -g -h ARCHIVE -o FILE2129  -u ODUSER -p xxxxxxxx -A 7 -N -c -v >> FILE2129.txt 2>&1

The exact scripted arsadmin unload looked like this:
${OD_BASE_CATALOG}/arsadmin unload -h ${OD_INSTANCE} -g ${APPGRP} -u ${USERNAME} -p ${PASSWORD} -Q -L ${LOAD_ID} > tmp.txt 2>&1

Instantiated, something like this:
arsadmin unload -h ARCHIVE -g OURAPPGROUP -u ODUSER -p xxxxxxxx -Q -L 6099-13-0-2129FAA-0-0 > tmp.txt 2>&1
« Last Edit: November 30, 2016, 06:47:19 AM by Justin Derrick »
OnDemand for MP expert. #Multiplatforms #Admin #Scripts #Performance #Support #Architecture #PDFIndexing #TSM/SP #DB2 #CustomSolutions #Integration #UserExits #Migrations #Workflow #ECM #Cloud #ODApi

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2231
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: arsdoc get not returning all hits?
« Reply #1 on: November 30, 2016, 06:47:30 AM »
Edited to remove passwords.
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

Lars Bencze

  • Full Member
  • ***
  • Posts: 116
  • CMOD Expert at Skandia
    • View Profile
    • INACTIVE - Bezland Consulting
Re: arsdoc get not returning all hits?
« Reply #2 on: December 02, 2016, 05:21:55 AM »
Edited to remove passwords.
Heh, thanks - although the ones I put there were of course not the real ones.
Quote
(Sensitive information changed for security reasons)
OnDemand for MP expert. #Multiplatforms #Admin #Scripts #Performance #Support #Architecture #PDFIndexing #TSM/SP #DB2 #CustomSolutions #Integration #UserExits #Migrations #Workflow #ECM #Cloud #ODApi

Lars Bencze

  • Full Member
  • ***
  • Posts: 116
  • CMOD Expert at Skandia
    • View Profile
    • INACTIVE - Bezland Consulting
Re: arsdoc get not returning all hits?
« Reply #3 on: December 02, 2016, 05:28:49 AM »
Anyways, due to this anomaly above, we cannot use "-X" with arsdoc if we want to export all documents inside a particular LoadId.
It is a pity that there is no "-L" parameter which works in the same manner as during "arsadmin unload ... -L <LOADID>" - that command appears to always find the exact right amount of records to delete.

Also, we have earlier run tests that show that we cannot use "-x" (lower case x) either, since this OnDemand system contains records that are so old, that "-x" won't produce the correct hits. (If I recall correctly, it cannot find the Load at all, this produces zero hits/exports zero documents)

So, how DO we get all records in a LoadId exported then????
Good question. We are currently experimenting with SQL commands like "WHERE DOC_NAME='<The LoadId plus an extra "A">' " and similar, but if one of you guys (JD?) have already solved this dilemma, please share!

All the best,

Lars
OnDemand for MP expert. #Multiplatforms #Admin #Scripts #Performance #Support #Architecture #PDFIndexing #TSM/SP #DB2 #CustomSolutions #Integration #UserExits #Migrations #Workflow #ECM #Cloud #ODApi

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2231
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: arsdoc get not returning all hits?
« Reply #4 on: December 02, 2016, 07:13:04 AM »
Hi Lars.

I have some extraction tools that you might be interested in.  I'll contact you off the forum.

-JD.
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

zancanaro

  • Guest
Re: arsdoc get not returning all hits?
« Reply #5 on: December 22, 2016, 09:14:10 AM »
Hi Lars,

I often copy many records from CMOD Product Instance to a CMOD Development  (between 2 physical AIX and so on..)   ; These copies between 1 to 6.000.000 records on each.

But I don't use a LoadId number but moreover ranges of dates  -->    If you know the exact dates of your LoadIds,  ;)   no problem for that.

The 2 Files System have exactly the same names == Production "/home/userXX"   AND  Devlopment "/home/userXX" too.  (This name is in the .index files)
For big ranges, be careful of disk spaces on the 2 AIX.
Then :
A)   From original CMOD Production
1 - Create parmfile (unload_ROCKANDROLL.parmf)   like this :
 [-a][-c][-g][-N][-d /home/userXX][-f ROCKANDROLL][-G ROCKANDROLL][-i "WHERE datedoc='10958'"][-o "ROCKANDROLL_001"]
2 - Execute this parmfile named "unload_ROCKANDROLL.parmf"
/apps/ondemand/ars/bin/arsdoc get -F   unload_ROCKANDROLL.parmf   -h ARCHIVE -uxxxxxxx -p /xxxxxxxxxxxxxxxxxx/ars.stash -v

B)   Copy all the files on the Devlopment AIX's CMOD (careful =  text, binary moves for .ARD and .ind files)

C)   On The Devlopment CMOD partition when received
/apps/ondemand/ars/bin/arsload -I archtest uxxxxxxx -p /xxxxxxxxxxxxxxxxxx/arsDEV.stash    -X G -g CLASSICMUSIC     -vfn  ROCKANDROLL_001

Some of these are Embedded, but manually, you can do that.

Best regards,
Stephane Z.