How can I use Load-Id in ODSearchCriteria? to retrieve a doc

[premk]

Input:
- I have a load-id, that's all.

Output:
- I need to retrive a relevant document [typically one document/hit for that specific load-id].

How can I set it as a search criteria?

Code: [Select]

         ODCriteria critApplicationComet = folder.getCriteria("LOAD_ID");

// currently I don't see LOAD_ID as one of the columns in ondemand search client.  Should I include LOAD_ID as a column for that folder - to have that available as ODCriteria? How - only using admin interface?

//I assume the load-id data is already available in system/db, how to make it available in a particular folder so that I can use it in getCriteria()?

         critApplicationComet.setOperator(ODConstant.OPEqual);
         critApplicationComet.setSearchValue("MYAPPNAME");
At the moment I don't have access to Ondemand admin interface, if there is a way I can do it programmatically that would be great.

Thanks
Prem

Integrating Ondemand to a centralized archival system - SOA architecture in banking sector.

[premk]

In another post I read:

You can retrieve a loaded file using the "arsdoc get" command -- specifically, use the -X option and specify the load ID in order to get the actual document back. 

That's exactly the same thing I want to do, but, using ODWEK java api.

[Alessandro Perucchi]

Hello Prem,

Well you cannot do that like that :-) that would be nice I agree :-D

The load id is a computed value that use many component of the CMOD system.

The first number is the Application Group ID (AGID) of the application group where the document is stored.
The second number is the primary storage node ID where the document is stored.
The third number is the seconday storage node ID where the document is stored
The fourth is the document name (number) or the load ID for that application group.
The fifth is the begin date of the segment date field for this load.
The sixth is the end date of the segment date field for this load.

Meaning, that if you have ONLY the load-ID with no other information, then you can find the document, but that's a difficult work, especially if you are using only ODWek.
I mean,if you had a DB2/Oracle JDBC connection where you could look directly into the CMOD tables it would be easy.

So if you had this DB2/Oracle JDBC connection here is how to do it:

1) get from the load ID the first number, and search in the ARSAG table to find the AGID corresponding.
2) Search all tables from the ARSSEG where you have the AGID you found in 1), and use the 5/6th number to search only in those tables (START_DATE and END_DATE field and the table name = TABLE_NAME field)
3) with the list of TABLE_NAME found in the ARSSEG in 2), you can search now for your documents. Let say that the table is called AJA3. Then with the second, third and fourth field, you can do a "select * from AJA3 where DOC_NAME like 'fourth_field%' and PRI_NID=second_field and SEC_NID=thrid_field"

At that point, you have done a "query" directly in the database but you cannot retrieve the documents.

But you know the application group, and all the fields to retrieve the documents with ODWek. You might even search for all the folder that are connected to that application group with the help of the table "ARSAG2FOL" and "ARSFOL". You could do it with ODWek, by scanning all the folders and their mapping with every application group...
So when you have the folder, and all the fields corresponding, then you could do a retrieval of all documents with odwek.


Another way to do it, is to wrap the arsdoc in Java (I think with some Reflection libraries).


Now to do it only with ODWek, I need to test some stuff... I don't even know if that is possible. I will do some search and keep you informed if I find a way :-)
But that's clear... "arsdoc get" command is a "few seconds/minutes" work. To do it only with ODWek, or with the help of DB JDBC Connections is a work of days/weeks....
What would be the most efficient? Probably to wrap the arsdoc command in Java.... it will spare you lots of work.
And it will be less fun :-D

Sincerely yours,
Alessandro

[Alessandro Perucchi]

Well, I've played a bit with ODWEK, and did the following code:

WARNING/ LEGAL Stuff: this is just a sample, it is not supposed to be pretty or to be fullproof !!! use it with your own risk. I don't want to be accused of anything if something goes wrong!
You can use this code without any restriction, I don't care what you'll do with it, and please again use it at your own risk :-D

Code: [Select]

import java.util.Enumeration;
import java.util.Vector;

import com.ibm.edms.od.*;

public class GetDocFromLoadID {

public static void main(String[] args) throws Exception {

String loadID = "5001-1-0-1FAA-1328202949-1328203141";

ODConfig ocfg = new ODConfig();
ODServer osrv = new ODServer(ocfg);
osrv.initialize("Testing");
osrv.setServerName("localhost");
osrv.setPort(1445);
osrv.setUserId("USER");
osrv.setPassword("PASSWORD");
osrv.logon();

String agidString = loadID.substring(0, loadID.indexOf("-"));
String restLoadID = loadID.substring(loadID.indexOf("-") + 1,
loadID.length());
String priNIDString = restLoadID.substring(0, restLoadID.indexOf("-"));
restLoadID = restLoadID.substring(restLoadID.indexOf("-") + 1,
restLoadID.length());
String secNIDString = restLoadID.substring(0, restLoadID.indexOf("-"));
restLoadID = restLoadID.substring(restLoadID.indexOf("-") + 1,
restLoadID.length());
String docNameString = restLoadID.substring(0, restLoadID.indexOf("-"));
restLoadID = restLoadID.substring(restLoadID.indexOf("-") + 1,
restLoadID.length());
String startDateNameString = restLoadID.substring(0,
restLoadID.indexOf("-"));
String endDateNameString = restLoadID.substring(
restLoadID.indexOf("-") + 1, restLoadID.length());

System.out.println("LoadID is " + loadID);
System.out.println("-> agid=" + agidString);
System.out.println("-> Pri_NID=" + priNIDString);
System.out.println("-> Sec_NID=" + secNIDString);
System.out.println("-> Doc_Name=" + docNameString);
System.out.println("-> Start_Date=" + startDateNameString);
System.out.println("-> End_Date=" + endDateNameString);
System.out.println();

ODApplicationGroup odag = osrv.getApplicationGroup(Long.parseLong(agidString));
Enumeration<ODApplicationGroupField> agfld = odag.getFields();
System.out.println("Name of the Application Group=" + odag.getName());

Enumeration<ODApplicationGroupField> odAGFlds = odag.getFields();
String agFieldSegment = "";
while (odAGFlds.hasMoreElements()) {
ODApplicationGroupField odAGFld = odAGFlds.nextElement();
if ((odAGFld.getFieldMask() & ODConstant.OD_FLD_SEGMENT) != 0) {
agFieldSegment = odAGFld.getName();
break;
}
}
if (agFieldSegment == "") {
System.out
.println("Could not find a Segment Field in Application Group");
System.exit(1);
}
System.out.println("Segment Field=" + agFieldSegment);

Enumeration<ODFolder> listODFolder = osrv.getFolders();
String folderToSearch = "";
OUT: while (listODFolder.hasMoreElements()) {

ODFolder folder = listODFolder.nextElement();
                        folder.open();
for (String agName : folder.getApplGroupNames()) {
if (agName.equals(odag.getName())) {
folderToSearch = folder.getName();
                                        folder.close();
break OUT;
}
}
                        folder.close();
}

if (folderToSearch == "") {
System.out
.println("Could not find a folder to search the documents");
System.exit(1);
}
System.out.println("Folder to search=" + folderToSearch);
ODFolder odfolder = osrv.openFolder(folderToSearch);
                odfolder.setApplGroupForSearchWithSQL(odag.getName());
Vector<ODHit> odhit = odfolder.search("where doc_name like '"
+ docNameString + "%' and PRI_NID=" + priNIDString
+ " and SEC_NID=" + secNIDString+ " and "+agFieldSegment+">="+startDateNameString+ " and "+agFieldSegment+"<="+endDateNameString);
int counter=0;
for (ODHit singleHit : odhit) {
System.out.println("Retrieving document #"+counter);
singleHit.getDocument("/tmp/"+loadID+"_"+counter+".out",true);
counter++;
}

osrv.logoff();

}

}

Well for me it works and here is the output:

LoadID is 5001-1-0-1FAA-1328202949-1328203141
-> agid=5001
-> Pri_NID=1
-> Sec_NID=0
-> Doc_Name=1FAA
-> Start_Date=1328202949
-> End_Date=1328203141

Name of the Application Group=System Log
Segment Field=time_stamp
Folder to search=System Log
Retrieving document #0
Retrieving document #1
Retrieving document #2
Retrieving document #3
Retrieving document #4
Retrieving document #5
Retrieving document #6
Retrieving document #7

and in my /tmp/ directory I have the following files:

Code: [Select]

$ ls -l /tmp/5001\-1\-0\-1FAA\-1328202949\-1328203141_*
-rw-rw-r--. 1 alex staff 1804 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_0.out
-rw-rw-r--. 1 alex staff 5266 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_1.out
-rw-rw-r--. 1 alex staff 1708 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_2.out
-rw-rw-r--. 1 alex staff 1717 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_3.out
-rw-rw-r--. 1 alex staff 1705 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_4.out
-rw-rw-r--. 1 alex staff  851 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_5.out
-rw-rw-r--. 1 alex staff  851 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_6.out
-rw-rw-r--. 1 alex staff  851 Feb  9 11:18 /tmp/5001-1-0-1FAA-1328202949-1328203141_7.out

I know the code is not documented, the code is not pretty, it needs a lot of refactorisation, but I did it in 30 minutes... with lots of API methods search, and tests....

I hope it helps in solving your question.

Sincerely yours,
Alessandro

[premk]

Hi Alessandro,

Thanks very much for your help, it helped me startup with less time. Here is an updated code if anyone needs it in future.

I've found a bug in the ODFolder.getApplGroupNames() method... it always returns an empty string...

I don't think it is a bug. If you execute myfolder.open(), it gives the app group names.

Here is a snippet code that is slightly more refined [server connection pooling and a entity class are not shown in this snippet ]

Code: [Select]

     public ODDocument getDocForLoadId(String loadId) throws Exception {
        ODDocument myDoc = new ODDocument(loadId);
        ODServer odServer = getServer();
        ODApplicationGroup odag = odServer.getApplicationGroup(myDoc.getAgid());
        Enumeration<ODApplicationGroupField> agfld = odag.getFields();
        log.debug("Name of the Application Group odag=" + odag.getName() + "  id="+odag.getId());

        Enumeration<ODApplicationGroupField> odAGFlds = odag.getFields();
        String agFieldSegment = null;
        while (odAGFlds.hasMoreElements()) {
            ODApplicationGroupField odAGFld = odAGFlds.nextElement();
            //if ((odAGFld.getFieldMask() & ODConstant.OD_FLD_SEGMENT) != 0) {
            if ((odAGFld.getFieldMask() & 0x4) != 0) {// :TODO this is a hack because the servers are 7.1.x and odwek-api is 8.4.x
                agFieldSegment = odAGFld.getName();
                break;
            }
        }
        if (agFieldSegment == null) {
            throw new Exception("Could not find a Segment Field in Application Group");
        }
        log.debug("Segment Field=" + agFieldSegment);

        //Enumeration<ODFolder> listODFolder = odServer.getFolders("LUCI Reports");
        Enumeration<ODFolder> listODFolder = odServer.getFolders();
        String folderToSearch = null;
        OUT: while (listODFolder.hasMoreElements()) {
            ODFolder folder = listODFolder.nextElement();
            log.debug("folName = " +folder.getName() + " apgpcount=" + folder.getNumApplGroups() );
            folder = odServer.openFolder(folder.getName());
            for( String folApGpNames : folder.getApplGroupNames()){
                log.debug("\tfolAGName = " +folApGpNames);
            }
            for (String folAGName : folder.getApplGroupNames()) {
                if (folAGName.equals(odag.getName())) { //For this user
                    folderToSearch = folder.getName();
                    break OUT;
                }
            }
        }
        if (folderToSearch == null) {
            log.error("Could not find a folder to search the documents");
        }
        log.debug("Folder to search=" + folderToSearch);
        ODFolder odfolder = odServer.openFolder(folderToSearch);
        odfolder.setApplGroupForSearchWithSQL(odag.getName());
       
        Vector<ODHit> odhits =
            odfolder.search("where doc_name like '"
                + myDoc.getDocName()
                + "%' and PRI_NID=" + myDoc.getPrimaryNodeId()
                + " and SEC_NID=" + myDoc.getSecondaryNodeId()
                + " and "+agFieldSegment+">="+myDoc.getStartDate()
                + " and "+agFieldSegment+"<="+myDoc.getEndDate());
       
        int counter=0;
//        if(odhits.size() >1){
//            //TODO: Handle multi hit scenario later
//            log.error("One LoadId must return just one hit LoadId="+ loadId);
//            throw new OndemandException("One LoadId must return just one hit LoadId="+ loadId + " hitsize=" + odhits.size());
//        }
        for (ODHit singleHit : odhits) {
            byte [] rawData = singleHit.getDocument();
            myDoc.setContent(rawData);
            counter++;
        }
       
        releaseServer(odServer);
        return myDoc;
    }


[Alessandro Perucchi]

Hello,

thank you for your feedback and the corrected version :-)

One thing... in your code:

Code: [Select]

if ((odAGFld.getFieldMask() & 0x4) != 0) {
you write because your server is 7.1.x, you need to do some hack... but why don't you use: ODConstant.OD_FLDMSK_SEGMENT instead of 0x4 ??
I just realize that my code had a small bug with this constant!! :-D I used the wrong one!

Nevertheless thank you for the correction of my so called "bug" :-D I will remember of this one :-D I will close the PMR.
And also I've corrected my code in my previous post in case somebody wants to use it :-D

Sincerely yours,
Alessandro