Is anyone using the load calls in ODWEK as their primary method for loading?

[Justin Derrick]

I'm at a customer site, and they're re-architecting their load process from lines of business.  Because Java programmers are a commodity, they've decided to create a 100% Java solution that can be deployed on their internal cloud.

One of the programmers came to my desk this morning, and asked me about the ODWEK LoadInit / LoadAddDoc / LoadCommit / LoadReset calls (which were introduced in CMOD v9.5).

I've never heard of someone using this functionality to perform loads.  They've apparently already written the code to parse the generic index files.  I'm concerned for the operational headaches that this could cause.  (What will appear in the System Log?  Where will Load ID's be stored?  How will we get logs of load failures for diagnostics?)

Any insight would be appreciated!

-JD.

[Lars Bencze]

Hi Justin!

We have been doing this for a few customers, as well as in lab & test environments of course. Although there are limits of course, we have had no issues with this method of loading data whatsoever.

First you need to set up a "landing pad" on the OD server. This is where files transferred with the Load API will "land" before they are loaded into OnDemand.
You set these two parameters in the ars.cfg file:

ARS_DOWNLOAD_DIR=/opt/ondemand/arsload2
ARS_DOWNLOAD_TMP_DIR=/opt/ondemand/arsload2/tmp

Failure to do so properly (folder security for instance) may render something like MsgNum 119, "Unable to write to <path>. The error number is 28.", or similar errors.

But once you get past that, you will start to see MsgNum 441 in the log for successful transfers:
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.081) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937617.139912288081664.ARD)
These will then (hopefully) be followed by an ordinary MsgNum 87, or an 88. (and their usual accompanying messages, like 86)

I included some manipulated System Log output for you to look at, I included some initial errors too:

Error   No      119   Unable to write to file >/opt/ondemand/arsload2/ARS.4352.00007F2C57FFF700.BULKLOG<.  The error number is 28  Srvr->localhost 127.0.0.1 non-SSL<-
Info   Yes       87   Application Group Load: Name(ARTURTEST) LoadId(5123-1-0-12FAA-20181022000000-20181022000000-5124) File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.133525823777.139918805210880.ARD) InputSize(1060) OutputSize(927) Rows(1) Time(0.0408) Appl(ARTURPDF) InputFileSize(1534)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1534) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.133525823777.139918805210880.ARD)
Info   Yes       87   Application Group Load: Name(ARTURTEST) LoadId(5123-1-0-11FAA-20181022000000-20181022000000-5124) File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.093116676754.139918805210880.ARD) InputSize(1060) OutputSize(927) Rows(1) Time(0.0416) Appl(ARTURPDF) InputFileSize(1534)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1534) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.093116676754.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(ARTURTEST) LoadId() File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.091817013480.139918805210880.ARD)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1622) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.091817013480.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(ARTURTEST) LoadId() File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.090522683240.139918805210880.ARD)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1622) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181022.090522683240.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(ARTURTEST) LoadId() File(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181019.130548870132.139918805210880.ARD)
Info   No      441   File Transfer:  Name(ARTURTEST.ARTURPDF.ODADMIN) Bytes(1622) Time(0.000) Location(/opt/ondemand/arsload2/ARTURTEST.ARTURPDF.ODADMIN.20181019.130548870132.139918805210880.ARD)
Error   Yes       88   Application Group Failed Load: Name(BANKDOC-NY) LoadId() File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDFs.ODADMIN.20180326.125734051301.139912253396736.ARD)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDFs.ODADMIN) Bytes(1444) Time(0.000) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDFs.ODADMIN.20180326.125734051301.139912253396736.ARD)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1583FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180320.122223189469.139912253396736.ARD) InputSize(1060) OutputSize(939) Rows(1) Time(0.1476) Appl(BANKDOC4PDF) InputFileSize(1456)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(1456) Time(0.000) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180320.122223189469.139912253396736.ARD)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1580FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937704.139912285980416.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.1764) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1581FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937617.139912288081664.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.1370) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1582FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937614.139912273372928.ARD) InputSize(3145728) OutputSize(3267) Rows(1) Time(0.1631) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.081) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937617.139912288081664.ARD)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.084) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937704.139912285980416.ARD)
Info   No      441   File Transfer:  Name(BANKDOC-NY.BANKDOC4PDF.ODADMIN) Bytes(3146122) Time(0.084) Location(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111313937614.139912273372928.ARD)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1573FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111312572140.139912300951296.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.2457) Appl(BANKDOC4PDF) InputFileSize(3146122)
Info   Yes       87   Application Group Load: Name(BANKDOC-NY) LoadId(5083-1-0-1574FAA-20180202000000-20180202000000-5089) File(/opt/ondemand/arsload2/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111312673416.139912288081664.ARD) InputSize(3145728) OutputSize(3268) Rows(1) Time(0.1244) Appl(BANKDOC4PDF) InputFileSize(3146122)

[Justin Derrick]

Hey Lars.  Thanks for the update...

But I see that you appear to be using this to upload individual documents.  Our Java development team wants to use this for *everything* that gets loaded into CMOD.  While I'm releived to see that there are still 87 messages in the System Log, I'm interested to see what's recorded for those messages.  Would you mind sharing them?  Also, do you expect a performance penalty for using this method?  I'm also curious to know how you're providing metadata -- are you still able to have the AG perform indexing by ACIF or the PDF indexer, or are you submitting metadata through the API calls along with the documents...  and how are you getting that metadata into your Java programs?

Sorry for the long list of questions, but this is the first time I've been asked to help with this, so I want to have a solid understanding of the pros and cons.

-JD.

[Stephen McNulty]

Hi JD

We looked a this a while ago, (for individual doc loads),  which then turned itno, lets use it for bulk loads, in the end decided just not worth the bother.  Rather than have scarce java developers put things together with ODWEK, it was easier to convince them to create a file landing area, and let an OnDemand monitor job pick up the files.

[Justin Derrick]

Yeah, this particular CMOD user has an elaborate "packaging" system that has lines of business create .tar.gz files with an 'index' of files inside the package to ensure that everything that was produced actually arrived, and produces files to notify the business of a successful load.

It's being re-architected, and the Java programmers are saying it's easier to make a few library calls than to try and control an external process and parse the results.

Since I've never seen it before...  it's cause for concern...  but it's an interesting experiment! 

-JD.

[Lars Bencze]

Hey again,
Of course you can look inside the 87's too.
In this environment, we did indeed only load one doc at a time. But loading bigger batches is simply a matter of "looping" the LoadAddDoc step, once per document, until you finally run LoadCommit only once.

That also answers your third question - to the best of my knowledge, the only way to enter index data is by using the LoadAddDoc step. That means you probably have to parse index data in your intended input files in order to index the documents properly. I currently do not know of any way to index the documents locally on the OD Server using ACIF or PDF indexer.

As per your question whether I expect a performance penalty: it may actually be the other way around. Sure, there is a penalty for transferring the file to the OD Server via ODWEK, but it is probably not a lot greater than transferring the file any other way. And when it comes to the indexing - there is none! Since you have already entered the index data programmatically during the LoadAddDoc step, on the ODWEK server. When I look in the System Load table, the column "Index elapsed time (seconds)" is always zero for these documents. So, you may actually have a performance bonus instead of a penalty - but I suggest you test this properly in the actual environment you will be using. (For example if you run ODWEK on the same server as your Load/Library server, you will probably not decrease the OD Server load very much, if at all.)
 

2018-03-14 12:12:20.325756: ARS4315I Processing file >/opt/ondemand/arscache/tmp/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111220153101.139912273372928.ARD<
2018-03-14 12:12:20.325860: ARS4334I Load Version <9.5.0.9>  Operating System <Linux> <#1 SMP Fri Sep 22 12:32:14 EDT 2017.2.6.32-696.13.2.el6.x86_64>  OS Userid <ODADMIN>  Install Location </home/odadmin/ibm/ondemand/V
2018-03-14 12:12:20.325890: ARS4335I Server Version <9.5.0.9>  Operating System <Linux> <#1 SMP Fri Sep 22 12:32:14 EDT 2017.2.6.32-696.13.2.el6.x86_64>  Database <DB2> <10.01.0006>
2018-03-14 12:12:20.345012: ARS4339I Application Group >BANKDOC-NY<
2018-03-14 12:12:20.345039: ARS4340I Application >BANKDOC4PDF<
2018-03-14 12:12:20.345048: ARS4341I Storage Set >Cache Only - Library Server<
2018-03-14 12:12:20.345056: ARS4342I Storage Node >Cache Only - Library Server<
2018-03-14 12:12:20.345098: ARS4312I Loading started, 3146122 bytes to process
2018-03-14 12:12:20.399537: ARS1144I OnDemand Load Id = >5083-1-0-1198FAA-20180202000000-20180202000000-5089<
2018-03-14 12:12:20.416971: ARS1146I Loaded 1 rows into the database
2018-03-14 12:12:20.424766: ARS1175I Document compression type used - OD77.  Bytes Stored = >3268< Rows = >1<
2018-03-14 12:12:20.424830: ARS4310I Loading completed
2018-03-14 12:12:20.427613: ARS4317I Processing successful for file >/opt/ondemand/arscache/tmp/BANKDOC-NY.BANKDOC4PDF.ODADMIN.20180314.111220153101.139912273372928.ARD<