OnDemand User Group

Support Forums => Documentation => Topic started by: zancanaro on October 09, 2014, 02:18:52 AM

Title: Loadids - Maximal number
Post by: zancanaro on October 09, 2014, 02:18:52 AM

Hi all,

Our CMOD is v.9 based on AIX, with a DB2 version 9 ( and not 10 ... )

We load, for a specific APP GR, all day long, every 2 minutes, little documents.
Then, the loadid number increases a lot.
Today, in the syslog, we are at :
Name(APPlicationGRoup) LoadId(1234-4-0-236199FAA-nnnnnnnnnnnnn-nnnnnnnnnnnnn-6934) File(...................

What is the last number available for nnnnFAA ?

Thanks a lot.
Stephane.

Title: Re: Loadids - Maximal number
Post by: Frederick Tybalt on October 09, 2014, 03:16:02 AM

Hope this helps

http://www-01.ibm.com/support/docview.wss?uid=swg21383154

Title: Re: Loadids - Maximal number
Post by: Justin Derrick on October 09, 2014, 04:15:06 AM

Hi Stephane.

In situations like yours, I recommend that customers have TWO Application Groups -- one that holds the most recent 30 days worth ("new"), and another, which holds the rest ("old"). Then, every 30 days, the data should be extracted as one big object from the 'new' AG, and loaded into the 'old' AG. After that, run expiration on the 'new' AG, and wipe out all the data you reloaded.

This has a few advantages:

It will improve compression, sometimes dramatically.
It will reduce overhead, with fewer records in the ars* tables for each month, keeping your database small, light, and tidy. (Replace 21,000+ Load ID's with just ONE.)
It will make expiration of the 'old' AG run much more quickly.
It will reduce the size of your TSM database (if you're using TSM to store these documents.)

Hope this helps!

-JD.

Title: Re: Loadids - Maximal number
Post by: zancanaro on October 20, 2014, 08:29:10 AM

Hi Justin,
It's exactly our environment.
Each document ( .pdf + .ind ) is duplicated when arrived :

The copies ( .pdf & .ind ) are loaded on a save FileSystem,
The originals are then loaded to a First CMOD APP GR for limited time of life ( 24 hours ), and are seen by customers immediatly ( 2 minutes )

Then at 12 pm, all the copies (15000 - 25000 docs by day) are loaded in two files - one for all the .pdf, another for all requested .ind,
Then all the docs in the First APP GR for limited time of life are deleted,
The two totals files are then loaded in the 'regular' APP GR.
By the way # of loadids for First CMOD APP GR increments a lot ( 15000 - 25000 by day )
Thaty's the problem.

Thanks Justin.

Title: Re: Loadids - Maximal number
Post by: zancanaro on October 20, 2014, 08:32:08 AM

Hi Frederick,

Thanks too !!

Title: Re: Loadids - Maximal number
Post by: Maciej Mieczakowski on November 05, 2015, 01:45:54 AM

I'm just curious, what will happen when you really load those 14 billion of rows into single APPGR? Will the new coming data start failing to load and you will see that explanation in system log, or will you face any unexpected behavior? Did any of you faced that? Any experience?

I do not check all the apps and the object counter, so I'm curious if I should start worrying about that and maybe prepare some preventive action for those with high load ratio ;)

Title: Re: Loadids - Maximal number
Post by: Alessandro Perucchi on November 05, 2015, 10:26:14 AM

Hello Maciej,

I've played around with my CMOD sandbox, and I tried to see what happens when you load documents past the counter limit, and here is what happens.

This is the last "ARSLOAD" that works "5288-3-0- 999998ZZZ-20151105000000-20151105000000-12890"

Quote

$ arsload -h archive -nvf -g b -a b b

2015-11-05 18:13:47.755259: ARS4315I Processing file >b<
2015-11-05 18:13:47.755371: ARS4334I Load Version <9.5.0.2> Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64> OS Userid <ADMIN> Install Location </home/archive/ibm/ondemand/V9.5/> Data(unlimited KB) Stack(8192 KB) Core(0 512-blocks) Cpu(unlimited seconds) File(unlimited 512-blocks) Nofiles(2048) Threads(0) Processes(4096)
2015-11-05 18:13:47.755452: ARS4335I Server Version <9.5.0.2> Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64> Database <DB2> <09.07.0010>
2015-11-05 18:13:47.778992: ARS4339I Application Group >b<
2015-11-05 18:13:47.779046: ARS4340I Application >b<
2015-11-05 18:13:47.779074: ARS4341I Storage Set >cmodduc<
2015-11-05 18:13:47.779099: ARS4342I Storage Node >cmodduc<
2015-11-05 18:13:47.779137: ARS4312I Loading started, --UNKNOWN-- bytes to process
2015-11-05 18:13:47.791999: ARS1144I OnDemand Load Id = >5288-3-0-999998ZZZ-20151105000000-20151105000000-12890<
2015-11-05 18:13:47.821300: ARS1146I Loaded 1 rows into the database
2015-11-05 18:13:47.836516: ARS1175I Document compression type used - OD77. Bytes Stored = >646< Rows = >1<
2015-11-05 18:13:47.836591: ARS4310I Loading completed
2015-11-05 18:13:47.842930: ARS4317I Processing successful for file >b<

And now, that's the moment where I should receive "5288-3-0- 999999ZZZ-20151105000000-20151105000000-12890"

Quote

$ arsload -h archive -nvf -g b -a b b

2015-11-05 18:14:15.722459: ARS4315I Processing file >b<
2015-11-05 18:14:15.722534: ARS4334I Load Version <9.5.0.2> Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64> OS Userid <ADMIN> Install Location </home/archive/ibm/ondemand/V9.5/> Data(unlimited KB) Stack(8192 KB) Core(0 512-blocks) Cpu(unlimited seconds) File(unlimited 512-blocks) Nofiles(2048) Threads(0) Processes(4096)
2015-11-05 18:14:15.722568: ARS4335I Server Version <9.5.0.2> Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64> Database <DB2> <09.07.0010>
2015-11-05 18:14:15.730772: ARS4339I Application Group >b<
2015-11-05 18:14:15.730818: ARS4340I Application >b<
2015-11-05 18:14:15.730844: ARS4341I Storage Set >cmodduc<
2015-11-05 18:14:15.730868: ARS4342I Storage Node >cmodduc<
2015-11-05 18:14:15.730916: ARS4312I Loading started, --UNKNOWN-- bytes to process
2015-11-05 18:14:15.763605: ARS1107E An error occurred. Contact your system administrator and/or consult the System Log. File=arsload.c, Line=10261
2015-11-05 18:14:15.763655: ARS1161E Unable to get LoadId
2015-11-05 18:14:15.763686: ARS1146I Loaded 0 rows into the database
2015-11-05 18:14:15.763732: ARS4311E Loading failed
2015-11-05 18:14:15.763763: ARS4318E Processing failed for file >b<
2015-11-05 18:14:15.792713: ARS4327E Processing has stopped. The remaining files will NOT be processed.

As you can see, when you've exhausted the last possible LOAD ID, you get the error ARS1161E Unable to get LoadId

So you could trap this from the output of arsload, and act accordingly ;D or better check for both of these values in ARSAG table:

Code: [Select]

$ db2 "select name,load_id from arsag where load_id_suffix='ZZZ'"

NAME                                                         LOAD_ID    
------------------------------------------------------------ -----------
b                                                                 999999

  1 record(s) selected.

--

Alessandro

Title: Re: Loadids - Maximal number
Post by: Maciej Mieczakowski on November 05, 2015, 10:56:25 PM

Great answer! :) Many thanks Alessandro. I don't have to bother about that till I really face with this ;)