Author Topic: Loadids - Maximal number  (Read 3164 times)

zancanaro

  • Guest
Loadids - Maximal number
« on: October 09, 2014, 02:18:52 AM »
Hi all,

Our CMOD is v.9 based on AIX, with a DB2 version 9 ( and not 10 ... )

We load, for a specific APP GR, all day long, every 2 minutes, little documents.
Then, the loadid number increases a lot.
Today, in the syslog, we are at :
Name(APPlicationGRoup) LoadId(1234-4-0-236199FAA-nnnnnnnnnnnnn-nnnnnnnnnnnnn-6934) File(...................

What is the last number available for nnnnFAA ?

Thanks a lot.
Stephane.

Frederick Tybalt

  • Full Member
  • ***
  • Posts: 124
    • View Profile
    • Personal Website
Re: Loadids - Maximal number
« Reply #1 on: October 09, 2014, 03:16:02 AM »
rIcK
======------------------======
www.rick.co.in | www.tekbytz.com

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 1681
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: Loadids - Maximal number
« Reply #2 on: October 09, 2014, 04:15:06 AM »
Hi Stephane.

In situations like yours, I recommend that customers have TWO Application Groups -- one that holds the most recent 30 days worth ("new"), and another, which holds the rest ("old").  Then, every 30 days, the data should be extracted as one big object from the 'new' AG, and loaded into the 'old' AG.  After that, run expiration on the 'new' AG, and wipe out all the data you reloaded.

This has a few advantages:

  • It will improve compression, sometimes dramatically.
  • It will reduce overhead, with fewer records in the ars* tables for each month, keeping your database small, light, and tidy.  (Replace 21,000+ Load ID's with just ONE.)
  • It will make expiration of the 'old' AG run much more quickly.
  • It will reduce the size of your TSM database (if you're using TSM to store these documents.)

Hope this helps!

-JD.
#Install, #Educate, #Repair, #Upgrade, #Migrate, #Enhance, #Optimize.

IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SA #Performance #Security #Audits #Customizing #Availability #HA #DR #Training

zancanaro

  • Guest
Re: Loadids - Maximal number
« Reply #3 on: October 20, 2014, 08:29:10 AM »
Hi Justin,
It's exactly our environment.
Each document ( .pdf + .ind ) is duplicated when arrived :

The copies ( .pdf & .ind ) are loaded on a save FileSystem,
The originals are then loaded to a First CMOD APP GR for limited time of life ( 24 hours ), and are seen by customers immediatly ( 2 minutes )

Then at 12 pm, all the copies (15000 - 25000 docs by day) are loaded in two files - one for all the .pdf,  another for all requested .ind,
Then all the docs in the First APP GR for limited time of life are deleted,
The two totals files are then loaded in the 'regular'  APP GR.
By the way # of loadids for  First CMOD APP GR  increments a lot ( 15000 - 25000 by day )
Thaty's the problem.

Thanks Justin.

zancanaro

  • Guest
Re: Loadids - Maximal number
« Reply #4 on: October 20, 2014, 08:32:08 AM »
Hi Frederick,

Thanks too !!

Maciej Mieczakowski

  • Jr. Member
  • **
  • Posts: 33
    • View Profile
Re: Loadids - Maximal number
« Reply #5 on: November 05, 2015, 01:45:54 AM »
I'm just curious, what will happen when you really load those 14 billion of rows into single APPGR? Will the new coming data start failing to load and you will see that explanation in system log, or will you face any unexpected behavior? Did any of you faced that? Any experience?

I do not check all the apps and the object counter, so I'm curious if I should start worrying about that and maybe prepare some preventive action for those with high load ratio ;)

Alessandro Perucchi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1001
    • View Profile
Re: Loadids - Maximal number
« Reply #6 on: November 05, 2015, 10:26:14 AM »
Hello Maciej,

I've played around with my CMOD sandbox, and I tried to see what happens when you load documents past the counter limit, and here is what happens.

This is the last "ARSLOAD" that works "5288-3-0- 999998ZZZ-20151105000000-20151105000000-12890"

Quote
$ arsload -h archive -nvf -g b -a b b

2015-11-05 18:13:47.755259: ARS4315I Processing file >b<
2015-11-05 18:13:47.755371: ARS4334I Load Version <9.5.0.2>  Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64>  OS Userid <ADMIN>  Install Location </home/archive/ibm/ondemand/V9.5/> Data(unlimited KB) Stack(8192 KB) Core(0 512-blocks) Cpu(unlimited seconds) File(unlimited 512-blocks) Nofiles(2048) Threads(0) Processes(4096)
2015-11-05 18:13:47.755452: ARS4335I Server Version <9.5.0.2>  Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64>  Database <DB2> <09.07.0010>
2015-11-05 18:13:47.778992: ARS4339I Application Group >b<
2015-11-05 18:13:47.779046: ARS4340I Application >b<
2015-11-05 18:13:47.779074: ARS4341I Storage Set >cmodduc<
2015-11-05 18:13:47.779099: ARS4342I Storage Node >cmodduc<
2015-11-05 18:13:47.779137: ARS4312I Loading started, --UNKNOWN-- bytes to process
2015-11-05 18:13:47.791999: ARS1144I OnDemand Load Id = >5288-3-0-999998ZZZ-20151105000000-20151105000000-12890<
2015-11-05 18:13:47.821300: ARS1146I Loaded 1 rows into the database
2015-11-05 18:13:47.836516: ARS1175I Document compression type used - OD77.  Bytes Stored = >646< Rows = >1<
2015-11-05 18:13:47.836591: ARS4310I Loading completed
2015-11-05 18:13:47.842930: ARS4317I Processing successful for file >b<

And now, that's the moment where I should receive "5288-3-0- 999999ZZZ-20151105000000-20151105000000-12890"

Quote
$ arsload -h archive -nvf -g b -a b b

2015-11-05 18:14:15.722459: ARS4315I Processing file >b<
2015-11-05 18:14:15.722534: ARS4334I Load Version <9.5.0.2>  Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64>  OS Userid <ADMIN>  Install Location </home/archive/ibm/ondemand/V9.5/> Data(unlimited KB) Stack(8192 KB) Core(0 512-blocks) Cpu(unlimited seconds) File(unlimited 512-blocks) Nofiles(2048) Threads(0) Processes(4096)
2015-11-05 18:14:15.722568: ARS4335I Server Version <9.5.0.2>  Operating System <Linux> <#1 SMP Thu Oct 8 03:23:55 UTC 2015.4.2.3-200.fc22.x86_64>  Database <DB2> <09.07.0010>
2015-11-05 18:14:15.730772: ARS4339I Application Group >b<
2015-11-05 18:14:15.730818: ARS4340I Application >b<
2015-11-05 18:14:15.730844: ARS4341I Storage Set >cmodduc<
2015-11-05 18:14:15.730868: ARS4342I Storage Node >cmodduc<
2015-11-05 18:14:15.730916: ARS4312I Loading started, --UNKNOWN-- bytes to process
2015-11-05 18:14:15.763605: ARS1107E An error occurred.  Contact your system administrator and/or consult the System Log.  File=arsload.c, Line=10261
2015-11-05 18:14:15.763655: ARS1161E Unable to get LoadId

2015-11-05 18:14:15.763686: ARS1146I Loaded 0 rows into the database
2015-11-05 18:14:15.763732: ARS4311E Loading failed
2015-11-05 18:14:15.763763: ARS4318E Processing failed for file >b<
2015-11-05 18:14:15.792713: ARS4327E Processing has stopped.  The remaining files will NOT be processed.

As you can see, when you've exhausted the last possible LOAD ID, you get the error ARS1161E Unable to get LoadId

So you could trap this from the output of arsload, and act accordingly  ;D or better check for both of these values in ARSAG table:

Code: [Select]
$ db2 "select name,load_id from arsag where load_id_suffix='ZZZ'"

NAME                                                         LOAD_ID   
------------------------------------------------------------ -----------
b                                                                 999999

  1 record(s) selected.

--

Alessandro
« Last Edit: November 05, 2015, 10:30:43 AM by Alessandro Perucchi »
Alessandro Perucchi

#Install #Migrations #Conversion #Educate #Repair #Upgrade #Migrate #Enhance #Optimize #AIX #Linux #Multiplatforms #DB2 #Windows #Oracle #TSM #Tivoli #Performance #Audits #Customizing #Availability #HA #DR #JavaApi #ContentNavigator #ICN #WEBi #ODWEK #Services #PDF #AFP #XML

Maciej Mieczakowski

  • Jr. Member
  • **
  • Posts: 33
    • View Profile
Re: Loadids - Maximal number
« Reply #7 on: November 05, 2015, 10:56:25 PM »
Great answer! :) Many thanks Alessandro. I don't have to bother about that till I really face with this ;)