Author Topic: ARS1400E  (Read 5146 times)

rbarrenechea

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
ARS1400E
« on: January 18, 2016, 02:54:48 PM »
Hi,

Getting this lovely message:

ARS1400E Loading was unable to continue due to encountering a previously failed load of the same name that has yet to be unloaded.  Failed LoadId >7975-3-0-5840FAA-20160101000000-20160101000000-7976< was attempted to be loaded at approximately 2016-01-15 17:05:51

I know the not so simple option of arsadmin unload -Q -L 7975-3-0-5840FAA-20160101000000-20160101000000-7976

And I say not simple as one file, OK but hundreds not so good.

Any ides/suggestions?

Thank you

<<< The light at the end of a tunnel is a TRAIN!!! >>>

Alessandro Perucchi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1002
    • View Profile
Re: ARS1400E
« Reply #1 on: January 19, 2016, 08:43:32 AM »
Hi,

Would it be possible to know the exact steps that you are using in order to get this error message?
I have never seen such errors, and I am really interested how you could get it.

I don't want the standard explanation of, I did an arsload, and I got this error message.

I would like to know exactly what happens to CMOD before to run this arsload. Did some failed arsload happened? Did everything went correctly?
Are you using CMOD on Linux / z/OS / Windows / Solaris / ... ???
Did someone touch the databases?
Did some partial restore happens in the database?
... well anything that is not the "standard" handling of CMOD.

I am asking that, because in nearly 17 years of CMOD, the errors which are really strange were when something strange happens to the database due to corruption in the database... either from disk corruptions, wrongly made restore, change in the database values, migrations, etc...

That's why I am asking a question in that direction in order to try to help you.
Alessandro Perucchi

#Install #Migrations #Conversion #Educate #Repair #Upgrade #Migrate #Enhance #Optimize #AIX #Linux #Multiplatforms #DB2 #Windows #Oracle #TSM #Tivoli #Performance #Audits #Customizing #Availability #HA #DR #JavaApi #ContentNavigator #ICN #WEBi #ODWEK #Services #PDF #AFP #XML

rbarrenechea

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
Re: ARS1400E
« Reply #2 on: January 28, 2016, 11:34:25 AM »
Hi Alessandro.

For the record I did manage to solve this by preventing it to happen again.

And here is the details:

CMOD 9.5.0.0 Windows 2012 R2 - 64 Bit.
TSM 7.1.1 Windows 2012 R2 - 64 Bit
The trick is on TSM, as the old system did use 1 mount point for each node, and only had one ARSLOAD process, then you never got this, but

On the New CMOD server, lots of power, multiple ARSLOAD, then the node failed to load the file:
1) CMOD process the index.
2) CMOD loads the Metadata to DB
3) CMOD stores the file to CACHE
4) CMOD fails to store the file in TSM because there aren't enough mount points, so I increased the mount points from 1 to 10 for all the nodes, and problem sorted, no more error on the loads.

It seems like CMOD 9.5 unlike the previous versions does consider the partial load to cache as successful. As in the old versions it will fail completely if one part of the cycle (like the Store to TSM) failed, here it does consider it valid, but the source file (.ARD) is changed to .ARD.failed), and when you retry  it comes with that error.

I hope the above is what you wanted to know.

Roberto.

Alessandro Perucchi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1002
    • View Profile
Re: ARS1400E
« Reply #3 on: January 28, 2016, 03:39:36 PM »
Roberto,

why do you still use a base version without any fix pack installed? There are a LOT of problems with this version (and also with Fix Pack 1 of V9.5), like the delete of documents are not in TSM deleted...
And with my experience with CMOD... never put in production a version .0, wait the first or second fix pack for it... This advice is also valid for any other product... going with the first big version is always a bad idea, except for developing and testing... but not for production.

And in addition to that, the minimum mount point (or sessions) for TSM for CMOD is 255 (and this is the minimum recommended value). If go lower than 255, then you can have such problems, especially with lots of load.
This is in the documentation when you follow the installation by the book (http://www-01.ibm.com/support/knowledgecenter/SSEPCD_8.5.0/com.ibm.ondemand.installingmp.doc/ars1i071458.htm%23wq539)

So going to 10, this is not a good idea at all, please use the minimum best practice value for TSM max session, which is at least 255.


To resume...

Try to install the latest fix pack V9.5.0.4, and retry your arsload with max session=1, to see if you get this "partial load" is still "accepted".
Do you use "-f" with arsload? In order to do a complete rollback in case it fails?
If even with that you get this kind of errors again, then please open a PMR, because this should not be the case...

I know that with V9.5, the load is done in a asynchronous way... but I have yet not experience this kind of error you describe... and this is for me quite critical...
Alessandro Perucchi

#Install #Migrations #Conversion #Educate #Repair #Upgrade #Migrate #Enhance #Optimize #AIX #Linux #Multiplatforms #DB2 #Windows #Oracle #TSM #Tivoli #Performance #Audits #Customizing #Availability #HA #DR #JavaApi #ContentNavigator #ICN #WEBi #ODWEK #Services #PDF #AFP #XML

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2228
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: ARS1400E
« Reply #4 on: January 29, 2016, 03:46:12 AM »

And in addition to that, the minimum mount point (or sessions) for TSM for CMOD is 255 (and this is the minimum recommended value). If go lower than 255, then you can have such problems, especially with lots of load.


This is only true if your TSM system only has disks.  You can't mount more tapes than you have drives.  ;)

Otherwise, Alessandro is right -- never use a .0 release in production!  Find the bugs with your QA system, report them to IBM, and wait for a patch or two.

-JD.
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

Alessandro Perucchi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1002
    • View Profile
Re: ARS1400E
« Reply #5 on: January 29, 2016, 04:29:35 AM »

And in addition to that, the minimum mount point (or sessions) for TSM for CMOD is 255 (and this is the minimum recommended value). If go lower than 255, then you can have such problems, especially with lots of load.

This is only true if your TSM system only has disks.  You can't mount more tapes than you have drives.  ;)

:-) You are right :-) I was a bit quick in my answer here!!
Alessandro Perucchi

#Install #Migrations #Conversion #Educate #Repair #Upgrade #Migrate #Enhance #Optimize #AIX #Linux #Multiplatforms #DB2 #Windows #Oracle #TSM #Tivoli #Performance #Audits #Customizing #Availability #HA #DR #JavaApi #ContentNavigator #ICN #WEBi #ODWEK #Services #PDF #AFP #XML

rbarrenechea

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
Re: ARS1400E
« Reply #6 on: February 29, 2016, 10:28:40 AM »
Hi, Update in here:

In my experience with CMOD (and also with CM) you never go and install the latest patch, just because. To explain in this case:

The error is present in FP1, FP2, FP3 and FP4, how I know, well I tried them all and if you use TSM and install FP4 you will damage your system:

IBM Flash on this:

                                                                     
 - TITLE: Customers using Content Manager OnDemand with TSM should not   
 upgrade to version 9.5.0.4 until it is fixed and reposted               
 - URL:                                                                 
                                                                         
 http://www-01.ibm.com/support/docview.wss?uid=swg21976832&myns=swgimgmt&
 mynp=OCSSEPCD&mync=E&cm_sp=swgimgmt-_-OCSSEPCD-_-E                     
                                                                         
 - ABSTRACT: There is an issue in Content Manager OnDemand (CMOD) version
 9.5.0.4 which can prevent access to historical data stored in TSM and   
 cause any newly loaded data to be orphaned once a fix is available. The
 only customers potentially affected are using TSM and have a CMOD       
 instance name of ARCHIVE.                                               

IBM regrettably does not have a great record of perfect patches, I learned this in CM, when they released one that did corrupt your DB (accidentally off-course) and they fixed this with an A package, just like ICN 2.0.3.6a,

And that is part of the purpose of having to use Install Manager now, this allows you to deploy a patch, test it and roll it back without having to go back to the base level.

So yes, I do agree that base is not good, but sometimes the FP makes no difference or just give you more problems. So before advicing to install the latest one I will advice my clients to test first ;-)

However still no solution for the ARS1400E, well other than make sure that it doesn't fail to load ;-)

Alessandro Perucchi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1002
    • View Profile
Re: ARS1400E
« Reply #7 on: March 01, 2016, 10:59:14 AM »
Wll I don't know what happens to your customer... I am really sorry.
But concerning this patch 4, yes this was a huge mistake from IBM.

BUT I can ensure you 100% that NONE of my customer that had a setup with "ARCHIVE" as the default name (which is normally not a good practice to do such a thing), had a problem with 9.5.0.0, 9.5.0.1, 9.5.0.2 and 9.5.0.3.
Only with 9.5.0.4.

:-( but in your case, I would stick with V9 if you had so many problems with V9.5... and wait until V9.5 is ok for you. We had a few customer that tested 9.5 and it was not ok for them, and they stayed with V9, or migrated from V8.X to V9.

So your statement:

Quote
In my experience with CMOD (and also with CM) you never go and install the latest patch

Is not correct, because you can always go to the latest fix pack in TEST. And only after testing it... then ONLY then... you can go in production. And if the tests are not goof... then test with the previous fix pack, etc... etc...
and if in the end, this is still not ok, then stay with the previous version.

But going with a .0 version... this is just a NO WAY.... this is just wanting to wait for problems.... and I can assure you 100% that you will get problems.

If the customer has no test system... then they are just playing with fire...
Alessandro Perucchi

#Install #Migrations #Conversion #Educate #Repair #Upgrade #Migrate #Enhance #Optimize #AIX #Linux #Multiplatforms #DB2 #Windows #Oracle #TSM #Tivoli #Performance #Audits #Customizing #Availability #HA #DR #JavaApi #ContentNavigator #ICN #WEBi #ODWEK #Services #PDF #AFP #XML

Lars Bencze

  • Full Member
  • ***
  • Posts: 116
  • CMOD Expert at Skandia
    • View Profile
    • INACTIVE - Bezland Consulting
Re: ARS1400E
« Reply #8 on: August 23, 2018, 08:26:06 AM »
About the error itself. This seems to happen "always" when there is a failed load... at least on Windows-based systems, starting with V9.5.
Any time a load fails, it cannot be loaded again unless you do an Unload first. (OK, maybe not EVERY time....but it sure feels like that!)
So if 800 files fail, you have to do 800 Unloads first. That sux bigtime.
I have written two scripts for this, "BulkUnload.bat" (which accepts a file with a list of LoadIds to Unload) and "BulkUnfail.bat", which removes the ".Failed" extension from the ARD files, so that the Load Daemon can find and re-attempt loading of this again.

Here is a valid answer I found online, which explains why the problem occurs - and how to resolve it:

Answer by eandreasen (1) | Oct 05, 2016 at 02:15 AM

The reason for this error message is the new table arsloadwork which was created in V9.5 to prevent the same file from being loaded twice by inserting the AG, APP and the input file names into the table. It serves the intended purpose, except when the network is interrupted before arsload completes. In this case, a row is already written into arsloadwork, but the indexes have not be inserted into the AG data table. Before you attempt to reload the same file, you need to manually clean up the arsloadwork table, e.g,

delete from arsloadwork where agid=5328 and aid=5359 and "loadid ='5358-1-0-5650FAA-20160913033728-20160913033728-5359'"


Be careful with fiddling with the OnDemand DB tables though...  :-X :-O

There is a big problem with this annoying error message. Sometimes, there is no corresponding MsgNum 88 in the system log, actually no trace at all in the System Log of the LoadId that "...has yet to be unloaded..."! These cases can however be unloaded by using this workaround:

1. Find and copy the LoadId from the NEW MsgNum 88 which you inevitably received when you tried to reload the file by removing the .Failed extension:
"... ARS1400E ... has yet to be unloaded.  Failed LoadId >5389-11-0-24944FAA-20180823000000-20180823000000-5390< was attempted to be loaded at approximately 2018-08-..."
2. Run arsadmin unload:
"D:\Program Files\IBM\OnDemand\V9.5\bin\arsadmin.exe" unload -g <YourApplicationGroup> -u YOURLOADUSER -p "D:\scripts\YourScriptsStashFile.stash" -h ARCHIVE -N -L <LoadId_You_Copied_In_Item_1_Above>

OnDemand for MP expert. #Multiplatforms #Admin #Scripts #Performance #Support #Architecture #PDFIndexing #TSM/SP #DB2 #CustomSolutions #Integration #UserExits #Migrations #Workflow #ECM #Cloud #ODApi