Author Topic: Upgrade and my 'personal' thoughts  (Read 3136 times)

hakan_carlberg

  • Jr. Member
  • **
  • Posts: 83
    • View Profile
Upgrade and my 'personal' thoughts
« on: March 12, 2019, 07:11:11 AM »
Hi

I'm about to run into one of my most hated issues at work,
upgrading OnDemand. This time from 9.5 -> 10.1

If you don't like to read things from a "Grumpy old man", then
stop reading now!!!!

I don't really understand how it can be so hard for IBM to make
proper instructions how to do an 'downgrade' for CMOD, if you
managed to do an upgrade and realized that you're into big trouble
and need to do an 'downgrade'.

I did open an PMR and asked the question...
And not suprisingly!!?? the answers were:

"
..
As I mentioned, I did a quick compare and saw 3 tables that had more   
columns in 10.1 than 9.5, ARSHOLDMAP, ARSRES, and ARSSYS but again, that
is not an official statement and there may be other changes that I am   
not aware of
..
"

and

"
..
I appreciate that you don't think that answer is acceptable, but it is 
and has always been the answer for this release and all others prior.   
Development has never documented what changes the arsdb command will   
introduce, it is always recommneded to take backups of all the OnDemand
system tables (ARS*) and have those available if a rollback is required.
..
"

I don't understand how it can be so easily written by IBM, in the README-file
"
...
Customers will need to have a database backup and recovery plan implemented prior to
performing this procedure.
..
"

And that we accept ??

How hard can it be for IBM to make a script, so that we can do an 'downgrade'!

As I see it right now, you can do an upgrade.. but if anything happen, and you
need to do an 'downgrade. God forbid if you've loaded anything !!, then you're
out of synch !!
It might even be so that directly after you've upgraded adn starts the ARSSOCKD,
The System Log switch and messages are written to a new table...
whar will happen if you reload ARSSYS/ARSAG ??

Anybody have any thoughts ?
/Hakan Carlberg

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2229
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: Upgrade and my 'personal' thoughts
« Reply #1 on: March 12, 2019, 07:34:15 AM »
I've always just backed up the ars* tables with arsdb -xlvf and performed the upgrade.  I've also never had to do a rollback from a major upgrade -- I still think that's more luck as much as it is skills.  :D

-JD.
IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

Alessandro Perucchi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1002
    • View Profile
Re: Upgrade and my 'personal' thoughts
« Reply #2 on: March 12, 2019, 01:27:34 PM »
Well, the recovery plan is ALWAYS do a backup, and if doesn't work, then restore it. That is not only with CMOD, but with any product you can find.
And a downgrade is not always possible, even with the OnDemand, which is the most perfect product in the whole universe!

As Justin said, you can do a backup / restore easily of the CMOD system tables with the command arsdb. You don't need a full backup of all the tables and segments, only the ARS% tables.

Alessandro Perucchi

#Install #Migrations #Conversion #Educate #Repair #Upgrade #Migrate #Enhance #Optimize #AIX #Linux #Multiplatforms #DB2 #Windows #Oracle #TSM #Tivoli #Performance #Audits #Customizing #Availability #HA #DR #JavaApi #ContentNavigator #ICN #WEBi #ODWEK #Services #PDF #AFP #XML

hakan_carlberg

  • Jr. Member
  • **
  • Posts: 83
    • View Profile
Re: Upgrade and my 'personal' thoughts
« Reply #3 on: March 12, 2019, 02:24:24 PM »
Hi Alessandro
I think you are missing the point
Why should you restore old Ars%-tables? If you cant keep them in synch with your other tables of the loaded data in all your appl.groups?

Who would like to be in a situation where the data you have is not correct?
You could find yourself in a situation where you have data in OAM, indextables where that data is not in the ARS-tables.

Alessandro Perucchi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1002
    • View Profile
Re: Upgrade and my 'personal' thoughts
« Reply #4 on: March 12, 2019, 03:05:40 PM »
Hi Alessandro
I think you are missing the point
Why should you restore old Ars%-tables? If you cant keep them in synch with your other tables of the loaded data in all your appl.groups?

Who would like to be in a situation where the data you have is not correct?
You could find yourself in a situation where you have data in OAM, indextables where that data is not in the ARS-tables.


I mean...
1) You stop CMOD
2) You do the system table backup + Full Backup of the segment tables
3) You do the upgrade
4) You start CMOD
5) You test if it works
6) It works, then you open for production. Finished.
7) It doesn't work, then stop CMOD
8 ) restore the system tables and possibly the impacted segment tables
9) be sure to use the old SW
10) start CMOD with the old binary version

If you go that way, you won't have any problems. I've done it countless of time with customer where we had some issues.
And depending on the tests, you might need also to restore some segment tables, because of the archived documents that was used during the tests of functionality.

In that case I would advise to do test in a dummy AG, which is not important if the restore makes it not in sync with the restored info from the system tables.

Of course, if after X days you see that it doesn't work... then yeah... both the segment and system table are not in sync anymore and you have a problem. And I agree with you, this is not something that one wants. Not at all.
Downgrading is not a light operation, and during the changes of version with CMOD, there were lot of time that the changes were so profound that a downgrade was simply impossible.
And a restore was the only way. Today the changes are not anymore that deep as in the V7.X and V8.X versions, but still sometimes they are still some modification here and there.

And to be honnest, I know very little product in the market that allow a downgrade without doing some kind of restore.
So I understand your personal thoughts... but I wonder what else could IBM do in that area... because some upgrades are not reversible.

Maybe doing more tests in your test system, to ensure that your prod will be not impacted with some issues?
But that's not IBM work, it is more on your side.

For my part, I am more in the side to do backup and restore. And if after a few days something is really wrong, since I tried to keep all the input data that is archived in CMOD for at least 1 week. If I need to restore something, then I can restore the DB, and reload everything between the moment of the last backup.
Call me paranoid... but I am never expecting a magical solution from vendors in a timely manner. IBM or not IBM.
If I can wait, then ok, but if that is not a possibility, then I have my safety bag.
And of course, then there is a SLA issue, but better that, than keeping with a unsolvable situation that is going worse and worse with time.

What I did many times, was to clone the production database in a test environment, remove the Storage Set/Node dependencies by setting in the local cache, or a test storage which is like Prod. And do my tests there with all the documents type that we load. And since we keep at least 1 week of input files from production, we can test all the possible scenarios.
Especially if we are not sure of the results of the upgrade.
That way we learn if everything is as we expect, and if not, we don't do the upgrade in prod until we have a fix from IBM or internal teams for some custom code.
« Last Edit: March 12, 2019, 03:25:36 PM by Alessandro Perucchi »
Alessandro Perucchi

#Install #Migrations #Conversion #Educate #Repair #Upgrade #Migrate #Enhance #Optimize #AIX #Linux #Multiplatforms #DB2 #Windows #Oracle #TSM #Tivoli #Performance #Audits #Customizing #Availability #HA #DR #JavaApi #ContentNavigator #ICN #WEBi #ODWEK #Services #PDF #AFP #XML

Justin Derrick

  • IBM Content Manager OnDemand Consultant
  • Administrator
  • Hero Member
  • *****
  • Posts: 2229
  • CMOD Guru for hire...
    • View Profile
    • Tenacious Consulting
Re: Upgrade and my 'personal' thoughts
« Reply #5 on: March 12, 2019, 06:53:23 PM »
I think the difference here is that Hakan wants to upgrade, run for several days, then have the ability to downgrade if there's an issue with the new version of the code, but without losing any of the loaded data.

As much as I like and respect Hakan, I'm not sure many others have the same 'use case'.  Sorry!  :)

-JD.


IBM CMOD Professional Services: http://TenaciousConsulting.com
Call:  +1-866-533-7742  or  eMail:  jd@justinderrick.com
IBM CMOD Wiki:  https://CMOD.wiki/
FREE IBM CMOD Education & Webinars:  https://CMOD.Training/

Interests: #AIX #Linux #Multiplatforms #DB2 #TSM #SP #Performance #Security #Audits #Customizing #Availability #HA #DR

hakan_carlberg

  • Jr. Member
  • **
  • Posts: 83
    • View Profile
Re: Upgrade and my 'personal' thoughts
« Reply #6 on: March 13, 2019, 09:05:28 AM »
Hi
Don't missunderstand me at all, as I've said a couple of times at ODUG-meetings:
- I do think that CMOD is doing what it's supposed to do
- I kind of like it, it's more or less self-running.

But..

I've upgraded CMOD a couple of times, and it really is a pain that there's no easy way of 'downgrading'.
regarding..
"
..I know very little product in the market that allow a downgrade without doing some kind of restore..
"
Well I can name a bunch of them.
Like DB2 on z/OS, where you can run 2 version on different members in a
datasharing group, and if you've upgraded to let's say V12 you can then goback to V11.
Until you've been running for a couple of weeks and decide to go to 'new function mode'.

Of course we will do extending tests in our different environment before going into production, but you can't
test everything.

I don't really see the point of restoring some segment tables, because if you've loaded a Bank-statement, and the
Customer has retrieved it, then you just can't remove it by using restore.
We don't keep 1 week of input files saved, they can be deleted/overwritten after the've been succesfully loaded.
(If it's loaded then it's stored in CMOD, why keep backups of them ??).
We also load from other environment's(except z/OS) like windows-clients, and we don't have any control of those.

"..What I did many times, was to clone the production database in a test environment.."
You're running on a distributed environment I presume ?
It's not that easy on z/OS.
And if you don't run with the Storage Set/Node dependecies, then it's not like your production environment, and hence the
test are not complete.


@Justin: I would say that if we ran into some problem after a couple of days, then I would say 'go forward'.
I'm more into be able, in an easy way, of being able to downgrade after a couple of hours in an easy way.
But thank for your 'respect' :-D

/Hakan

Ed_Arnold

  • Hero Member
  • *****
  • Posts: 1199
    • View Profile
Re: Upgrade and my 'personal' thoughts
« Reply #7 on: February 18, 2020, 02:41:49 PM »
I just re-read this topic.

Good news!

Backing out from 10.1 -> 9.5 is now supported.

Specifically Version 9.5 of CMOD can run with V10.1 table definitions.

So --- say you do the arsdb -vu upgrade step.

You bring up CMOD V10.1.

You run into a problem you can't immediately resolve.

Do not back out the table change.

Just switch back to V9.5 SARSLOAD and HFS.

Yes, the arsdb -vu is a one-way trip.

But now using the executables is not.

This is supported.

It has been tested and no issues have been reported.

Ed
#zOS #ODF