How Do I Migrate Indices & Objects

[JJeffrey]

We have CMOD 7.1.2.9 on AIX.  Our flagship application was created (and indexed) by IBM consultants, with CACHE ONLY storage in production.  After 2 years of loading we now have 40 million documents in CACHE.  I want to change this application to use CACHE & TSM storage so that I can start maintenance up for this application.

How do I move/export/convert the 40 million documents and indices from CACHE only to CACHE & TSM?

[sisusteve]

Your IBM Consultants didn't do a very good if they knew that you were going to have this volume and set it up as a Cache Only system.  As far as I know, the only way you can change storage node information is to create new application groups that use TSM, extract existing documents based on LOADID, and then reload into new application group.

Other than that, you're going to have to call IBM for possible help

[Steve C]

I'm facing a similar challange on z/OS.

So far I'm thinking along the lines of running "arsmaint -m" for a given application group to 'copy' from cache storage to 'archive' storage.
From what I read, if you create a storage definition to use a storage manager such as TSM along with cache then OD will keep a copy
on disk as well as TSM getting a copy at initial load time. The cache can then be allowed to expire after so many days.

If you only have storge with cache only, then I believe (and this is what I'm shortly to test) you create a new node with with TSM (as well as with optional cache)  storage and then AppGrp by AppGrp run the "arsmain -m -g" to copy to your archive storage.

Lets see how this flies - will be doing it first on throw away test data till I'm happy.

[Justin Derrick]

Hi JJeffrey...

Once an Application Group is designated Cache Only, there's no quick and clever way to change the Storage Set to another type (there's good reason for this, which I won't get into now).

SteveC:

I know that things are vastly different on the Z/Series systems, but I don't think what you're suggesting will work either.  Without somewhere to PUT the new data (as defined by the storage set), the migration can't work.

Steve is correct -- that creating Application Groups as cache only *is* very short-sighted -- it's bitten more than a few of my customers over the years.  None of the options are particularly palatable, but here they are...

1)  Extract & Reload into a new Application Group.  It's time consuming, if nothing else, and you need a big pool of disk for it to work properly.

2)  Create a new AG with secondary storage defined, then migrate the old AG into a THIRD Application Group, and tie the two AG's together at the folder level.  This has performance implications.

3)  Keep adding disk.  Chart out your growth and see if this is doable.  Depending on what your secondary storage media is, this might even be economical.  Just make sure the disk is heavily protected with RAID disk and/or regular backups.

Don't hesitate to give me a call to talk about your specific situation.

[wwwalton]

I'm curious, one of the 'get off the Centera scenarios' is to run with cache only and not use TSM.   What would be the drawbacks in cache only, especially if replication is in place on the jfs?
Thanks,
-walt

[Justin Derrick]

Hi Walt.

The single major drawback to not using TSM is a complete loss of flexibility in storage management.

Without TSM, you can't:

• Migrate unused or infrequently accessed data to cheaper storage.
• Handle an influx of historical data from a merger or acquisition.
• Utilize tape or optical to store data that is literally never accessed.
• Provide for 'higher availability' than just using disk-based cache.

Here's two examples of the costs of not using TSM:

1)  A large financial services firm stored a particular type of extremely-high-volume documents for 90 days in cache.  When a customer sued them, the court mandated that NO DATA was to be deleted from the system for the duration of the case.  The volume of documents balooned, quickly filling their available disk.  They were adding 1TB of cache every three to four months, costing hundreds of thousands of dollars to purchase the additional disk.  (Nevermind the cost of operating it -- floor space in the datacenter, electricity, and cooling.)  Configuring the Application Group with TSM from the start would have allowed them to migrate this data off to tape where it would comply with the court order, but cost hundreds of dollars, not hundreds of thousands of dollars.

2)  A health care firm was using CMOD in a cache-only environment.  In the middle of the day, a technician re-zoned disks on the SAN, and wiped out disks on a handful of servers, including some of CMOD's cache filesystems.  With TSM and a directly connected tape library, (and some help) they would have been able to re-populate the cache from the TSM tapes.  Without TSM, they suffered a 48h outage while tapes were pulled from their offsite location, and queued up behind the other systems that were damaged, and finally restored.

In summary, TSM is AWESOME, and anyone using CMOD for archival purposes should be using it.   

-JD.

[rbarrenechea]

Hi:

Two comments here:

1) IBM does advice that you can run a system without "CACHE" by using TSM on a IBM DR550 system. But Disk today is disk is not exactly cheap, but more accesible.

2) You cannot change your storage ( I bet you already know this) BUT do you know that you can move all the data from the cache as well as your indexes utilisng a few DB2 and OnDemand Commands?
 - The first part of this is to create a new application (basically a copy of your "Cache Only" one that utilise TSM.
 - Then you load "test file" and the you unload the test file. This does create the table space.
 - You export your existing app table data, and import it on the new app.
 - You run the table statistics.
 - Then with the aramsint command you start extracting the "load" object name, the easiest way to get this value is by running a "select distinct" to the table, and the use the arsmaint to reload to the new app (no index required).

Now this is very high level and I know a few IBMers who will called un-spported, but I have done it a few times now, and a migration of 5 months (normal method - 3 million rows) took me less than 10 days. But not every environment is the same and the above is very high level. So good luck.

[Alessandro Perucchi]

Hello rbarrenechea,

your idea seems nice, but can you explain better the last point?

 > Then with the aramsint command you start extracting the "load" object name, the easiest way to get this value is by running a "select distinct" to the table, and the use the arsmaint to reload to the new app (no index required).

Because, from what I see in your explanation, all you have done is the following: Create a new segment table, copy all data from the old to the new... but nothing explains how to copy data from cache to TSM nodes.

Alex