1
MP Server / Re: Ondemand - data restoration after removing app gr
« on: June 27, 2019, 10:29:07 PM »
Good point, thanks for your all replies! 
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
Pages: [1] 2
2
MP Server / Re: Ondemand - data restoration after removing app gr
« on: June 26, 2019, 11:36:02 PM »
You are right that nothing can protect you from yourself, for me it was a good lesson & chance to validate our disaster recovery policies. I found out that it would be much faster when objects would not be removed in a fly, but it was in that only scenario when you remove AG. But technically the approach I described seems to be logic for me, regardless if there is a sense of implementing it into the product or not. This is what I wanted to ask you about because I don't see any more benefits for this instant removal, except that it is fast and definitive.
3
MP Server / Re: Ondemand - data restoration after removing app gr
« on: June 26, 2019, 05:05:18 AM »
Hi,
We restored objects, it's not a point here. I'm rather talking about the general approach for removing data this way.
The expiration policy uses the number of days that passed after the segment date that is set for the most common LOAD type. For segment type, it's a different approach, but in both cases removing the application group cause that tables have to be removed from DB and all stored objects from TSM/cache.
So normal expiration policy could expire documents and remove the table later on, the same thing you can accomplish by typing the future date (that exceed a total amount of days to keep data that is set in AG) in arsmaint command for certain AG for instance, -t parameter. I don't see a problem here. For me, this mechanism is already in place.
Of course, it's a human mistake, some cleaning actions due to GDPR, after migrating some data to another archive system. One extra AG was removed. It could be restored more easily I think if data was not removed in a fly.
It's not a common thing, but still, I think it's worth to discuss if there is any benefit from removing data in fly as it is done now.
I'm simply curious.
We restored objects, it's not a point here. I'm rather talking about the general approach for removing data this way.
The expiration policy uses the number of days that passed after the segment date that is set for the most common LOAD type. For segment type, it's a different approach, but in both cases removing the application group cause that tables have to be removed from DB and all stored objects from TSM/cache.
So normal expiration policy could expire documents and remove the table later on, the same thing you can accomplish by typing the future date (that exceed a total amount of days to keep data that is set in AG) in arsmaint command for certain AG for instance, -t parameter. I don't see a problem here. For me, this mechanism is already in place.
Of course, it's a human mistake, some cleaning actions due to GDPR, after migrating some data to another archive system. One extra AG was removed. It could be restored more easily I think if data was not removed in a fly.
It's not a common thing, but still, I think it's worth to discuss if there is any benefit from removing data in fly as it is done now.
I'm simply curious.
4
MP Server / Ondemand - data restoration after removing app gr
« on: June 25, 2019, 01:54:42 AM »
Hi,
I have quite an interesting topic concerning data recovery due to the accidental removal of the application group.
As you are aware, removing an application group is not a common thing that admin does, but still it possible to remove application group quite easily after confirmation.
After that, all links to objects are removed from CMOD DB, and what's even worse, all objects are removed in a fly from cache/TSM in a fly. OnDemand sends delete filespace command to TSM. It removes the data on the primary and eventually the copy pools and there is no retention policies in effect at all at that point. The delete filespace command in TSM is under the understanding that you don't want the data anymore, so it is gone out of the TSM database.
It is still on the copy pool because the reuse delay on the data is a few days, but the only way to get it back it restoring the TSM database.
Restoring the TSM database to get a small piece of data back is not a good solution to this. I feel we shouldn't be restoring TSM databases for data retrieval unless it is a disaster recovery situation because it is a major undertaking to do it to another server.
I'm wondering why it is done like that. Why data is not marked for be removed after next run of arsmaint instead of removing all of it in a fly. I have a feeling that having it pushed through the general expiration policy would be a much more secure approach, letting you revoke the bad decision, but maybe there is something I'm not aware of.
What do you think about that? Have you ever deal with such issue before? Do you have any experience how to secure such scenario?
I'd appreciate getting some feedbacks on how is that solved on your end. I see no good approach here. I started a talk with IBM to understand what options we have and got a reply that it would require enhancement to the product. I'm not sure if this is in scope.
What is your point of view? Any comments highly appreciated
I have quite an interesting topic concerning data recovery due to the accidental removal of the application group.
As you are aware, removing an application group is not a common thing that admin does, but still it possible to remove application group quite easily after confirmation.
After that, all links to objects are removed from CMOD DB, and what's even worse, all objects are removed in a fly from cache/TSM in a fly. OnDemand sends delete filespace command to TSM. It removes the data on the primary and eventually the copy pools and there is no retention policies in effect at all at that point. The delete filespace command in TSM is under the understanding that you don't want the data anymore, so it is gone out of the TSM database.
It is still on the copy pool because the reuse delay on the data is a few days, but the only way to get it back it restoring the TSM database.
Restoring the TSM database to get a small piece of data back is not a good solution to this. I feel we shouldn't be restoring TSM databases for data retrieval unless it is a disaster recovery situation because it is a major undertaking to do it to another server.
I'm wondering why it is done like that. Why data is not marked for be removed after next run of arsmaint instead of removing all of it in a fly. I have a feeling that having it pushed through the general expiration policy would be a much more secure approach, letting you revoke the bad decision, but maybe there is something I'm not aware of.
What do you think about that? Have you ever deal with such issue before? Do you have any experience how to secure such scenario?
I'd appreciate getting some feedbacks on how is that solved on your end. I see no good approach here. I started a talk with IBM to understand what options we have and got a reply that it would require enhancement to the product. I'm not sure if this is in scope.
What is your point of view? Any comments highly appreciated
5
MP Server / Re: Extract reports and their index files from CMOD on AS400
« on: March 14, 2017, 01:22:27 PM »
For sure its arsdoc get
Read some manual about it and you can start with this:
http://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_72/com.ibm.ondemand.administeringi5os.doc/doddo009.htm
Read some manual about it and you can start with this:
http://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_72/com.ibm.ondemand.administeringi5os.doc/doddo009.htm
6
MP Server / Re: Showing Chinese Characters (UTF-08) in OnDemand Client Searching
« on: March 14, 2017, 12:55:27 PM »
Hi,
What is your DB code page? I assume it's UTF8 (ex 1208). For me it looks as those Chinesse characters got converted into different codeset during loading, that's why you don't see them. But it's just my guess.
Could you send the indexfile you used for loading? It could clear some doubts here.
In general if you load data together with indexfile, you shall use save it as UTF-8, to not loose those characters (UltraEdit shows it pretty good, you can use it for checking). Additionally you should use appropriate codepage in indexfile that match that codeset and of course you must have UTF DB But it's obvious.
That shall be it. I could assist you more if you provide me the indexfile.
What is your DB code page? I assume it's UTF8 (ex 1208). For me it looks as those Chinesse characters got converted into different codeset during loading, that's why you don't see them. But it's just my guess.
Could you send the indexfile you used for loading? It could clear some doubts here.
In general if you load data together with indexfile, you shall use save it as UTF-8, to not loose those characters (UltraEdit shows it pretty good, you can use it for checking). Additionally you should use appropriate codepage in indexfile that match that codeset and of course you must have UTF DB
But it's obvious.That shall be it. I could assist you more if you provide me the indexfile.
7
MP Server / Re: Specifying the codepage (or charset) of the command line arguments of arsdoc
« on: March 08, 2017, 04:17:24 AM »
Hi,
I assume your DB has 1209 codepage. What is the locale /charset for your OS? I guess that your command line locale settings might have an impact here.
I can show you how to do this on AIX:
1. simply check you locale setting:
#locale -a
2. Check if your desired UTF locale is already installed, the list for locale you can find here:
https://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.nlsgdrf/support_languages_locales.htm
3. If desired UTF locale is missing (with capital letters are those for UTF) please add it
#smit
->System Environments->Manage Language Environment->Add Additional Language Environments
4. Now change locale in fly in console by setting LANG for environment and for LC_ALL=PL_PL.
You can read more about this here:
https://www.ibm.com/support/knowledgecenter/en/SSSHTQ_8.1.0/com.ibm.netcool_OMNIbus.doc_8.1.0/omnibus/wip/install/concept/omn_con_settingyourlocale.html
"If your system does not support double-byte character sets, you see garbage characters or small block-like characters when you display operating-system specific man pages.
Set the environment variables LC_ALL and LANG to the desired locale in which you want to run the commands by using the export command. For example, run export - for instance export LC_ALL=PL_PL and export LANG=PL_PL. "
Now your command should succeed, if you choosed correct locale.
BR
MM
I assume your DB has 1209 codepage. What is the locale /charset for your OS? I guess that your command line locale settings might have an impact here.
I can show you how to do this on AIX:
1. simply check you locale setting:
#locale -a
2. Check if your desired UTF locale is already installed, the list for locale you can find here:
https://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.nlsgdrf/support_languages_locales.htm
3. If desired UTF locale is missing (with capital letters are those for UTF) please add it
#smit
->System Environments->Manage Language Environment->Add Additional Language Environments
4. Now change locale in fly in console by setting LANG for environment and for LC_ALL=PL_PL.
You can read more about this here:
https://www.ibm.com/support/knowledgecenter/en/SSSHTQ_8.1.0/com.ibm.netcool_OMNIbus.doc_8.1.0/omnibus/wip/install/concept/omn_con_settingyourlocale.html
"If your system does not support double-byte character sets, you see garbage characters or small block-like characters when you display operating-system specific man pages.
Set the environment variables LC_ALL and LANG to the desired locale in which you want to run the commands by using the export command. For example, run export - for instance export LC_ALL=PL_PL and export LANG=PL_PL. "
Now your command should succeed, if you choosed correct locale.
BR
MM
8
MP Server / Re: Helpl Loading PDF Files
« on: March 08, 2017, 01:08:22 AM »
My approach for very fast loading is to create one indexfile that contains set of indexes to thousands of documents in it, each for separate file. You can concatenate all single index files into one, with small changes (no need of codepage definition for instance), use GROUP_OFFSET:0 and GROUP_LENGTH:0, as you don;t want to devide input date into smaller pieces, as Lars mentioned and then simply run that indexfile with empty trigger ard file. Indexfile have relative information there the input data is, and all indexes with it.
I used to load even 50000 of documents within few minutes with this approach. I have not tested how much data could be loaded within an hour, but for sure many times more instead of using ordinary arsload process.
I used to load even 50000 of documents within few minutes with this approach. I have not tested how much data could be loaded within an hour, but for sure many times more instead of using ordinary arsload process.
9
MP Server / Re: ARSMAINT slowness
« on: March 08, 2017, 12:53:13 AM »Slowness in arsmaint is normal and expected as your system grows. There are also a few things that can cause arsmaint to take a long time... Loading millions / billions of small files results in millions/billions of small files and database entries if your Expiration Type setting in the Application Group is Load, or if you have a particularly enormous Application Group whose Expiration Type is set to Document.
For one system I worked on, due to just a few configuration mistakes and combined with the size of their system, meant that arsmaint took more than 24 hours to complete -- so instead of scheduling it to run nightly, they just left it running in a loop -- 24x7x365.
It's not optimal, but it doesn't really hurt anything.
-JD.
Justin - that's interesting what you are saying here. I thought that running arsmaint at least for db expiration is not recommended together with the loading process. So they remove data and load in the same time? With no harm? I know it's not optimal, but once I got feedback from IBM to not do that. I don't know if this was due to their experience or simply the recommendations.
10
MP Server / Re: Clearing the cache
« on: June 01, 2016, 10:08:55 PM »
Thanks!
I know how to do this, I just wanted to make sure my approach is correct. Thanks for your explanation JBNC
I know how to do this, I just wanted to make sure my approach is correct. Thanks for your explanation JBNC
11
MP Server / Re: Making Java connection to AIX Server for remote ARSLOAD API calls
« on: May 31, 2016, 05:07:57 AM »
Another approach is to use DDE interface, especially that calls will be invoked from Windows. Of course CMOD client is needed to perform that calls but it works pretty fine. It can be really useful in some scenarios. Check Windows Client Customization Guide
https://www-05.ibm.com/e-business/linkweb/publications/servlet/pbi.wss?CTY=US&FNC=SRX&PBL=SC19-3357-01
https://www-05.ibm.com/e-business/linkweb/publications/servlet/pbi.wss?CTY=US&FNC=SRX&PBL=SC19-3357-01
12
MP Server / Re: Sync user password between 2 Ondemand instances
« on: May 31, 2016, 04:42:09 AM »
Unfortunately I have no experience in exporting passwords between CMOD instances, but it's maybe worth thinking about switching to LDAP authentication instead, if you have such possibility?
Maybe it's worth also consolidating those both servers into one, it will solve many problems I understand there must be good reason for having them both, but still sometimes it's easier to force such kind of change instead of creating a hybrid, that is hard to maintain and support later on.
Maybe it's worth also consolidating those both servers into one, it will solve many problems
I understand there must be good reason for having them both, but still sometimes it's easier to force such kind of change instead of creating a hybrid, that is hard to maintain and support later on.
13
MP Server / Clearing the cache
« on: May 31, 2016, 03:59:38 AM »
Hi,
I'd like to check with you if my approach for clearing the cache is a good one in scenario when you want to keep the systemlog information untouched.
We received a big batches of data with wrong segment date for dozens of applications (set for 2050 and later) that we did not noticed . It caused that we have really much separate directories in /arscache dir keeping that data. Due to our policy we will keep that data in Ondemand (TSM), but I want to get rid of it from cache. I know I could run arsmaint -c for certain applications with future date (param -t )but it takes really long to process every single application this way.
What I was thinking is to start with cache from scratch, but I'd also like to keep systemlog's daily log for last 2 years untouched. To do this I want to copy all links from /arscache/retr/SL and all data they points to. Copy it to another destination, and remove all data in /arscache dir. After all I would recover retr for SL and data for those links as well.
Is that a good approach or is there a better way to do this? Did I missed anything?
I'm looking for your comments
I'd like to check with you if my approach for clearing the cache is a good one in scenario when you want to keep the systemlog information untouched.
We received a big batches of data with wrong segment date for dozens of applications (set for 2050 and later) that we did not noticed . It caused that we have really much separate directories in /arscache dir keeping that data. Due to our policy we will keep that data in Ondemand (TSM), but I want to get rid of it from cache. I know I could run arsmaint -c for certain applications with future date (param -t )but it takes really long to process every single application this way.
What I was thinking is to start with cache from scratch, but I'd also like to keep systemlog's daily log for last 2 years untouched. To do this I want to copy all links from /arscache/retr/SL and all data they points to. Copy it to another destination, and remove all data in /arscache dir. After all I would recover retr for SL and data for those links as well.
Is that a good approach or is there a better way to do this? Did I missed anything?
I'm looking for your comments
14
MP Server / Restore indexes from FAA1
« on: December 09, 2015, 04:50:11 AM »
I know when data is stored in Ondemand, it's split to two data files located in TSM/cache: compressed object file FAAA and index values FAA1.
Both files are compressed or these are blobs. For sure it's not readable format.
Now I'd like to know how to get access to FAA1 data and recover it back to DB. What is best procedure to do this.
Many Thanks for your guidance.
Both files are compressed or these are blobs. For sure it's not readable format.
Now I'd like to know how to get access to FAA1 data and recover it back to DB. What is best procedure to do this.
Many Thanks for your guidance.
15
Documentation / Re: Loadids - Maximal number
« on: November 05, 2015, 10:56:25 PM »
Great answer! Many thanks Alessandro. I don't have to bother about that till I really face with this
Many thanks Alessandro. I don't have to bother about that till I really face with this
Pages: [1] 2