Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - Norbert Novotny

Pages: [1] 2 3
1
MP Server / Re: S3 encryption with CMOD
« on: August 28, 2023, 12:10:11 PM »
Thanks Justin,
All that is already in place, sure.
However, this is the encryption on the storage level S3. This is where the 123FAAA will get written to the S3 and should be encrypted by S3 server with your keys.
(as far as my understanding of the process goes)
So, unless there is a way of configuring it via a "undocumented" parameter in the config/s3.cfg file or by convoluting the bucket name entry (something like: my_key_file_path@buckect_name [I am just making it up :-) ]) than I would say it won't work.
Yet, again for me this is steep learning curve, just started with encryption of S3.

Thx,
 N.

2
MP Server / S3 encryption with CMOD
« on: August 28, 2023, 07:39:31 AM »
Hi guys,
Anyone does have any experience with CMOD connected to S3 and using SSE-C (S3 Server Side Encryption with Customer managed keys)?
Not sure if this is even possible, if so, please share some info on how to configure.

Thank you,
 N.

3
Other / Re: Is System Log data stored in DB only or in Cache?
« on: June 21, 2021, 08:08:53 AM »
Just a side note: as Darrell mentioned: "Log message that has View = Yes". and this concern only the output information, not the complete system log record. Probably the most common is output of "arsload -v" or "arsmaint", but there are others as well.

However, all other (complete) information found in System log is *DB only*.
With a help of "arslog - OnDemand User Exit Logging Facility" you could push the messages to a file.

Cheers,
 N.

4
MP Server / Re: determine oldest doc in an app group?
« on: March 03, 2021, 02:56:13 PM »
IMHO, there is more to the question; how you define "old" segment date field value could be back dated e.g. inserted (loaded) today with a date of 10 years back.

However, if you are after e.g. the first document (rather load) ever loaded in this ag, then I would:
a. search for an oldest segment in ARSSEG where agid = ...
b. select the smallest doc_name or better to get min(loadid) example:
Code: [Select]
min(int(left(doc_name,locate(trim(translate(doc_name,'','0123456789')),doc_name)-1))) than look for min doc_off, but that's not relevant

 :)
N.

5
Hi,
Would this be any help to you?

Code: [Select]
select distinct f.name as folder, ag.name as applGroup, a.name as appl from arsag2fol a2f
  inner join arsfol f
    on a2f.fid = f.fid
  inner join arsapp a
    on a2f.agid = a.agid
   and (not exists (select 1 from arsag2fol where aid != 0 and agid = a2f.agid and fid = a2f.fid)
    or a2f.aid=a.aid)
  inner join arsag ag
    on a2f.agid = ag.agid
where f.name in ('My Folder A','My Folder B')
order by 1,2,3;

... probably not the most optimal SQL :-[

Cheers,
 N.

6
MP Server / Re: arsload stores the same AFP Resource many times
« on: May 24, 2020, 12:07:39 AM »
Hi guys,
Thank you so much for helping.
I was not aware of such mechanism, but sure now it make sense, to prevent resource expiry on TSM when TSM is configured with time opposed to event based expiry. Also, the cache does not need to be re-written/updated as the resource is stored under /0/.

... gr8, now I need to close the PMR with IBM :D

@Ed, our setting is retver=14, but I think our issue is that we have not ran tsm expiry, That keeps 27k copies of the same resource.

Thank you,
 N.


7
MP Server / arsload stores the same AFP Resource many times
« on: May 20, 2020, 09:22:57 AM »
Hi guys,
not sure I my understanding of treating resources is correct or not. I have AG/AP with ACIF "RESTYPE=ALL" which does separate resource from data. While resource id 2739 is correctly found as matched,
Code: [Select]
2020-05-20 17:11:33.037909: ARS4312I Loading started, 6286 bytes to process
2020-05-20 17:11:33.075577: ARS1140I Resource /tmp/KS.70636113826651.41b25594-78ca-42a4-b051-a084c1c78101.6395.2020.05.20-11.25.39.afpds.res matches the resource >2739-106-0<

and the resource found in cache is from March-9 in TSM the same resource is stored 26'679 (!) times, given that it is still the same name TSM will try to version it. With so many versions, it starts to slow down the loads as long as 222 seconds ... not talking about space used.

The resource compare is set to 50 as per default, but I don't think so this has anything to do as the resource id is still the same and the content is identical.


Am I missing something? ...or is the is kind of "old" bug, we are on 10.1.0.5 and TSM 6.3 with SSAM mode ON.

Thank you,
 N.

8
MP Server / Re: AWS S3 configuration with third party S3
« on: April 28, 2020, 08:48:44 AM »
Hi guys,

for all interested I will share my experience with EMC ECS S3.

The issue described in my initial post, turn out to be a bug in our load balancer which is not part of std. ECS. Once fixed the addressing worked as expected.

However, there is bug in Dell/EMC ECS as well as in IBM CMOD; this is represented silently not removing / unloading documents from storage while index is removed. The only indication is arssockd trace or by using aws cli to observe that documents are not removed while indices are gone.

 The ECS bug is related to date format with POST/delete command is send the ISO date is not recognized/rejected. The Dell/EMC promptly provided a fix, to cover this issue.

  The IBM CMOD bug is related to format of of the request, where all requests are AWS4 compliant, but unloaded uses mixed AWS4 (get) and AWS2 (delete). The AWS2 request format is soon to be deprecated by Amazon and is already not supported by all AWS regions. (e.g. eu-central-1)

Note: Soon AWS is going to support legal retention period lock down, which will be reflected in ECS S3 as well. I wonder if IBM is going to implement relevant changes  ;)

Hope this helps,
 N.

9
MP Server / Re: OnDemand Server Sizing help for Linux system
« on: March 05, 2020, 12:53:50 AM »
We need to configure a CMOD system to run on Linux.  It's a sizeable requirement with the system having to satisfy a number of CMOD queries/retrievals per second at peak times.  We need to size this system;  that is decide on how many hardware processors, which processors and how much memory is required to give satisfactory fast response for the users.

One aspect to take in consideration, what do you call a "CMOD System":

  • arssockd + arsload + DB2 + TSM(with DB2)
  • arssockd + arsload + DB2 & SM is external e.g. different system aka cloud object store or remote TSM
  • arssockd + arsload & DB2 or Oracle & SM remote
  • arssockd & arsload on different system e.g. via ODWEK (though this is very unusual)

Also need to consider any other Services you may wish to run on the same box, such WAS/ICN, SAMBA (server and/or client), NFSd, sshd

Another question, how is the storage attached to the system, SAN, NAS or direct attached

As a rule of thumb I would say CMOD system is more IO intensive (Network & Disk)  rather than CPU, but someone could argue that point. I am assuming moderate balanced ratio between need to compress and pre-compressed, as well as more searches than actual retrievals.

Well, here is my 5¢  :)
N.

10
Hi,

I would go with export/import similar as I have just reply to your other older post/question.
However this covers Archive indices only, e.g. a case where you need to just migrate CMOD and (e.g. TSM) storage stays the same.
If you also need to export underlying data, I have used export using dsmc (as it is a bit faster) as:
Code: [Select]
dsmc retrieve  -server=TSMServerName -virtualnode=TSMNode -password=NODEPasswd -replace=no -ifnewer -filesonly -filelist=InputFileList MyLocalDir/
The "InputFileList " has all of the doc names but no more then ~5000 per one execution (this is just my experience) one name per line.

Important: Don't forget each of doc_name as 1234FAAA does have hidden 1234FAA1 file not visible in CMOD but present on the storage.

However, arsadmin retrieve works also just fine. One comment, though:  In a case there is an error while retrieving data from storage e.g. corruption, not found or so, then arsadmin would stop with partially retrieved file and would not progress with rest in acase you have specified multiple files on one command line!!
Doing arsadmin retrieve one by one would be very slow as it does: instantiate process, authenticate, write system log record and clean up after, for each of your 100M files !!! That's a lot of fluff for nothing. However, if you still want to go this route I would suggest at least to disable logging for respective AG(s).

Once exported you can loaded to target system with arsadmin store. Also, if you have configured a new Node names on your target and you have copied your segment tables, don't forget to update NIDs (mostly pri_nid) as well arsres.pri_nid

Hope this helps,
 N.


11
MP Server / AWS S3 configuration with third party S3
« on: March 04, 2020, 08:15:21 AM »
Hi guys,

anybody has an experience configuring CMOD with Amazon S3 using non-Amazon service? e.g. Dell EMC ECS with S3 protocol.

As I understand CMOD supports this by configuring the host in config/ars.s3 config file putting there something target ECS host instead of amazonaws.com.

So far so good, AWS supports two means of addressing your bucket: path style and virtual host style.
To be more specific:
path style: https://S3host.your.domain/yourBucket/yourObject
vs
virt. host: https://yourBucket.S3host.your.domain/yourObject

Here is my question:
I am under impression the CMOD supports virtual host addressing ONLY, am I right? ... or is there a way to tweak it to support path style (the old method)? I mean by some sort of "undisclosed" config parameter(s)

I've managed to get it partially working, by configuring your.domain as host (looks odd) and S3host as your Bucket, then put the real bucket name as HLD. That worked for must of cases e.g. load/unload search and retrieve (not cached), however if your folder is configured to display object location icon at first column, this produces errors in System log. Not sure what else would not work as expected, due to this twisted configuration.

Any help here or experience will be appreciated.

Thank you,
 N.

12
Other / Re: Upgrading TSM using Centera
« on: March 04, 2020, 07:34:55 AM »
Hi Norbert, Just wondering about this post again, what approach did you use to migrate the database.

What I did (not a 100% supported by IBM perhaps  ;D )  speaking of CMOD migration or clonning:
  • assuming you have target DB2 instance for CMOD ready with a database created (I have never used the arsdb -c)
  • Export DDL from your primary DB2 using: db2look -d ARCHIVE -a -e -l -x  -o ARCHIVE.sql
  • Edit ARCHIVE.sql and delete / change all what is not related to table spaces related to segment tables, segment Tables and Indices
  • Execute ARCHIVE.sql using db2 -t on your target DB2 (pay attention not to overwrite your primary :-) )
  • On the primary instance export ARS system tables using arsdb -x
  • On the target instance import ARS tables using: arsdb -i Note: don't remember if arsdb -t is required prior arsdb -i, but it doesn't hurt
  • On the primary instance list all your tables "select table_name from arsseg" and put it in an SQL script as "export to ABA1.ixf of IXF select * from ABA1;" one table per line
  • on primary instance execte SQL using db2 -t -z myexport.log -o- -f myexport.sql
  • write your import SQL script similar to export: load from ABA1.ixf of IXF insert into ABA1 nonrecoverable;
  • execute on target system using db2 -t -o- -z myimport.log -f myimport.sql
  • Assuming you have configured your ars.ini and ars.cfg you should be able to boot up you new target instance, if this is just a cloning, for upgrade I would suggest to run following before booting up the instance: arsdb -e -f -F -v to drop all indices, then arsdb -u -v this will alter all tables as necessary per target CMOD version, then arsdb -r -f -v and arsdb -s -v to recreate indices and recalc stats
  • Once the target instance is up&running: arssyscr -l -m -a to re-create System folders
  • to rebuild stats on target segment tables: arsmaint -r
  • Note: don't forget your caches, specially system log and all AGs where you load to cache first than to storage or where storage is cache only
  • run a few testes before releasing to production, load/unload search and retrieve new and old records

For TSM it is similar (somewhat easier for cloning only) , but it has been a while I've done it.

Hope this helps, please bare in mind this is most likely not supported by IBM  ;D and I would not blame them
N.

13
Other / Re: Upgrading TSM using Centera
« on: February 20, 2019, 02:44:24 AM »
Hi,

Perhaps a bit too late for your project now.
I did the TSM migration with Centera between Solaris TSM 5.5 and Linux (RHEL) TSM 7.1 (not in SSAM mode)
The TSM migration was done over the network using this https://www.ibm.com/support/knowledgecenter/en/SSGSG7_7.1.0/com.ibm.itsm.srv.upgrd.doc/t_srv_upgrd_dbmove_net_s4.html as guide. As this was my first migration I did have a few try and fail attempts, but at the end it worked.

Sure, you would need to take the same PEA file (the key file for your Centera pool). To migrate CMOD DB2 you could use db2move or write a simple script to do export to IXF and load from IXF. It is fast and does the endian conversion.

If you would prefer data export / import the method described by Alessandro is the right one in essence. You could consider running multi process/thread your export/import to gain speed.

Hope this helps,
N.

14
MP Server / Re: Unable to store object Failure
« on: February 20, 2019, 02:03:07 AM »
Hi guys, yes I can confirm this issue as well.

Our setup is CMOD on AIX with TSM on AIX accessing ECS via CAS API.

02/20/19   08:10:19      ANR2547E A Centera device (5) reported error "Unknown error (transid='sb002017/9008600/WRITE_BLOB'),FP_SERVER_-ERR" during command BlobWrite. (SESSION: 702829)
02/20/19   08:10:19      ANR0523W Transaction failed for session 702829 for node NODE01 (CMOD) - error on output storage device. (SESSION: 702829)
02/20/19   08:10:19      ANR0514I Session 702829 closed volume 00003C41.CNT. (SESSION: 702829)
02/20/19   08:10:19      ANR0403I Session 702829 ended for node NODE01 (CMOD). (SESSION: 702829)


After a re-submit all worked well.
Cheers,
 N.

15
MP Server / Re: General ARSMAINT questions....
« on: January 13, 2019, 06:24:13 AM »
ARSMAINT has two main modes -d (data expiry) and -c (cache expiry).

The -d does document EXPIRY (careful if you never ran this) as it will purge production documents forever !!

You have  also mentioned "All of our archives reside in cache." Not sure if you mean using Cache as a primary storage;
in which case the cache maintenance "-c" will not do anything

 or

each AG is also cached while using other primary storage e.g. TSM or a Cloud Service. In this case the -c "The cache maintenance" will purge cache only while keeping the document(s) in the main storage area, and it is safe to use.

Otherwise, as Justin mentioned run it first on a small AG first, ideal in a test env and review system log afterwards, also it is good take df -m /my_cache_dir before and after so you could see how many MB was released.


Pages: [1] 2 3