AWS S3 configuration with third party S3

[Norbert Novotny]

Hi guys,

anybody has an experience configuring CMOD with Amazon S3 using non-Amazon service? e.g. Dell EMC ECS with S3 protocol.

As I understand CMOD supports this by configuring the host in config/ars.s3 config file putting there something target ECS host instead of amazonaws.com.

So far so good, AWS supports two means of addressing your bucket: path style and virtual host style.
To be more specific:
path style: https://S3host.your.domain/yourBucket/yourObject
vs
virt. host: https://yourBucket.S3host.your.domain/yourObject

Here is my question:
I am under impression the CMOD supports virtual host addressing ONLY, am I right? ... or is there a way to tweak it to support path style (the old method)? I mean by some sort of "undisclosed" config parameter(s)

I've managed to get it partially working, by configuring your.domain as host (looks odd) and S3host as your Bucket, then put the real bucket name as HLD. That worked for must of cases e.g. load/unload search and retrieve (not cached), however if your folder is configured to display object location icon at first column, this produces errors in System log. Not sure what else would not work as expected, due to this twisted configuration.

Any help here or experience will be appreciated.

Thank you,
 N.

[Justin Derrick]

Sounds like a good enhancement request. 

-JD.

[RHPharr]

We are using EMC ECS via S3 for compliant (SEC 17a-4) storage.  We do not use show doc location.

[Norbert Novotny]

Hi guys,

for all interested I will share my experience with EMC ECS S3.

The issue described in my initial post, turn out to be a bug in our load balancer which is not part of std. ECS. Once fixed the addressing worked as expected.

However, there is bug in Dell/EMC ECS as well as in IBM CMOD; this is represented silently not removing / unloading documents from storage while index is removed. The only indication is arssockd trace or by using aws cli to observe that documents are not removed while indices are gone.

 The ECS bug is related to date format with POST/delete command is send the ISO date is not recognized/rejected. The Dell/EMC promptly provided a fix, to cover this issue.

  The IBM CMOD bug is related to format of of the request, where all requests are AWS4 compliant, but unloaded uses mixed AWS4 (get) and AWS2 (delete). The AWS2 request format is soon to be deprecated by Amazon and is already not supported by all AWS regions. (e.g. eu-central-1)

Note: Soon AWS is going to support legal retention period lock down, which will be reflected in ECS S3 as well. I wonder if IBM is going to implement relevant changes 

Hope this helps,
 N.

[jsquizz]

However, there is bug in Dell/EMC ECS as well as in IBM CMOD; this is represented silently not removing / unloading documents from storage while index is removed. The only indication is arssockd trace or by using aws cli to observe that documents are not removed while indices are gone.

Heads up, this issue also exists in IBM Cloud Object Storage!

[Justin Derrick]

That sounds like a Service Request that someone needs to submit. 

-JD.