Cliver workshop: object storage organisation

[tinaok]

Following conversation at #21 (for Cliver workshop's object storage organisation), I made this issue to discuss the object storage architecture adapted for workflow of cliver workshop.

• During the workshop we'll have different (about 6, 7) working group.
• Each working group probably use common input dataset.
• Some data are already on cloud. But if the data will be repeatedly accessed, it is not optimal to each time use the data from there, but better to create neighbouring cloud Zarr copy, in a optimised chunk for the computation.
• Some (especially observation ) are not on the cloud, we need to store them in neighbouring cloud.

[tinaok]

@annefou

Maybe we should separate input data from data generated by students (that is usually what I do with my courses). With students it can become quickly very messy if they all have write access to "input" data. Mentors are usually in charge of organizing data for their respective groups so maybe mentors could have write access and students read access only do "input" data.

Good idea.
We have tutorial, then workshop.
During workshop I think each group need their own read/write s3 space. If thats private access (limited to all Pangeo enrolled person ) that would be better.

For tutorial, we can create /tmp tutorial output disk space bucket where all tutorial Zarr output will be written (with public access?)

@guillaumeeb would that be possible ?

[tinaok]

@sebastian-luna-valero @annefou

Correct. As explained in #17 (comment) Pangeo users need to enroll the vo.pangeo.eu in aai.egi.eu. So far they are enrolling into the aai-dev.egi.eu instance instead.

I made bucket as

• input-data
• tmp
• WG1
  ( & I'll make WG2, ... later )

input-data is tutorial input datas that only Pangeo admins will push.

tmp is for all attendee to be able to read/ write.

WG1 is also for attendee to be able to write/read, It would better if we can issue ACL control like all vo.pangeo can read, but only some with right access key can write

@sebastian-luna-valero, If I understood correct attendee to be able to write in the tmp folder, they need to enroll the vo.pangeo.eu in aai.egi.eu.
Please let me know which link they can click to make it send petition? Would it also have 'statement' as it was in egi-dev instance?

Thank you

[sebastian-luna-valero]

As it stands at the moment:

• members of vo.pangeo.eu in aai.egi.eu will have read/write access to object storage at CESNET.
• members of vo.pangeo.eu in aai-dev.egi.eu will have read-only access to object storage at CESNET.

Containers created as public will be readable by everybody, but only members of vo.pangeo.eu in aai.egi.eu can write to them.

By the way, @tinaok you are creating containers/buckets in OpenStack project vo.pangeo.eu but you should be using OpenStack project vo.pangeo.eu-swift.

[tinaok]

Thank you @sebastian-luna-valero
I deleted the buckets from vo.pangeo.eu and made them in vo.pangeo.eu-swift

Which link should I send to attendee of workshop to be able to read/write there?

Instead of
https://aai-dev.egi.eu/registry/co_petitions/start/coef:290

I tried https://aai.egi.eu/registry/co_petitions/start/coef:290
but it wasn't written as Pangeo ;-)

Thank you!

[sebastian-luna-valero]

The link is: https://aai.egi.eu/registry/co_petitions/start/coef:386

The statement of purpose needs also to be filled out in there.

[guillaumeeb]

members of vo.pangeo.eu in aai-dev.egi.eu will have read-only access to object storage at CESNET.

@sebastian-luna-valero I didn't get this, currently, they have read access to any buckets in vo.pangeo.eu or vo.pangeo.eu-swift even if the buckets are not public?

@sebastian-luna-valero, If I understood correct attendee to be able to write in the tmp folder, they need to enroll the vo.pangeo.eu in aai.egi.eu.

@sebastian-luna-valero

• If they enroll on vo.pangeo.eu in aai.egi.eu, they won't be admin on compute or storage part of Openstack, right?
• On object storage part, does that make a difference to be admin or just member of vo.pangeo.eu? There are admins, right?
• If they enroll on vo.pangeo.eu in aai.egi.eu they will be able to read, write, create or delete any buckets, correct?

If we want something else, we need to pursue #17, and test https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object-acl.html.

[sebastian-luna-valero]

@sebastian-luna-valero I didn't get this, currently, they have read access to any buckets in vo.pangeo.eu or vo.pangeo.eu-swift even if the buckets are not public?

Maybe the table below clarify things:

Visit it here if you want to comment.

If they enroll on vo.pangeo.eu in aai.egi.eu, they won't be admin on compute or storage part of Openstack, right?

Correct. See table above.

On object storage part, does that make a difference to be admin or just member of vo.pangeo.eu? There are admins, right?

See the difference on the table above.

If they enroll on vo.pangeo.eu in aai.egi.eu they will be able to read, write, create or delete any buckets, correct?

Any buckets on the "vo.pangeo.eu-swift" Object Store. See table above. This is what we are trying to solve in #17 .

I hope it helps.

[tinaok]

@sebastian-luna-valero
Thank you very much it is very clear with the table!

[sebastian-luna-valero]

Closing as obsolete.