Skip to content

[S3C 4319] How to test Milestone 2 (in progress)

Jump to bottom
Anurag Mittal edited this page Jun 17, 2021 · 25 revisions

Deployment Instructions

Known issues

Prepare S3 Connector

Install RING v8.4 + S3Connector

Note: For credentials to download packages, please contact [email protected]

Prepare network

  • Open ports 8600 and 8500 between stateful node and stateless node of S3 Connector (OSIS will be installed on the stateless node)
  • Add stateless node IP in Vault config under s3 interface, to allow AssumeRoleBackbeat to be called over port: 8500. By default, localhost is allowed. Hence if the stateful and stateless nodes are the same, no config change is needed.
# Get the location of the config for vault container on the storage node you want OSIS to communicate with
docker inspect scality-vault-1 | grep Source | grep conf
                "Source": "/scality/g1ssd1/s3/scality-vault-1/conf",

# In this case the config is located at /scality/g1ssd1/s3/scality-vault-1/conf/config.json
# Edit the config.json by adding the IP address of stateless node in the below array.
# Below we added `10.100.5.210`

.
.
.
 "interfaces": {
        "S3": {
            "address": "127.0.0.1",
            "port": 8500,
            "allowFrom": [
                "127.0.0.1",
                "::1",
                "10.100.5.203",
                "10.100.5.210"            ]
        },
.
.
.

# Save the config and restart vault container
docker restart scality-vault-1

# Now stateless node on which OSIS is deployed should be able to communicate with Vault over port 8500 for AssumeRoleBackbeat calls

Run OSIS

Prepare OSIS machine

  • Use a stateless node of S3 Connector to run the OSIS docker container. If stateless and stateful nodes are on the same storage nodes in the lap, you can deploy it there.

  • Login to docker registry:

# You will need access to `registry.scality.com/vmware-ose-scality-dev`. For credentials and access to the image repository please contact [email protected].
docker login -u <username> registry.scality.com
  • Pull this docker image(currently REDIS cache not supported)
# [TODO] Change the below image and add Redis cache steps:
docker pull registry.scality.com/vmware-ose-scality-dev/vmware-ose-scality:609c784

  • Generate a self-signed SSL certificate and store it in a binary PKCS#12 format file with extension .p12 file. Instructions here

  • Create an application.properties environment variables file. A sample application.properties file can be found here.

    • Update server.ssl.key-store-password and server.ssl.key-alias with the .p12 file's key-store password and key-store alias, respectively.
    • If OSIS is deployed on a stateless node update the IP address vault endpoint and vault S3 interface endpoint
    • Update the super admin access key ID and secret key ID with that of S3 Connector
      • Default location for admin credentials in s3 connector(lab environments): /srv/scality/s3/s3-offline/federation/env/s3config/vault/admin-clientprofile/admin1.json
    • Update redis sentinel name and endpoints here
      • Example: 
        # Redis config
spring.cache.type=redis
spring.redis.ssl=false
spring.redis.timeout=60000
spring.redis.password=
spring.redis.sentinel.master=scality-s3
spring.redis.sentinel.nodes=10.100.2.202:16379, 10.100.6.138:16380, 10.100.2.80:16379
spring.redis.lettuce.shutdown-timeout=200ms
osis.scality.redis.credentials.hashKey=s3credentials

  • TODO add steps for crypto.yml post PR merge.

  • Run OSIS docker container by following instructions here

    • Sample command when the certificate is stored in the directory /root 
      docker run \
   --env-file application.properties \
   --network host \
   -it \
   -p 8443:8443 \
   -v /root/keyStore.p12:/app/lib/osis.p12:ro,z \
   -v /root/crypto.yml:/app/config/crypto.yml:ro,z \
   registry.scality.com/vmware-ose-scality-dev/vmware-ose-scality:<SHA1>

Use Scality Application with VMWare Object Storage extension

Configure Scality OSIS and Scality S3Connector in VMware cloud director

  • For Admin use the IP address of the stateless node on which OSIS is running with port 8443
  • For S3, configure any stateful node on which cloudserver is running. To route traffic through nginx use port 80. To allow OSE to communicate directly with cloudserver use port 8000

Create Account

  • Log in as a vCloud Director Administrator onto vCloud Director provider portal(https:///provider). The landing page should similar to this. vCloud-Director-UI-administrator-login

  • Create a new organization. This will create a new tenant on vcloud Director create tenant

  • To create a respective account on S3Connector

    • Go to "Object Storage" under "More" object-storage-tab
    • Select your tenant from the list of tenants(in this case "myaccount") select-your-tenant
    • Enable the tenenat(disabled by default) enable-tenant
    • Choose "Default" as the storage tenant ID and select "Enable". This means vCloud director will generate a new storage tenant ID. default storage tenant ID
    • Once enabled the status should have a green check on the Tenants page. This status also confirms the creation of a root user on S3Connector tenant-enabled-and-root-user-created

Use Object Storage

  • Create a new user on Cloud Director, which will be used to login to the tenant portal

    • On the Cloud Director provider portal(https://<vcloud-director-name>/provider) goto resources and open tenant portal while logged in as administrator open-tenant-portal-as-administrator
    • Goto the "Administration" tab and create a new user create-new-user
    • Please add a password that will be used to login to the tenant portal and an appropriate role. Here "vApp User" role was used new user details

  • Login to the tenant portal by going to: https://<vcloud-director-name>/tenant/

  • Enter the name of the organization/tenant created above and select "Next". In this case, "myaccount" tenant-as-organization

  • Use the credentials of the user-created above to log in. login-to-tenant-as-user

  • Create buckets and Objects

    • Once logged in goto object storage tab goto-object-storage-as-user
    • Select "Buckets" on the left sidebar. Currently no buckets exist as this is a new tenant. create a new bucket, by selecting "Get Started" get-started
    • Select the newly created bucket(click on bucket name) and upload objects bucket-created upload-object

Network Troubleshooting

  • Ensure that the stateless node on which OSIS is running can make connections on the configured stateful node on ports,

    • Vault administration interface: 8600
    • Vault s3 interface: 8500
    • Nginx: 80
    • Cloudserver - 8000 (cloudserver)

  • Ensure the server running vCloud Director object storage extension can reach the IP address of the stateless node over port 8443 on which OSIS is running.

Clone this wiki locally