this is a test

[jasoncoposky]

Dummy issue to use the editor

[jasoncoposky]

Storage Tiering

The storage tiering rule base provides iRODS the capability of automatically moving data between identified tiers of storage. Each storage resource is tagged as part of a group with an indicator as to where in the group the resource resides. Each resource is also tagged with a value which describes the maximum time a data object may reside on that resource. Should a data object's last time of access exceed this given value it will be migrated to the next resource in the tier. This package also includes a rule base feature which tags objects with their last time of access for this computation.

Rulebase Configuration

Add the four included rule bases above "core" in the re_rule_base_set array of the native iRODS rule language plugin:

"rule_engines": [
    {    
        "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",
        "plugin_name": "irods_rule_engine_plugin-irods_rule_language",
        "plugin_specific_configuration": {  
            "re_data_variable_mapping_set": [
                "core"
            ],
            "re_function_name_mapping_set": [
                "core"
            ],
            "re_rulebase_set": [
                "access_time_configuration",
                "apply_access_time",
                "storage_tiering_configuration",
                "apply_storage_tiering",
                "example_tiering_policy",
                "core"
            ],
            "regexes_for_supported_peps": [
                "ac[^ ]*",
                "msi[^ ]*",
                "[^ ]*pep_[^ ]*_(pre|post)"
            ]
        },
        "shared_memory_instance": "irods_rule_language_rule_engine"
    },
]

Creating a Tier Group

Tier groups are defined via metadata attached to the resources which participate in the group. In iRODS terminology the attribute is defined by a function in the rule base storage_tiering_configuration.re, which by default is irods::storage_tier_group. The value of the metadata triple is the name of the tier group, and the Unit holds the numeric position of the resource within the group. To define a tier group simply choose a name and apply metadata to the selected root resources of given compositions. For example:

imeta add -R fast_resc irods::storage_tier_group example_group 0
imeta add -R medium_resc irods::storage_tier_group example_group 1
imeta add -R slow_resc irods::storage_tier_group example_group 2 

Setting Tiering Policy

Once a tier group is defined, the age limit for each tier must also be configured via metadata. Once a data object has remained unaccessed on a given tier for more than the configured time, it will be staged to the next tier in the group and then trimmed from the previous tier. This is configured via the default attribute irods::storage_tier_time, which is also defined in the storage_tiering_configuration.re rulebase. In order to configure the tiering time, apply a tag to the resource using the given attribute and a positive numeric value in seconds. For example, to configure the fast_resc to only hold data for 30 minutes:

imeta add -R fast_resc irods::storage_tier_time 1800

We can then configure the _medium_resc to hold data for 8 hours:

imeta add -R medium_resc irods::storage_tier_time 28800

Customizing the Violating Objects Query

A tier within a tier group may identify data objects which are in violation by an additional mechanism beyond the built in time based constraint. This allows the data grid administrator to take additional context into account when identifying data objects to migrate. Data objects given a metadata attribute, a specific collection, a particular user or a project may be identified through a custom query attached to the root resource of a given tier within a group. The default attribute irods::storage_tier_query is used to hold the query. To configure the custom query, attach the query to the root resource of the tier within the tier group. This query will be used in place of the default.

imeta set -R rnd1 irods::storage_tier_query "select META_DATA_ATTR_VALUE, DATA_NAME, COLL_NAME where RESC_NAME = 'ufs2' || = 'ufs3' and META_DATA_ATTR_NAME = 'irods::access_time' and META_DATA_ATTR_VALUE < 'TIME_CHECK_STRING'"

This example implements the default query. Note that the string TIME_CHECK_STRING is used in place of an actual time. This string will be replaced by the storage tiering framework with the appropriately computed time given the previous parameters.

[jasoncoposky]

SC17 Workshop Demo Configuration

Add to /etc/irods/server_config.json

                "access_time_configuration",
                "apply_access_time",
                "storage_tiering_configuration",
                "apply_storage_tiering",
                "example_tiering_policy",

Create resources and annotate

iadmin mkresc ufs0 unixfilesystem `hostname`:/tmp/irods/ufs0
iadmin mkresc ufs1 unixfilesystem `hostname`:/tmp/irods/ufs1
iadmin mkresc ufs2 unixfilesystem `hostname`:/tmp/irods/ufs2
iadmin mkresc ufs3 unixfilesystem `hostname`:/tmp/irods/ufs3
iadmin mkresc ufs4 unixfilesystem `hostname`:/tmp/irods/ufs4
iadmin mkresc ufs5 unixfilesystem `hostname`:/tmp/irods/ufs5
iadmin mkresc rnd0 random
iadmin mkresc rnd1 random
iadmin mkresc rnd2 random
iadmin addchildtoresc rnd0 ufs0
iadmin addchildtoresc rnd0 ufs1
iadmin addchildtoresc rnd1 ufs2
iadmin addchildtoresc rnd1 ufs3
iadmin addchildtoresc rnd2 ufs4
iadmin addchildtoresc rnd2 ufs5
imeta add -R rnd0 irods::storage_tier_group example_group 0
imeta add -R rnd1 irods::storage_tier_group example_group 1
imeta add -R rnd2 irods::storage_tier_group example_group 2
imeta add -R rnd0 irods::storage_tier_time 30
imeta add -R rnd1 irods::storage_tier_time 60

imeta set -R rnd1 irods::storage_tier_query "select META_DATA_ATTR_VALUE, DATA_NAME, COLL_NAME where RESC_NAME = 'ufs2' || = 'ufs3' and META_DATA_ATTR_NAME = 'irods::access_time' and META_DATA_ATTR_VALUE < 'TIME_CHECK_STRING'"

Teardown

iadmin rmchildfromresc rnd0 ufs0
iadmin rmchildfromresc rnd0 ufs1
iadmin rmchildfromresc rnd1 ufs2
iadmin rmchildfromresc rnd1 ufs3
iadmin rmchildfromresc rnd2 ufs4
iadmin rmchildfromresc rnd2 ufs5
iadmin rmresc rnd0
iadmin rmresc rnd1
iadmin rmresc rnd2
iadmin rmresc ufs0
iadmin rmresc ufs1
iadmin rmresc ufs2
iadmin rmresc ufs3
iadmin rmresc ufs4
iadmin rmresc ufs5
iadmin rum

[jasoncoposky]

iRODS C++ REST Mid-Tier API

Building this repository

This is a standard CMake project which may be built with either Ninja or Make.

This code base is built with the iRODS toolchain, which uses Clang. Since this project depends on Pistache, we also need to build Pistache with Clang in order to link against that project.

First clone the iRODS externals repository.

https://github.com/irods/externals

Install the latest iRODS CMake and Clang packages

irods-cmake
irods-externals-clangruntime
irods-externals-clang

Then within the externals repository, build Pistache with

make pistache

Then install Pistache with

cd pistache<VERSION>-0_src/pistache/build/
make install

You will then need to install iRODS packages

irods-dev
irods-runtime
irods-externals-boost
irods-externals-json

Once this is done you can create a build directory for the REST API and run CMake, then run

make package

to build the REST API package.

Configuring the service

The REST API provides an executable for each individual API endpoint. These endpoints may be grouped behind a reverse proxy in order to provide a single port for access.

The services rely on a configuration file in /etc/irods/ which dictates which ports are used for each service. Two template files are placed there by the package:

/etc/irods/irods_client_rest_cpp.json.template
/etc/irods/irods-client-rest-cpp-reverse-proxy.conf.template

/etc/irods/irods_client_rest_cpp.json.template should be copied to /etc/irods/irods_client_rest_cpp.json and modified if different ports are desired. The service can then be restarted with service irods_client_rest_cpp restart, or however your specific platform manages services.

Once the REST API is running install nginx and then copy /etc/irods/irods-client-rest-cpp-reverse-proxy.conf.template to /etc/nginx/sites-available/irods-client-rest-cpp-reverse-proxy.conf and then symbolically link it to /etc/nginx/sites-enabled/irods-client-rest-cpp-reverse-proxy.conf. Nginx will then need to be restarted with sudo service nginx restart, or however your specific platform manages services.

Interacting with the API endpoints

The design of this API using JWTs to contain authorization and identity. The Auth endpoint must be invoked first in order to authenticate and receive a JWT. This token will then need to be included in the Authorization header of each subsequent request.

/Access

This endpoint provides a service for the generation of an iRODS ticket to a given logical path, be that a collection or a data object.

Method : POST

Parameters:

• path: The logical path to a collection or data object for which access is desired

Example CURL Command:

curl -X POST -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/access?path=%2FtempZone%2Fhome%2Frods%2Ffile0"

Returns

An iRODS ticket token within the X-API-KEY header, and a URL for streaming the object.

{
  "headers": [
    "X-API-KEY: CS11B8C4KZX2BIl"
  ],
  "url": "/irods-rest/1.0.0/stream?path=/tempZone/home/rods/file0&offset=0&limit=33064"
}

/Admin

The administration interface to the iRODS Catalog which allows the creation, removal and modification of users, groups, resources, and other entities within the zone.

Method : POST

Parameters

• action : dictates the action taken: add, modify, or remove
• target : the subject of the action: user, zone, resource, childtoresc, childfromresc, token, group, rebalance, unusedAVUs, specificQuery
• arg2 : generic argument, could be user name, resource name, depending on the value of action and target
• arg3 : generic argument , see above
• arg4 : generic argument , see above
• arg5 : generic argument , see above
• arg6 : generic argument , see above
• arg7 : generic argument , see above

Example CURL Command:

curl -X POST -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/admin?action=add&target=resource&arg2=ufs0&arg3=unixfilesystem&arg4=/tmp/irods/ufs0&arg5=&arg6=tempZone"

Returns

"Success" or an iRODS exception

/Auth

This endpoint provides an authentication service for the iRODS zone, using one of the available iRODS authentication methodologies, such as: native, pam, krb or gsk.

Method : POST

Parameters:

• user_name : the iRODS user which wishes to authenticate
• password : The user's secret, depending on the method of authentication
• auth_type : The iRODS method of authentication - native, pam, krb, gsk, etc.

Example CURL Command:

export TOKEN=`curl -X POST "http://localhost:80/irods-rest/1.0.0/auth?user_name=rods&password=rods&auth_type=native"`

Returns:

An encrypted JWT which contains everything necessary to interact with the other endpoints. This token is expected in the Authorization header for the other services.

/List

This endpoint provides a recursive listing of a collection, or stat, metadata, and access control information for a given data object

Method : GET

Parameters

• path : The logical path which is to be listed
• stat : Boolean flag to indicate stat information is desired
• permissions : Boolean flag to indicate access control information is desired
• metadata : Boolean flag to indicate metadata is desired
• offset : number of records to skip for pagination
• limit : number of records desired per page

Example CURL Command:

curl -X GET -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100" | jq

Returns

A JSON structured response within the body containing the listing, or an iRODS exception

{
  "_embedded": [
    {
      "logical_path": "/tempZone/home/rods/subcoll",
      "type": "collection"
    },
    {
      "logical_path": "/tempZone/home/rods/subcoll/file0",
      "type": "data_object"
    },
    {
      "logical_path": "/tempZone/home/rods/subcoll/file1",
      "type": "data_object"
    },
    {
      "logical_path": "/tempZone/home/rods/subcoll/file2",
      "type": "data_object"
    },
    {
      "logical_path": "/tempZone/home/rods/file0",
      "type": "data_object"
    }
  ],
  "_links": {
    "first": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100",
    "last": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=UNSUPPORTED&limit=100",
    "next": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=100&limit=100",
    "prev": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100",
    "self": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100"
  }
}

Query

This endpoint provides access to the iRODS General Query language, which is a generic query service for the iRODS catalog.

Method : GET

Parameters

• query_string : The General Query
• query_limit : Number of desired rows
• row_offset : Number of rows to skip for paging
• query_type : Either 'general' or 'specific'

Example CURL Command:

curl -X GET -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/query?query_limit=100&row_offset=0&query_type=general&query_string=SELECT%20COLL_NAME%2C%20DATA_NAME%20WHERE%20COLL_NAME%20LIKE%20%27%2FtempZone%2Fhome%2Frods%25%27" | jq

Returns
A JSON structure containing the query results

{
  "_embedded": [
    [
      "/tempZone/home/rods",
      "file0"
    ],
    [
      "/tempZone/home/rods/subcoll",
      "file0"
    ],
    [
      "/tempZone/home/rods/subcoll",
      "file1"
    ],
    [
      "/tempZone/home/rods/subcoll",
      "file2"
    ]
  ],
  "_links": {
    "first": "/irods-rest/1.0.0query?query_string=SELECT%20COLL_NAME%2C%20DATA_NAME%20WHERE%20COLL_NAME%20LIKE%20%27%2FtempZone%2Fhome%2Frods%25%27&query_limit=100&row_offset=0&query_type=general",
    "last": "/irods-rest/1.0.0query?query_string=SELECT%20COLL_NAME%2C%20DATA_NAME%20WHERE%20COLL_NAME%20LIKE%20%27%2FtempZone%2Fhome%2Frods%25%27&query_limit=100&row_offset=0&query_type=general",
    "next": "/irods-rest/1.0.0query?query_string=SELECT%20COLL_NAME%2C%20DATA_NAME%20WHERE%20COLL_NAME%20LIKE%20%27%2FtempZone%2Fhome%2Frods%25%27&query_limit=100&row_offset=0&query_type=general",
    "prev": "/irods-rest/1.0.0query?query_string=SELECT%20COLL_NAME%2C%20DATA_NAME%20WHERE%20COLL_NAME%20LIKE%20%27%2FtempZone%2Fhome%2Frods%25%27&query_limit=100&row_offset=0&query_type=general",
    "self": "/irods-rest/1.0.0query?query_string=SELECT%20COLL_NAME%2C%20DATA_NAME%20WHERE%20COLL_NAME%20LIKE%20%27%2FtempZone%2Fhome%2Frods%25%27&query_limit=100&row_offset=0&query_type=general"
  },
  "count": "4",
  "total": "4"
}

Stream

Stream data into and out of an iRODS data object

Method : GET and PUT

Parameters

• path : url encoded logical path to a data object
• offset : offset in bytes into the data object
• limit : maximum number of bytes to read

Returns
PUT : Nothing, or iRODS Exception

GET : The data requested in the body of the response

Example CURL Command:

curl -X GET -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/stream?path=%2FtempZone%2Fhome%2Frods%2Ffile0&offset=0&limit=1000" 

or

cat README.md | curl -d - -X PUT -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/stream?path=%2FtempZone%2Fhome%2Frods%2FfileX&offset=0&limit=1000"

Zone Report

Requests a JSON formatted iRODS Zone report, containing all configuration information for every server in the grid.

Parameters
None

Example CURL Command:

curl -X POST -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/zone_report" | jq

Returns
JSON formatted Zone Report

{
  "schema_version": "file:///var/lib/irods/configuration_schemas/v3/zone_bundle.json",
  "zones": [
    {
    <snip>
    }]
}