Skip to content

Latest commit

 

History

History
224 lines (172 loc) · 5.69 KB

install-audit.md

File metadata and controls

224 lines (172 loc) · 5.69 KB

Audit Plugin Installation Instructions

Install OpenSearch

For native installation, this needs to be installed on Ubuntu 20.04.

  1. Install Docker

    Set up Docker's Apt repository.

    sudo apt install ca-certificates curl gnupg
sudo install --directory --mode=0755 /etc/apt/keyrings
curl --fail --location --show-error --silent https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor --output /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
printf \
    'deb [arch=%s signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu %s stable\n' \
    "$(dpkg --print-architecture)" \
    "$(. /etc/os-release && echo "$VERSION_CODENAME")" \
  | sudo tee /etc/apt/sources.list.d/docker.list \
  > /dev/null

    Install the Docker packages.

    sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

  2. Configure host for OpenSearch

    Disable memory paging and swapping.

     sudo swapoff --all

    Edit the sysctl config file, /etc/sysctl.conf, setting the max map count to 262,144.

    echo vm.max_map_count=262144 | sudo tee --append /etc/sysctl.conf > /dev/null

    Reload the kernel parameters.

    sudo sysctl --load

  3. Get the All-in-One Compose file

    wget \
  https://raw.githubusercontent.com/opensearch-project/documentation-website/2.9/assets/examples/docker-compose.yml

  4. Start OpenSearch

    sudo docker compose up --detach

Add AMQP 1.0 support to Data Store's RabbitMQ Broker

The AMQP 1.0 plugin needs to be configured for the RabbitMQ broker.

  1. Enable AMQP 1.0 plugins

    sudo rabbitmq-plugins enable rabbitmq_amqp1_0

  2. Configure AMQP 1.0 plugin

    cat <<EOF | sudo tee --append /etc/rabbitmq/rabbitmq.conf > /dev/null
amqp1_0.default_vhost = $DS_VHOST
amqp1_0.convert_app_props_to_amqp091_headers = true
EOF

  3. Restart broker

    sudo systemctl restart rabbitmq-server

Install LogStash

  1. Add Elastic Apt repository.

    curl --fail --location --show-error --silent https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor --output /usr/share/keyrings/elastic-keyring.gpg
sudo apt install apt-transport-https
echo \
    'deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main' \
  | sudo tee --append /etc/apt/sources.list.d/elastic-8.x.list \
  > /dev/null

  2. Install logstash

    sudo apt update
sudo apt install logstash

  3. Install OpenSearch and RabbitMQ logstash plugins

    sudo /usr/share/logstash/bin/logstash-plugin install \
  logstash-output-opensearch logstash-input-rabbitmq

  4. Configure pipeline

    sudo cat <<EOF | tee /etc/logstash/conf.d/irods.conf > /dev/null
input {
   rabbitmq {
      host => "$RABBITMQ_HOST"
      vhost => "$DS_VHOST"
      user => "$LOGSTASH_RABBITMQ_USERNAME"
      password => "$LOGSTASH_RABBITMQ_PASSWORD"
      exchange => "irods"
      key => "audit"
      codec => plain { charset => "ISO-8859-1" }
   }
}

filter {
   # Remove AMQP 1.0 header
   mutate { gsub => [ "message", "[^{]*(.*)", "\1" ] }

   json {
      source => "message"
      target => "message"
   }

   # Replace @timestamp with the timestamp stored in message field @timestamp
   date { match => [ "[message][@timestamp]", "UNIX_MS" ] }

   # TODO: Verify which fields need to be converted to integers for iRODS 4.3.1
   mutate {
      convert => {
         "[message][file_size]" => "integer"
         "[message][data_size]" => "integer"
      }
   }

   # Convert nameless int fields to integers and removed ERROR fields
   ruby {
      code => '
         event.get("message").to_hash.each { |k, v|
            if ( k =~ /^int(__[0-9]+)?$/ )
               event.set("[message][" + k + "]", v.to_i)
            elsif ( k =~ /^ERROR(__[0-9]+)?$/ )
               event.remove("[message][" + k + "]")
            end
         }
      '
   }
}

output {
   opensearch {
      hosts => [ "https://${OPENSEARCH_HOST}:9200" ]
      user => "$LOGSTASH_OPENSEARCH_USERNAME"
      password => "$LOGSTASH_OPENSEARCH_PASSWORD"
      index => "irods-audit-%{+YYYY.MM.dd}"
   }
}
EOF

  5. Run logstash

    sudo systemctl enable logstash
sudo systemctl start logstash

Add iRODS Audit Plugin to Data Store iRODS catalog service provider

  1. Install the iRODS audit plugin

    sudo apt install irods_rule-engine-plugin-audit-amqp

  2. Configure iRODS to use audit plugin

    Prepend the audit plugin configuration to the list of rule engine plugin configurations.

    jq --from-file /dev/stdin /etc/irods/server_config.json \
<<JQ | sudo sponge /etc/irods/server_config.json
.plugin_configuration.rule_engines =
   [
      {
         instance_name: "irods_rule_engine_plugin-audit_amqp-instance",
         plugin_name: "irods_rule_engine_plugin-audit_amqp",
         plugin_specific_configuration : {
            amqp_location:
               "amqp://$IRODS_RABBITMQ_USERNAME:$IRODS_RABBITMQ_PASSWORD@$RABBITMQ_HOST:5672",
            amqp_topic: "/exchange/irods/audit",
            pep_regex_to_match: "pep_.+"
         }
      }
   ]
   + .plugin_configuration.rule_engines
JQ

  3. Restart iRODS

    sudo systemctl restart irods