diff --git a/opencti-documentation/docs/assets/getting-started/screenshot.png b/opencti-documentation/docs/assets/getting-started/screenshot.png index 399b7227782c..7b83fb625990 100644 Binary files a/opencti-documentation/docs/assets/getting-started/screenshot.png and b/opencti-documentation/docs/assets/getting-started/screenshot.png differ diff --git a/opencti-documentation/docs/assets/installation/connectors_status.png b/opencti-documentation/docs/assets/installation/connectors_status.png index c77b2d3c2190..3e950254a25b 100644 Binary files a/opencti-documentation/docs/assets/installation/connectors_status.png and b/opencti-documentation/docs/assets/installation/connectors_status.png differ diff --git a/opencti-documentation/docs/development/installation.md b/opencti-documentation/docs/development/installation.md index b17d74bbd249..e37d7e7e64d0 100644 --- a/opencti-documentation/docs/development/installation.md +++ b/opencti-documentation/docs/development/installation.md @@ -48,7 +48,7 @@ $ cd opencti ### Install the API dependencies ```bash -$ cd opencti-graphql +$ cd opencti-platform/sopencti-graphql $ yarn install ``` @@ -61,12 +61,7 @@ $ yarn install ### Install the worker dependencies ```bash -$ pip3 install -r requirements.txt -``` - -### Install the integration dependencies - -```bash +$ cd ../../opencti-worker $ pip3 install -r requirements.txt ``` @@ -111,24 +106,7 @@ Change the *config.yml* file according to your #### Start ```bash -$ python3 worker_export.py & -$ python3 worker_import.py & -``` - -### Integration - -#### Configure - -```bash -$ cd opencti-integration -$ cp config.yml.sample config.yml -``` -Change the *config.yml* file according to your - -#### Start - -```bash -$ python3 connectors_scheduler.py +$ python3 worker.py & ``` ### Frontend diff --git a/opencti-documentation/docs/getting-started/requirements.md b/opencti-documentation/docs/getting-started/requirements.md index ef9dbf952943..d874138dda8b 100644 --- a/opencti-documentation/docs/getting-started/requirements.md +++ b/opencti-documentation/docs/getting-started/requirements.md @@ -36,6 +36,14 @@ ElasticSearch is also a JAVA process that needs a minimal amount of memory and C > In order to setup the JAVA memory allocation, you can use the environment variable `ES_JAVA_OPTS`. You can find more information in the [official ElasticSearch documenation](ttps://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html). +### Minio + +Minio has a very small footprint but depending on what you intend to store on OpenCTI, it could require disk space: + +| CPU | RAM | Disk type | Disk space | +| ------------- |---------------| ---------------------------|-----------------------------------| +| 1 core | 128MB | Normal | 1GB | + ### Redis Redis has a very small footprint and only needs a tiny configuration: diff --git a/opencti-documentation/docs/installation/connectors.md b/opencti-documentation/docs/installation/connectors.md index 9c19b9a66f45..13446162b057 100644 --- a/opencti-documentation/docs/installation/connectors.md +++ b/opencti-documentation/docs/installation/connectors.md @@ -6,15 +6,38 @@ sidebar_label: Enable connectors ## Introduction -Connectors are standalone processes that are independant of the rest of the platform. They are using RabbitMQ to push data to OpenCTI, through a dedicated queue for each instance of connector. Depending on your deployment, you can enable connectors by using the connectors Docker images or launch them manually. +Connectors are standalone processes that are independant of the rest of the platform. They are using RabbitMQ to consume or push data to OpenCTI, through a dedicated queue for each instance of connector. Depending on your deployment, you can enable connectors by using the connectors Docker images or launch them manually. ## Connector configurations -All connectors have 2 mandatory configuration parameters, the `name` and the `confidence_level`. The `name` is the name of the instance of the connector. For instance, for the MISP connector, you can launch as many MISP connectors as you need, if you need to pull data from multiple MISP instances. +All connectors have to be able to access to the OpenCTI API. To allow this connection, they have 2 mandatory configuration parameters, the `OPENCTI_URL` and the `OPENCTI_TOKEN`. In addition of these 2 parameters, connectors have 5 other mandatory parameters that need to be set in order to get them work. -> The `name` of each instance of connector must be unique. +``` +- CONNECTOR_ID=ChangeMe +- CONNECTOR_TYPE=INTERNAL_EXPORT_FILE +- CONNECTOR_NAME=ExportFileStix2 +- CONNECTOR_SCOPE=application/json +- CONNECTOR_CONFIDENCE_LEVEL=3 +- CONNECTOR_LOG_LEVEL=info +``` + +> The `CONNECTOR_ID` must be a valid UUIDv4 + +> The `CONNECTOR_TYPE` must be a valid type, the possible types are: +> - EXTERNAL_IMPORT: from remote sources to OpenCTI stix2 +> - INTERNAL_IMPORT_FILE: from OpenCTI file system to OpenCTI stix2 +> - INTERNAL_ENRICHMENT: from OpenCTI stix2 to OpenCTI stix2 +> - INTERNAL_EXPORT_FILE: from OpenCTI stix2 to OpenCTI file system + +> The `CONNECTOR_NAME` is an arbitrary name + +> The `CONNECTOR_SCOPE` is the scope handled by the connector: +> - EXTERNAL_IMPORT: entity types that have to be imported by the connectors, if the connector provide more, they will be ignored +> - INTERNAL_IMPORT_FILE: files mime types to support (application/json, ...) +> - INTERNAL_ENRICHMENT: entity types to support (Report, Hash, ...) +> - INTERNAL_EXPORT_FILE: files mime types to generate (application/pdf, ...) -> The `confidence_level` of the connector will be used to set the `confidence_level` of the relationships created by the connector. If a connector needs to create a relationship that already exists, it will check the current `confidence_level` and if it is lower than its own, it will update the relationship with the new information. If it is higher, it will do nothing and keep the existing relationship. +> The `CONNECTOR_CONFIDENCE_LEVEL` of the connector will be used to set the `CONNECTOR_CONFIDENCE_LEVEL` of the relationships created by the connector. If a connector needs to create a relationship that already exists, it will check the current `CONNECTOR_CONFIDENCE_LEVEL` and if it is lower than its own, it will update the relationship with the new information. If it is higher, it will do nothing and keep the existing relationship. ## Docker activation @@ -26,21 +49,23 @@ For instance, to enable the MISP connector, you can add a new service to your `d ``` connector-misp: - image: opencti/connector-misp:{RELEASE_VERSION} + image: opencti/connector-misp:2.0.0 environment: - - RABBITMQ_HOSTNAME=localhost - - RABBITMQ_PORT=5672 - - RABBITMQ_USERNAME=guest - - RABBITMQ_PASSWORD=guest - - MISP_NAME=MISP\ Circle - - MISP_CONFIDENCE_LEVEL=3 - - MISP_URL=http://localhost - - MISP_KEY=ChangeMe - - MISP_TAG=OpenCTI:\ Import - - MISP_UNTAG_EVENT=true - - MISP_IMPORTED_TAG=OpenCTI:\ Imported + - OPENCTI_URL=http://localhost + - OPENCTI_TOKEN=ChangeMe + - CONNECTOR_ID=ChangeMe + - CONNECTOR_TYPE=EXTERNAL_IMPORT + - CONNECTOR_NAME=MISP + - CONNECTOR_SCOPE=misp + - CONNECTOR_CONFIDENCE_LEVEL=3 + - CONNECTOR_LOG_LEVEL=info + - MISP_URL=http://localhost # Required + - MISP_KEY=ChangeMe # Required + - MISP_TAG=OpenCTI:\ Import # Optional, tags of events to be ingested (if not provided, import all!) + - MISP_UNTAG_EVENT=true # Optional, remove the tag after import + - MISP_IMPORTED_TAG=OpenCTI:\ Imported # Required, tag event after import + - MISP_FILTER_ON_IMPORTED_TAG=true # Required, use imported tag to know which events to not ingest - MISP_INTERVAL=1 # Minutes - - MISP_LOG_LEVEL=info restart: always ``` diff --git a/opencti-documentation/docs/installation/docker.md b/opencti-documentation/docs/installation/docker.md index 48dbb660859a..b2dae8f87c50 100644 --- a/opencti-documentation/docs/installation/docker.md +++ b/opencti-documentation/docs/installation/docker.md @@ -23,7 +23,7 @@ Before running the docker-compose command, please change the admin token (this t - APP__ADMIN__TOKEN=ChangeMe ``` -And change the variable `OPENCTI_TOKEN` (for `worker-import` and `worker-export`) according to the value of `APP__ADMIN__TOKEN` +And change the variable `OPENCTI_TOKEN` (for the `worker` and all connectors) according to the value of `APP__ADMIN__TOKEN` ```yaml - OPENCTI_TOKEN=ChangeMe @@ -80,6 +80,11 @@ volumes: driver_opts: o: bind type: none + s3data: + driver: local + driver_opts: + o: bind + type: none ``` ## Memory configuration @@ -118,6 +123,10 @@ Redis has a very small footprint and only provides an option to limit the maximu You can find more information in the [Redis docker hub](https://hub.docker.com/r/bitnami/redis/). +### Minio + +Minio is a small process and does not require a high amount of memory. More information are available for Linux here on the [Kernel tuning guide](https://github.com/minio/minio/tree/master/docs/deployment/kernel-tuning). + ### RabbitMQ The RabbitMQ memory configuration can be find in the [RabbitMQ official documentation](https://www.rabbitmq.com/memory.html). Basically RabbitMQ will consumed memory until a specific threshold. So it should be configure along with the Docker memory limitation. diff --git a/opencti-documentation/docs/installation/manual.md b/opencti-documentation/docs/installation/manual.md index 4b5ab64dc879..3e2002170aae 100644 --- a/opencti-documentation/docs/installation/manual.md +++ b/opencti-documentation/docs/installation/manual.md @@ -7,12 +7,12 @@ sidebar_label: Manual deployment ## Prerequisites - Node.JS (>= 10) -- Grakn (>= 1.5.7) +- Grakn (>= 1.5.78 - Redis (>= 3.0) - ElasticSearch (>= 7.x) +- Minio (>= 20191012) - RabbitMQ (>= 3.7) - ## Prepare the installation ### Installation of dependencies @@ -65,49 +65,19 @@ $ node dist/server.js & The default username and password are those you put in the `config/production.json` file. -## Install the workers - -2 different workers must be configured to allow the platform to import and export data. One is for import and the other for export. - -### Install the import worker +## Install the worker -#### Configure the import worker +The OpenCTI worker is used to write the data coming from the RabbitMQ messages broker. -Just copy the worker directory to a new one, named `worker-import`. +#### Configure the worker ```bash -$ cp -a worker worker-import -$ cd worker-import +$ cd worker $ pip3 install -r requirements.txt $ cp config.yml.sample config.yml ``` -Change the *config.yml* file according to your OpenCTI token and RabbitMQ configuration. - -> The worker type must be set to "import" - -#### Start as many workers as you need -```bash -$ python3 worker.py & -$ python3 worker.py & -``` - -### Install the export worker - -#### Configure the export worker - -Just copy the worker directory to a new one, named `worker-export`. - -```bash -$ cd .. -$ cp -a worker worker-export -$ cd worker-export -$ cp config.yml.sample config.yml -``` - -Change the *config.yml* file according to your OpenCTI token and RabbitMQ configuration. - -> The worker type must be set to "export" +Change the *config.yml* file according to your OpenCTI token. #### Start as many workers as you need ```bash @@ -124,4 +94,4 @@ $ npm run schema $ npm run migrate ``` -Then start the platform. +Then start the platform. \ No newline at end of file diff --git a/opencti-documentation/website/package.json b/opencti-documentation/website/package.json index 0b29481d32f9..8c59265f92d0 100644 --- a/opencti-documentation/website/package.json +++ b/opencti-documentation/website/package.json @@ -10,6 +10,6 @@ "rename-version": "docusaurus-rename-version" }, "devDependencies": { - "docusaurus": "^1.9.0" + "docusaurus": "^1.14.0" } } diff --git a/opencti-documentation/website/sidebars.json b/opencti-documentation/website/sidebars.json index 1292846a25ef..452215bf4120 100644 --- a/opencti-documentation/website/sidebars.json +++ b/opencti-documentation/website/sidebars.json @@ -14,6 +14,7 @@ "Usage": [ "usage/overview", "usage/model", + "usage/knowledge-create", "usage/reports-create", "usage/report-knowledge" ], diff --git a/opencti-documentation/website/versions.json b/opencti-documentation/website/versions.json index 68efd95a781b..8ffaf3b2b5b9 100644 --- a/opencti-documentation/website/versions.json +++ b/opencti-documentation/website/versions.json @@ -1,4 +1,5 @@ [ + "2.0.0", "1.1.2", "1.1.1", "1.1.0",