|
| 1 | +# Timestream Output Plugin |
| 2 | + |
| 3 | +The Timestream output plugin writes metrics to the [Amazon Timestream] service. |
| 4 | + |
| 5 | +### Configuration |
| 6 | + |
| 7 | +```toml |
| 8 | +# Configuration for sending metrics to Amazon Timestream. |
| 9 | +[[outputs.timestream]] |
| 10 | + ## Amazon Region |
| 11 | + region = "us-east-1" |
| 12 | + |
| 13 | + ## Amazon Credentials |
| 14 | + ## Credentials are loaded in the following order: |
| 15 | + ## 1) Assumed credentials via STS if role_arn is specified |
| 16 | + ## 2) Explicit credentials from 'access_key' and 'secret_key' |
| 17 | + ## 3) Shared profile from 'profile' |
| 18 | + ## 4) Environment variables |
| 19 | + ## 5) Shared credentials file |
| 20 | + ## 6) EC2 Instance Profile |
| 21 | + #access_key = "" |
| 22 | + #secret_key = "" |
| 23 | + #token = "" |
| 24 | + #role_arn = "" |
| 25 | + #profile = "" |
| 26 | + #shared_credential_file = "" |
| 27 | + |
| 28 | + ## Endpoint to make request against, the correct endpoint is automatically |
| 29 | + ## determined and this option should only be set if you wish to override the |
| 30 | + ## default. |
| 31 | + ## ex: endpoint_url = "http://localhost:8000" |
| 32 | + # endpoint_url = "" |
| 33 | + |
| 34 | + ## Timestream database where the metrics will be inserted. |
| 35 | + ## The database must exist prior to starting Telegraf. |
| 36 | + database_name = "yourDatabaseNameHere" |
| 37 | + |
| 38 | + ## Specifies if the plugin should describe the Timestream database upon starting |
| 39 | + ## to validate if it has access necessary permissions, connection, etc., as a safety check. |
| 40 | + ## If the describe operation fails, the plugin will not start |
| 41 | + ## and therefore the Telegraf agent will not start. |
| 42 | + describe_database_on_start = false |
| 43 | + |
| 44 | + ## The mapping mode specifies how Telegraf records are represented in Timestream. |
| 45 | + ## Valid values are: single-table, multi-table. |
| 46 | + ## For example, consider the following data in line protocol format: |
| 47 | + ## weather,location=us-midwest,season=summer temperature=82,humidity=71 1465839830100400200 |
| 48 | + ## airquality,location=us-west no2=5,pm25=16 1465839830100400200 |
| 49 | + ## where weather and airquality are the measurement names, location and season are tags, |
| 50 | + ## and temperature, humidity, no2, pm25 are fields. |
| 51 | + ## In multi-table mode: |
| 52 | + ## - first line will be ingested to table named weather |
| 53 | + ## - second line will be ingested to table named airquality |
| 54 | + ## - the tags will be represented as dimensions |
| 55 | + ## - first table (weather) will have two records: |
| 56 | + ## one with measurement name equals to temperature, |
| 57 | + ## another with measurement name equals to humidity |
| 58 | + ## - second table (airquality) will have two records: |
| 59 | + ## one with measurement name equals to no2, |
| 60 | + ## another with measurement name equals to pm25 |
| 61 | + ## - the Timestream tables from the example will look like this: |
| 62 | + ## TABLE "weather": |
| 63 | + ## time | location | season | measure_name | measure_value::bigint |
| 64 | + ## 2016-06-13 17:43:50 | us-midwest | summer | temperature | 82 |
| 65 | + ## 2016-06-13 17:43:50 | us-midwest | summer | humidity | 71 |
| 66 | + ## TABLE "airquality": |
| 67 | + ## time | location | measure_name | measure_value::bigint |
| 68 | + ## 2016-06-13 17:43:50 | us-west | no2 | 5 |
| 69 | + ## 2016-06-13 17:43:50 | us-west | pm25 | 16 |
| 70 | + ## In single-table mode: |
| 71 | + ## - the data will be ingested to a single table, which name will be valueOf(single_table_name) |
| 72 | + ## - measurement name will stored in dimension named valueOf(single_table_dimension_name_for_telegraf_measurement_name) |
| 73 | + ## - location and season will be represented as dimensions |
| 74 | + ## - temperature, humidity, no2, pm25 will be represented as measurement name |
| 75 | + ## - the Timestream table from the example will look like this: |
| 76 | + ## Assuming: |
| 77 | + ## - single_table_name = "my_readings" |
| 78 | + ## - single_table_dimension_name_for_telegraf_measurement_name = "namespace" |
| 79 | + ## TABLE "my_readings": |
| 80 | + ## time | location | season | namespace | measure_name | measure_value::bigint |
| 81 | + ## 2016-06-13 17:43:50 | us-midwest | summer | weather | temperature | 82 |
| 82 | + ## 2016-06-13 17:43:50 | us-midwest | summer | weather | humidity | 71 |
| 83 | + ## 2016-06-13 17:43:50 | us-west | NULL | airquality | no2 | 5 |
| 84 | + ## 2016-06-13 17:43:50 | us-west | NULL | airquality | pm25 | 16 |
| 85 | + ## In most cases, using multi-table mapping mode is recommended. |
| 86 | + ## However, you can consider using single-table in situations when you have thousands of measurement names. |
| 87 | + mapping_mode = "multi-table" |
| 88 | + |
| 89 | + ## Only valid and required for mapping_mode = "single-table" |
| 90 | + ## Specifies the Timestream table where the metrics will be uploaded. |
| 91 | + # single_table_name = "yourTableNameHere" |
| 92 | + |
| 93 | + ## Only valid and required for mapping_mode = "single-table" |
| 94 | + ## Describes what will be the Timestream dimension name for the Telegraf |
| 95 | + ## measurement name. |
| 96 | + # single_table_dimension_name_for_telegraf_measurement_name = "namespace" |
| 97 | + |
| 98 | + ## Specifies if the plugin should create the table, if the table do not exist. |
| 99 | + ## The plugin writes the data without prior checking if the table exists. |
| 100 | + ## When the table does not exist, the error returned from Timestream will cause |
| 101 | + ## the plugin to create the table, if this parameter is set to true. |
| 102 | + create_table_if_not_exists = true |
| 103 | + |
| 104 | + ## Only valid and required if create_table_if_not_exists = true |
| 105 | + ## Specifies the Timestream table magnetic store retention period in days. |
| 106 | + ## Check Timestream documentation for more details. |
| 107 | + create_table_magnetic_store_retention_period_in_days = 365 |
| 108 | + |
| 109 | + ## Only valid and required if create_table_if_not_exists = true |
| 110 | + ## Specifies the Timestream table memory store retention period in hours. |
| 111 | + ## Check Timestream documentation for more details. |
| 112 | + create_table_memory_store_retention_period_in_hours = 24 |
| 113 | + |
| 114 | + ## Only valid and optional if create_table_if_not_exists = true |
| 115 | + ## Specifies the Timestream table tags. |
| 116 | + ## Check Timestream documentation for more details |
| 117 | + # create_table_tags = { "foo" = "bar", "environment" = "dev"} |
| 118 | +``` |
| 119 | + |
| 120 | +### Batching |
| 121 | + |
| 122 | +Timestream WriteInputRequest.CommonAttributes are used to efficiently write data to Timestream. |
| 123 | + |
| 124 | +### Multithreading |
| 125 | + |
| 126 | +Single thread is used to write the data to Timestream, following general plugin design pattern. |
| 127 | + |
| 128 | +### Errors |
| 129 | + |
| 130 | +In case of an attempt to write an unsupported by Timestream Telegraf Field type, the field is dropped and error is emitted to the logs. |
| 131 | + |
| 132 | +In case of receiving ThrottlingException or InternalServerException from Timestream, the errors are returned to Telegraf, in which case Telegraf will keep the metrics in buffer and retry writing those metrics on the next flush. |
| 133 | + |
| 134 | +In case of receiving ResourceNotFoundException: |
| 135 | + - If `create_table_if_not_exists` configuration is set to `true`, the plugin will try to create appropriate table and write the records again, if the table creation was successful. |
| 136 | + - If `create_table_if_not_exists` configuration is set to `false`, the records are dropped, and an error is emitted to the logs. |
| 137 | + |
| 138 | +In case of receiving any other AWS error from Timestream, the records are dropped, and an error is emitted to the logs, as retrying such requests isn't likely to succeed. |
| 139 | + |
| 140 | +### Logging |
| 141 | + |
| 142 | +Turn on debug flag in the Telegraf to turn on detailed logging (including records being written to Timestream). |
| 143 | + |
| 144 | +### Testing |
| 145 | + |
| 146 | +Execute unit tests with: |
| 147 | + |
| 148 | +``` |
| 149 | +go test -v ./plugins/outputs/timestream/... |
| 150 | +``` |
| 151 | + |
| 152 | +[Amazon Timestream]: https://aws.amazon.com/timestream/ |
0 commit comments