elastic · kurtiam · Aug 23, 2023 · Aug 23, 2023 · Aug 23, 2023 · Aug 24, 2023
@@ -33,7 +33,7 @@ Info on {agent} and agent integrations:
 
 Info on using a proxy server:
 
-* {fleet-guide}//fleet-agent-proxy-support.html[Using a proxy server with {agent} and {fleet}]
+* {fleet-guide}/fleet-agent-proxy-support.html[Using a proxy server with {agent} and {fleet}]
 
 Info on {es}:
 

@@ -111,7 +111,7 @@ image::images/create-agent-policy.png[{fleet} in {kib}]
 +
 . Create the agent policy:
 * To use the UI, click **Create agent policy**.
-* beta[] To use the {fleet} API, click **Preview API request** and run the
+* To use the {fleet} API, click **Preview API request** and run the
 request.
 
 Also see <<create-a-policy-no-ui>>.
@@ -141,7 +141,7 @@ policies.
 +
 --
 * To use the UI, click **Save and continue**.
-* beta[] To use the {fleet} API, click **Preview API request** and run the
+* To use the {fleet} API, click **Preview API request** and run the
 request.
 --
 
@@ -195,6 +195,8 @@ If you make a mistake, you can always re-configure or re-add an integration.
 
 Any saved changes are immediately distributed and applied to all {agent}s enrolled in the given {policy}.
 
+To update any secret values in an integration policy, refer to <<agent-policy-secret-values>>.
+
 [discrete]
 [[copy-policy]]
 == Copy a policy
@@ -274,6 +276,32 @@ that you added. Lack of connectivity will prevent the {agent}
 from checking in with the {fleet-server} and receiving policy updates, but the agents
 will still forward data to the cluster.
 
+[discrete]
+[[agent-policy-secret-values]]
+== Policy secret values
+
+When you create an integration policy you often need to provide sensitive information such as an API key or a password. To help ensure that data can't be accessed inappropriately, any secret values used in an integration policy are stored separately from other policy details.
+
+As well, after you've saved a secret value in {fleet}, the value is hidden in both the {fleet} UI and in the agent policy definition. When you view the agent policy (**Actions -> View policy**), an environment variable is displayed in place of any secret values, for example `${SECRET_0}`.
+
+WARNING: In order for sensitive values to be stored secretly in {fleet}, all configured {fleet-server}s must be on version 8.10.0 or higher.
+
+Though secret values stored in {fleet} are hidden, they can be updated. To update a secret value in an integration policy:
+
+. In {fleet}, click **Agent policies**.
+Select the name of the policy you want to edit.
+
+. Search or scroll to a specific integration.
+Open the **Actions** menu and select **Edit integration**. Any secret information is marked as being hidden.
+
+. Click the link to replace the secret value with a new one.
++
+[role="screenshot"]
+image::images/fleet-policy-hidden-secret.png[Screen capture showing a hidden secret value as part of an integration policy]
+// This graphic should be updated once a higher resolution version is available.
+
+. Click **Save integration**. The original secret value is overwritten in the policy.
+
 [discrete]
 [[agent-policy-scale]]
 == Policy scaling recommendations

@@ -15,8 +15,6 @@ prevention. The {agent} can be deployed in two different modes:
 
 ** *Standalone mode* -- All policies are applied to the {agent} manually as a YAML file. This is intended for more advanced users.
 See <<install-standalone-elastic-agent>> for more information.
-+
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
 
 The method you use depends on your use case, which features you need, and
 whether you want to centrally manage your agents.
@@ -90,8 +88,8 @@ NOTE: {elastic-defend} and APM Server have a different output matrix.
 
 |Kafka
 |{y}
-|Under consideration
-|Under consideration
+|{y} (beta)
+|{y} (beta)
 
 |Redis
 |{y}

@@ -1,5 +1,5 @@
 [[elastic-agent-ssl-configuration]]
-= Configure SSL/TLS
+= Configure SSL/TLS for standalone {agent}s
 
 ++++
 <titleabbrev>SSL/TLS</titleabbrev>

@@ -9,4 +9,22 @@ There are two different ways to use autodiscovery:
 
 * <<conditions-based-autodiscover>>
 
-* <<hints-annotations-autodiscovery>>
+* <<hints-annotations-autodiscovery>>
+
+
+[discrete]
+== How to configure autodiscovery
+
+`Conditions Based Autodiscovery` is more suitable for scenarios when users know the different group of containers they want to monitor in advance. It is advisable to choose conditions-based configuration when administrators can configure specific conditions that match their needs. Conditions are supported in both Managed and Standalone {agent}.
+
+`Hints Based Autodiscovry` is suitable for more generic scenarios, especially when users don't know the exact configuration of the system to monitor and can not create in advance conditions. Additionally a big advantage of Hints Autodiscovery is the ability to offer dynamic configuration of inputs based on annotations from Pods/Containers. If dynamic configuration is needed, then Hints should be enabled. Hints are supported only in Standalone {agent} mode.
+
+*Best Practises when you configure autodiscovery:*
+
+- Always define alternatives and default values to your variables that are used in conditions or [hint templates](eg. See `auth.basic` set as `auth.basic.user: ${kubernetes.hints.nginx.access.username|kubernetes.hints.nginx.username|''}`` in [nginx.yml](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/nginx.yml#L8))
+
+IMPORTANT: When an input uses a variable substitution that is not present in the current key/value mappings being evaluated, the input is removed in the result. (See more information in <<dynamic-input-configuration>>)   
+
+- To debug configurations that include variable substitution and conditions, use the inspect command of {agent}. (See more information in <<dynamic-input-configuration>> in *Debugging* Section)
+
+- In Condition Based autodiscovery is advisable to define a generic last condition that will act as your default condition and will be validated when all others fail or don't apply. If applicable, such conditions might help to identify processing and troubleshoot possible problems.
@@ -83,6 +83,64 @@ The stream to use for logs collection, for example, stdout/stderr.
 
 If the specified package has no logs support, a generic container's logs input will be used as a fallback. See the `Hints autodiscovery for kubernetes log collection` example below.
 
+[float]
+=== `co.elastic.hints/processors`
+
+Define a processor to be added to the  input configuration. See <<elastic-agent-processor-configuration>> for the list of supported processors.
+
+If the processors configuration uses list data structure, object fields must be enumerated. For example, hints for the rename processor configuration below
+
+[source,yaml]
+----
+processors:
+  - rename:
+      fields:
+        - from: "a.g"
+          to: "e.d"
+      fail_on_error: true
+----
+
+will look like:
+
+[source,yaml]
+----
+co.elastic.hints/processors.rename.fields.0.from: "a.g"
+co.elastic.hints/processors.rename.fields.1.to: "e.d"
+co.elastic.hints/processors.rename.fail_on_error: 'true'
+----
+
+If the processors configuration uses map data structure, enumeration is not needed. For example, the equivalent to the `add_fields` configuration below
+
+[source,yaml]
+----
+processors:
+  - add_fields:
+      target: project
+      fields:
+        name: myproject
+----
+
+is
+
+[source,yaml]
+----
+co.elastic.hints/processors.1.add_fields.target: "project"
+co.elastic.hints/processors.1.add_fields.fields.name: "myproject"
+----
+
+In order to provide ordering of the processor definition, numbers can be provided. If not, the hints builder will do arbitrary ordering:
+
+[source,yaml]
+----
+co.elastic.hints/processors.1.dissect.tokenizer: "%{key1} %{key2}"
+co.elastic.hints/processors.dissect.tokenizer: "%{key2} %{key1}"
+----
+
+In the above sample the processor definition tagged with `1` would be executed first.
+
+IMPORTANT: Processor configuration is not supported on the datastream level, so annotations like `co.elastic.hints/<datastream>.processors` are ignored.
+
+
 [discrete]
 == Available packages that support hints autodiscovery
 
@@ -280,6 +338,46 @@ providers:
     scope: node
 ----
 
+[discrete]
+=== Hints autodiscovery for kubernetes logs with JSON decoding
+
+Based on the previous example, users might want to perform extra processing on specific logs, for example to decode specific fields containing JSON strings. Use of <<decode-json-fields>> is advisable as follows:
+
+You need to have enabled hints autodiscovery, as described in the previous `Hints autodiscovery for Kubernetes log collection` example.
+
+The pod that will produce JSON logs needs to be annotated with:
+
+[source,yaml]
+----
+
+ annotations:
+        co.elastic.hints/package: "container_logs"
+        co.elastic.hints/processors.decode_json_fields.fields: "message"
+        co.elastic.hints/processors.decode_json_fields.add_error_key: 'true'
+        co.elastic.hints/processors.decode_json_fields.overwrite_keys: 'true'
+        co.elastic.hints/processors.decode_json_fields.target: "team"
+----
+
+> NOTE: These parameters for the `decode_json_fields` processor are just an example.
+
+The following log entry:
+
+[source,json]
+----
+{"myteam": "ole"}
+----
+
+Will produce both fields: the original `message` field and also the target field `team`.
+
+[source,json]
+----
+
+"team": {
+      "myteam": "ole"
+    },
+
+"message": "{\"myteam\": \"ole\"}",
+----
 
 [discrete]
 == Troubleshooting

@@ -1,8 +1,6 @@
 [[create-standalone-agent-policy]]
 = Create a standalone {agent} policy
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 To get started quickly, use {kib} to add integrations to an agent policy, then
 download the policy to use as a starting point for your standalone {agent}
 policy. This approach saves time, is less error prone, and populates the

@@ -1,11 +1,10 @@
 [[elastic-agent-configuration]]
 = Configure standalone {agent}s
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 TIP: To get started quickly, use {kib} to create and download a standalone
 policy file. You'll still need to deploy and manage the file, though. For more
-information, refer to <<create-standalone-agent-policy>>.
+information, refer to <<create-standalone-agent-policy>> or try out our example: 
+<<example-standalone-monitor-nginx,Use standalone {agent} to monitor nginx>>.
 
 Standalone {agent}s are manually configured and managed locally on the systems
 where they are installed. They are useful when you are not interested in

@@ -5,8 +5,6 @@
 <titleabbrev>Monitoring</titleabbrev>
 ++++
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 {agent} monitors {beats} by default. To turn off or change monitoring
 settings, set options under `agent.monitoring` in the `elastic-agent.yml` file.
 

@@ -0,0 +1,56 @@
+[[elastic-agent-standalone-download]]
+= Configure download settings for standalone {agent} upgrades
+
+++++
+<titleabbrev>Agent download</titleabbrev>
+++++
+
+The `agent.download` section of the elastic-agent.yml config file contains settings for where to download and store artifacts used for {agent} upgrades.
+
+[[elastic-agent-standalone-download-settings]]
+.{agent} download settings
+[cols="2*<a"]
+|===
+| Setting | Description
+
+|
+[[agent.download.sourceURI]]
+`sourceURI`
+
+| (string) Path to the location of artifacts used during {agent} upgrade.
+
+// =============================================================================
+
+|
+[[agent.download.target_directory]]
+`target_directory`
+
+| (string) Path to the directory where download artifacts are stored.
+
+// =============================================================================
+
+|
+[[agent.download.timeout]]
+`timeout`
+
+| (string) The HTTP request timeout in seconds for the download package attempt.
+
+// =============================================================================
+
+|
+[[agent.download.install_path]]
+`install_path`
+
+| (string) The location of installed packages and programs, as well as program specifications.
+
+// =============================================================================
+
+|
+[[agent.download.retry_sleep_init_duration]]
+`retry_sleep_init_duration`
+
+| (string) The duration in seconds to sleep for before the first retry attempt.
+
+// =============================================================================
+
+|===
@@ -5,8 +5,6 @@
 <titleabbrev>Feature flags</titleabbrev>
 ++++
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 The Feature Flags section of the elastic-agent.yml config file contains settings in {agent} that are disabled by default. These may include experimental features, changes to behaviors within {agent} or its components, or settings that could cause a breaking change. For example a setting that changes information included in events might be inconsistent with the naming pattern expected in your configured {agent} output.
 
 To enable any of the settings listed on this page, change the associated `enabled` flag from `false` to `true`.

@@ -5,8 +5,6 @@
 <titleabbrev>Logging</titleabbrev>
 ++++
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 The Logging section of the `elastic-agent.yml` config file contains settings for configuring the logging output.
 The logging system can write logs to the `syslog`, `file`, `stderr`, `eventlog`, or rotate log files.
 If you do not explicitly configure logging, the `stderr` output is used.

@@ -1,8 +1,9 @@
 [[elastic-agent-input-configuration]]
-= Configure inputs for Standalone {agent}s
-
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
+= Configure inputs for standalone {agent}s
 
+++++
+<titleabbrev>Inputs</titleabbrev>
+++++
 
 The `inputs` section of the `elastic-agent.yml` file specifies how {agent} locates and processes input data.
 

@@ -5,8 +5,6 @@
 <titleabbrev>Outputs</titleabbrev>
 ++++
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 The `outputs` section of the `elastic-agent.yml` file specifies where to
 send data. You can specify multiple outputs to pair specific inputs with
 specific outputs.

@@ -7,8 +7,6 @@
 <titleabbrev>{es}</titleabbrev>
 ++++
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 The {es} output sends events directly to {es} by using the {es} HTTP API.
 
 *Compatibility:* This output works with all compatible versions of {es}. See the

@@ -7,8 +7,6 @@
 <titleabbrev>{ls}</titleabbrev>
 ++++
 
-include::{fleet-repo-dir}/standalone-note.asciidoc[]
-
 The {ls} output uses an internal protocol to send events directly to {ls} over
 TCP. {ls} provides additional parsing, transformation, and routing of data
 collected by {agent}.

@@ -48,8 +48,8 @@ escaping.
 [id="{type}-worker-setting"]
 `worker`
 
-| (int) The number of workers per configured host publishing events to
-{output-type}. This is best used with load balancing mode enabled. Example: If
+| (int) The number of workers per configured host publishing events. 
+This is best used with load balancing mode enabled. Example: If
 you have two hosts and three workers, in total six workers are started (three
 for each host).