Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

calyptia: switch to YAML for Fleet config #9698

Open
wants to merge 56 commits into
base: master
Choose a base branch
from

Conversation

patrick-stephens
Copy link
Contributor

@patrick-stephens patrick-stephens commented Dec 9, 2024

Updates to using YAML for Fleet configuration now it is completed in 3.2 series.
We have added this as a configurable option now with the default set to the current TOML to keep backwards behaviour - users can opt-in to YAML format and then we can also switch the default in the future.

fleet_config_legacy_format on == current INI/TOML format and the default
fleet_config_legacy_format off == YAML format

In each case note the initial configuration will be assumed to be TOML, not changing installation and packaging for that.

Includes changes from cprofiles to resolve macOS failures, these should not cause an issue once that PR is merged: fluent/cprofiles#7
Resolves the usage of bool type in the plugin and replaces with standard int flag approach.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[INPUT]
    name cpu
    tag  cpu.local

    # Read interval (sec) Default: 1
    interval_sec 1

[CUSTOM]
    name calyptia
    API_Key ***
    Fleet_Name yaml-fleet
    fleet_config_legacy_format off

[OUTPUT]
    name  null 
    match *
  • Debug log output from testing the change
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [debug] [input:calyptia_fleet:calyptia_fleet.2] parsing configuration timestamp from path: 1733416866.yaml
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [ info] [input:calyptia_fleet:calyptia_fleet.2] loading configuration from /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/new.yaml.
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [ info] [input:calyptia_fleet:calyptia_fleet.2] changing to config dir: /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [ info] [input:calyptia_fleet:calyptia_fleet.2] changing to config dir: /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866
...
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [custom:calyptia:calyptia.0] read UUID (334583af-cb56-4a5f-bdfa-8b4ab3f5896b) from file: /tmp/calyptia-fleet/machine-id.conf
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [custom:calyptia:calyptia.0] custom initialized!
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [simd    ] disabled
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [cmetrics] version=0.9.9
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [ctraces ] version=0.5.7
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:dummy:dummy.0] initializing
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [dummy:dummy.0] created event channels: read=20 write=21
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:fluentbit_metrics:fluentbit_metrics.1] initializing
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:fluentbit_metrics:fluentbit_metrics.1] storage_strategy='memory' (memory only)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [fluentbit_metrics:fluentbit_metrics.1] created event channels: read=22 write=25
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] initializing
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] storage_strategy='memory' (memory only)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [calyptia_fleet:calyptia_fleet.2] created event channels: read=26 write=27
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] initializing calyptia fleet input.
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [input:calyptia_fleet:calyptia_fleet.2] initial collector interval: sec=-1 nsec=-1
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] invalid interval settings, using defaults: sec=15 nsec=0
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [input:calyptia_fleet:calyptia_fleet.2] we are already using a configuration file: /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/new.yaml
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [input:calyptia_fleet:calyptia_fleet.2] parsing configuration timestamp from path: 1733416866.yaml
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] fleet collector initialized with interval: 15 sec 0 nsec
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [stdout:stdout.0] created event channels: read=28 write=29
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [calyptia:calyptia.1] created event channels: read=37 write=38
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [output:calyptia:calyptia.1] machine_id=dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [output:stdout:stdout.0] worker #0 started
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [http_client] not using http_proxy for header
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [ info] [output:calyptia:calyptia.1] connected to Calyptia, agent_id='100ddda4-d2e1-4edb-8a24-a442fa253219'
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [ info] [output:calyptia:calyptia.1] agent registration successful
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule dummy.0:stdout.0
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule fluentbit_metrics.1:stdout.0
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule fluentbit_metrics.1:calyptia.1
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule calyptia_fleet.2:stdout.0
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [ info] [sp] stream processor started
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753221.654021075, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [out flush] cb_destroy coro_id=0
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753222.654082768, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [out flush] cb_destroy coro_id=1
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753223.654070928, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [out flush] cb_destroy coro_id=2
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753224.654117339, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [out flush] cb_destroy coro_id=3
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:06] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:06] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:06] [debug] [task] created direct task=0x70c01da36e60 id=1 OK
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753225.654339922, {}], {"message"=>"custom dummy"}]

Confirmed all fleet config files are YAML:

$ sudo find /tmp/calyptia-fleet/
/tmp/calyptia-fleet/
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/cur.yaml
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/header.yaml
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866.yaml
/tmp/calyptia-fleet/machine-id.conf
$ sudo cat /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866.yaml
includes: 
    - /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/header.yaml
pipeline:
    inputs:
        - name: dummy
          dummy: '{"message": "custom dummy"}'
    outputs:
        - name: stdout
          match: '*'
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@patrick-stephens patrick-stephens added the ok-package-test Run PR packaging tests label Dec 9, 2024
Signed-off-by: Patrick Stephens <[email protected]>
@patrick-stephens

This comment was marked as outdated.

edsiper and others added 22 commits December 19, 2024 15:53
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
This change adds support to parse and register the resources and scope metadata
coming from Logs which are ingested from a JSON payload.

Signed-off-by: Eduardo Silva <[email protected]>
the new configuration option 'data_mode' allows to control what type
of data the plugin pass to the callback set by the caller. By default
it was always passing one log record at a time in the configured 'format'; now
with 'data_mode' the user can set 'single_record' (default) or 'chunk'.

When the option 'chunk' is passed, the whole binary chunk is passed to the callback
as a reference (it's up to the caller to unpack and validate the content)

Signed-off-by: Eduardo Silva <[email protected]>
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](helm/kind-action@v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* workflows: resolve ubuntu 24 packaging problems

Signed-off-by: Patrick Stephens <[email protected]>

* workflows: remove unavailable macOS 12 runners

Signed-off-by: Patrick Stephens <[email protected]>

---------

Signed-off-by: Patrick Stephens <[email protected]>
@patrick-stephens
Copy link
Contributor Author

Ah rebase messed up, let me fix it up...

@niedbalski
Copy link
Collaborator

@patrick-stephens can you squash into smaller amount of commits?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants