Releases: alibaba/loongcollector
loongcollector-alpha
Loongcollector Alpha 版发布
经过几个月的努力与技术演进,现在 LoongCollector Alpha 版( 0.2 )发布,我们诚挚邀请广大开发者下载试用,并希望基于实际使用体验向我们提供宝贵的反馈意见。感谢您的支持!
LoongCollector 源自阿里云可观测性团队所开源的 iLogtail 项目,在继承了 iLogtail 强大的日志采集与处理能力的基础上,进行了全面的功能升级与扩展。从原来单一日志场景,逐步扩展为可观测数据采集、本地计算、服务发现的统一体。
LoongCollector 是一款集卓越性能、超强稳定性和灵活可编程性于一身的数据采集器,专为构建下一代可观测 Pipeline 设计。愿景是:打造业界领先的“统一可观测 Agent(Unified Observability Agent)”与“端到端可观测 Pipeline(End-to-End Observability Pipeline)”。
新特性
- 从日志到全面数据可观测
相对于 iLogtail 专注于日志采集,LoongCollector 将实现采集能力的全面升级,做到可观测采集的 OneAgent 化。
Input 插件(含文档) | 描述 | 状态(含提案,已发布的可不提供) |
---|---|---|
input_container_stdio | 高性能、高可靠容器标准输出。 | 已发布 |
service_kubernetes_meta | 定时采集Kubernetes元数据,包括Pod、Deployment等资源及其之间的关系。 | 已发布 |
input_prometheus | 高性能 Prometheus 抓取。 | 开发中 Prometheus采集能力提案 |
input_file_security input_process_security input_network_security | 基于 eBPF 采集文件、进程、网络相关安全事件。 | 开发中 |
input_network_observer | 基于 eBPF 采集网络可观测数据。 | 已发布 网络可观测提案链接 |
主机指标 | 定时采集主机系统指标(CPU、内存等)和主机、进程元数据(工作目录、进程语言等) | 开发中 主机指标提案链接 |
- 性能与稳定性
-
K8s 标准输出采集性能、稳定性提升(已发布)
-
通过重构标准输出采集插件,推出了新版插件 input_container_stdio,该插件支持日志轮转队列,显著增强了标准输出采集的稳定性。
-
在性能方面,新插件表现优异,在 containerd 和 Docker 环境下,单行日志的平均采集速度分别达到了 250MB/s 和 150MB/s,相比旧版新版插件 service_docker_stdout,提升了至少100%。
-
-
日志典型场景:极简模式、多行切分性能提升 (已发布)
-
通过 Event 池化与内存零拷贝等手段,对框架组件的性能进行调优。
-
单行日志与多行日志的平均采集性能分别达到 300MB/s 和 125MB/s,单行日志采集相比 iLogtail 提升了100%。
-
-
基础稳定性 (已发布)
-
支持采集配置热加载隔离,避免了单个采集配置变更导致其它采集配置短时暂停。
-
C++ 处理、发送队列实现 Pipeline 级隔离。
-
- 更灵活的编程管道
-
C++ 全面插件化(已发布):同时提供了充足的组件可供插件自由组合,极大地方便社区新增高性能的输入和输出能力,C++原生插件开发指南详见如何开发原生Input插件和如何开发原生Flusher插件。
-
C++ Input 可使用原生 Processor(已发布):C++ Input插件能够与原生及扩展的Processor插件配合使用,并支持SPL插件。这意味着C++ Input插件不仅可以利用原生Processor提供的高性能来解析日志,还能通过丰富的扩展Processor功能进一步处理日志,具体详情请参阅文档处理插件和输入插件。
-
Golang Input 可使用原生 Processor (开发中):Go Input 支持多种灵活的数据源输入,而原生处理插件提供了高性能的数据处理。结合两者的优势,可以构建出既高效又能适应多种应用场景的数据处理流水线。详见 Issue。
-
SPL 处理模式(已发布):SPL 处理模式支持用户通过 SPL 语句实现对数据的处理。无需编写代码开发插件,极大地拓展了 LoongCollector 可应用的场景。详见文档 SPL 处理。
-
协议修订:约定注册、心跳、配置拉取、状态上报等核心管控流程。
-
功能增强:支持心跳数据压缩,支持配置结果反馈上报,使用标志位支持功能兼容扩展。
-
管控范围:从采集配置管控扩展为通用常驻任务、一次性任务和进程级配置。
-
整体进展:协议完成、客户端支持、服务端开发中。 [Discussion] ConfigServer v2 Implementation
iLogtail 开源初期有较多历史包袱,造成了工作目录结构不清晰。LoongCollector 专门进行了目录结构优化,将配置、日志、运行数据等进行分离。
- 自监控全面优化,更完整、清晰展示 Loongcollector 自身运行状态
针对 Loongcollector 的自监控指标,进行了全面的优化改进,使得自身运行状态的展示更加完整和清晰。用户可以直观地了解到各项指标的实时变化,快速定位潜在问题。此外,通过合理的数据分类和分级展示,用户可以更方便地进行系统调优和资源管理,从而提高整体运行效率。我们将自监控指标改造为了内置 Pipeline,可以将自监控指标自定义发送到不同的位置。
- 全新的官网 (已发布)
鉴于 gitbook 网络不稳定,将官网移植到“可观测中文社区”,以便享受更便捷的文档服务。
可观测中文社区作为一个以“运维可观测”为核心的开放、包容、分享的技术社区,旨在聚集运维专家、开发者和爱好者,共同探讨、学习和分享可观测最佳实践与最新技术。
限制说明
-
eBPF 相关 Pipeline 支持的最低内核版本为 4.19,支持的架构为 x86 64 位,操作系统支持 Linux。
-
Prometheus 指标数据采集依赖 Operator 提供的 TargetAllocator 能力,暂未开源。
-
Windows 版暂未推出,敬请期待。
不兼容变更说明
-
文件目录布局与文件命名跟 iLogtail 2.0 版本有所变化,如果某些环境对特定目录、文件有所依赖,则需要注意该变化。
-
部分自监控指标命名跟 iLogtail 2.0 版本不一致,LoongCollector 重新规范了所有自监控指标的命名和上报方式。
-
开发镜像升级,新增部分依赖库。使用 iLogtail 开发镜像开发 Loongcollector 会出现依赖库链接错误。建议使用loongcollector 开发镜像 2.0.5 版本及以上。
版本发布时间
2024.11.27
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
loongcollector-0.2.0.linux-amd64.tar.gz | Linux | x86-64 | loongcollector-0.2.0.linux-amd64.tar.gz.sha256 |
loongcollector-0.2.0.linux-arm64.tar.gz | Linux | arm64 | loongcollector-0.2.0.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/loongcollector-community-edition/loongcollector:0.2.0
loongcollector-0.1.0
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
loongcollector-0.1.0.linux-amd64.tar.gz | Linux | x86-64 | loongcollector-0.1.0.linux-amd64.tar.gz.sha256 |
loongcollector-0.1.0.linux-arm64.tar.gz | Linux | arm64 | loongcollector-0.1.0.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/loongcollector-community-edition/loongcollector:0.1.0
v2.0.7 release
2.0.7
Changes
All issues and pull requests are here.
Features
Fixed
- [public] [both] [fixed] fix: JSON truncation caused by escaped zero byte (#1594) (#1596)
- [public] [both] [fixed] fix core caused by concurrent use of non-thread-safe gethostbyname (#1611)
- [public] [both] [fixed] Fix issue that guage metric miss labels (#1618)
- [public] [both] [fixed] recover readers exactly from checkpoint (#1620) (#1635)
- [public] [both] [fixed] fix checkpoint dump idx in reader array (#1638)
- [public] [both] [fixed] fix: GTID Truncation Issue and Improve Consistency in Checkpoint Management (#1648)
- [public] [both] [fixed] suicide immediately when resource hard limit is met (#1649)
Doc
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-2.0.7.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-2.0.7.linux-amd64.tar.gz.sha256 |
ilogtail-2.0.7.linux-arm64.tar.gz | Linux | arm64 | ilogtail-2.0.7.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.7
What's Changed
- 2.0 hotfix: Resolve intermittent crash due to zero-byte reads in log reading by @Abingcbc in #1501
- add timeout warning by @Abingcbc in #1506
- fix: Fixed the issue of missing Profile data when the Pipeline contains a Golang plugin. by @linrunqi08 in #1521
- fix timeout force read when still can read by @Abingcbc in #1551
- backport some feature from main to 2.0 by @linrunqi08 in #1553
- [backport] fix: disable nanosecond timestamp when only EnableTimestampNanosecond is enabled (#1528) by @henryzhx8 in #1552
- [backport] fix: JSON truncation caused by escaped zero byte (#1594) by @yyuuttaaoo in #1596
- [backport] recover readers exactly from checkpoint by @Abingcbc in #1635
- [backport] fix checkpoint dump idx in reader array by @Abingcbc in #1638
- [backport 2.0] fix: GTID Truncation Issue and Improve Consistency in Checkpoint Management by @yyuuttaaoo in #1648
Full Changelog: v2.0.4...v2.0.7
v2.0.4 release
Changes
All issues and pull requests are here.
Features
- [public] [both] [added] Improve sender log condition to record valuable debug info (#1359)
- [public] [both] [added] wait for read flush timeout instead of flushing last line immediately on file deletion (#1418)
- [public] [both] [added] refactor logic of last matched line (#1426)
- [public] [both] [added] test: add a case to test timeformat without year (#1432)
- [public] [both] [added] Update chatgroup image (#1445)
- [public] [both] [added] refactory: enable move for SourceBuffer (#1457)
- [public] [both] [added] summer-ospp-2024 (#1461)
- [public] [both] [added] default enable spl (#1475)
- [public] [both] [added] allow empty endpoint in enterprise flusher_sls to avoid unsuccessful load of config (#1486)
Fixed
- [public] [both] [fixed] fix dockershim cause containerd invalid (#1424)
- [public] [both] [fixed] fix: correct pipeline version recognition issue (#1427)
- [public] [both] [fixed] fix: refactor multiline split state (#1410)
- [public] [both] [fixed] fix: update base image to resolve 'Illegal instruction' error on old CPUs (#1429)
- [public] [both] [fixed] ensure config name in golang config and ctx matching C++ config name (#1437)
- [public] [both] [fixed] fix missing user agent info (#1444)
- [public] [both] [fixed] fix plugin alarm with default version (#1443)
- [public] [both] [fixed] fix: make LOGTAIL_LOG_LEVEL env variable effective for ilogtail.LOG (#1440)
- [public] [both] [fixed] Fix parameter errors for container metadata preview. (#1430)
- [public] [both] [fixed] fix: Ensure alarms raised during collection for profile projects are reported (#1455)
- [public] [both] [fixed] fix missing uuid and incorrect instance id (#1476)
Doc
- [public] [both] [doc] document: fix document for flusher pulsar (#1460)
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-2.0.4.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-2.0.4.linux-amd64.tar.gz.sha256 |
ilogtail-2.0.4.linux-arm64.tar.gz | Linux | arm64 | ilogtail-2.0.4.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4
v1.8.8 release
1.8.8
Changes
All issues and pull requests are here.
Features
- [public] [both] [added] Processor rate limit #1321
Fixed
- [public] [both] [fixed] Fix apsara base field parse error #1307
- [public] [both] [fixed] Fix microsecond handling in Apsara log parser #1309
- [public] [both] [fixed] Resolve unintended Go module loading caused by high-precision timestamp settings #1336
- [public] [both] [fixed] Ensure process is able to exit when history file import is ever used #1369
- [public] [both] [fixed] Fix missing Connection_string tag in Skywalking database Span #1407
- [public] [both] [fixed] Maintain sequential info of the original log content and allow multivalue for single key for apsara processor #1416
- [public] [both] [fixed] Improve sender log condition to record valuable debug info #1419
Doc
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-1.8.8.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-1.8.8.linux-amd64.tar.gz.sha256 |
ilogtail-1.8.8.linux-arm64.tar.gz | Linux | arm64 | ilogtail-1.8.8.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.8.8
v2.0.0 release
2.0.0
iLogtail 2.0 is a completely new version with upgraded architecture, and its configuration files are not fully compatible with the 1.x version. For the upgrade guide, please refer to the change log #1294 and example configs.
The configuration method of version 2.0 perfectly supports cascading between C++ native plugins and cascading between C++ native plugins and Go plugins. At the same time, it introduces a more efficient data processing language, SLS Processing Language (SPL), making on-device data processing more effortless.
Changes
All issues and pull requests are here.
Features
- [public] [both] [added] add input and flusher interface #1184
- [public] [both] [added] complete pipeline definition #1185
- [public] [both] [added] support V2 config #1188
- [public] [both] [added] Support jsonline protocol #1265
- [public] [both] [added] Support SPL processor in ilogtail for enhanced data processing #1278
- [public] [both] [added] add UsingOldContentTag. When UsingOldContentTag is set to false, the Tag is now placed in the Meta instead of Logs during cgo #1169
- [public] [both] [added] Upgrade environment variable logic using new Pipeline API for logtail environment configuration management #1282
- [public] [both] [updated] improve source file structure #1182
- [public] [both] [updated] feat: add flusher_http queue buffer and async interceptor settings #1203
- [public] [both] [updated] Enhancing the vscode development experience #1219
- [public] [both] [updated] feat: add kafka net config MaxOpenRequests #1224
- [public] [both] [updated] feat: support labels for self_metric #1240
- [public] [both] [updated] Support contents only for loki flusher #1256
- [public] [both] [updated] Fix issue where container IP is occasionally empty for Kubernetes pods in HostNetwork mode #1280
- [public] [both] [updated] Improve goprofile plugin to support real local IP inclusion #1281
- [public] [both] [updated] Added a switch to control the loading of the processor_spl, which by default remains unloaded #1312
Fixed
- [public] [both] [fixed] fix: resolve crash by setting CURLOPT_NOSIGNAL for thread safety #1283
- [public] [both] [fixed] fix: Prevent crash due to concurrent map access in Go plugin log processing #1284
- [public] [both] [fixed] Fix delimiter parsing for logs with leading spaces #1289
- [public] [both] [fixed] Fix timezone handling in native processors #1293
- [public] [both] [fixed] Fix log content duplication in ParseJsonNativeProcessor #1296
- [public] [both] [fixed] fix memory leak in ProcessorParseDelimiterNative #1300
- [public] [both] [fixed] fix: reset checkpoint dump time after dir registration completes to avoid log duplication #1291
- [public] [both] [fixed] Fix timestamp parsing compatibility issue for apsara log #1285
- [public] [both] [fixed] Fix: Delete mSourceKeyOverwritten in processorParseApsaraNative to prevent concurrency issues #1305
- [public] [both] [fixed] Fix apsara base field parse error and microsecond handling in Apsara log parser #1308
Doc
- [public] [both] [doc] update docs related to V2 config #1287
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-2.0.0.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-2.0.0.linux-amd64.tar.gz.sha256 |
ilogtail-2.0.0.linux-arm64.tar.gz | Linux | arm64 | ilogtail-2.0.0.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.0
v1.8.4 release
1.8.4
Changes
All issues and pull requests are here.
Features
- [public] [both] [updated] Improve goprofile plugin to support real local IP inclusion #1281
Fixed
- [public] [both] [fixed] fix panic when k8s_event.GetEventRecorder() is nil #1264
- [public] [both] [fixed] When REGEX_LOG is set, mTimeKey will be set to "time". #1272
- [public] [both] [fixed] [backport]fix: resolve crash by setting CURLOPT_NOSIGNAL for thread safety #1286
- [public] [both] [fixed] [backport]Fix delimiter parsing for logs with leading spaces #1288
- [public] [both] [fixed] [backport]Fix timezone handling in native processors #1293
- [public] [both] [fixed] [backport]Fix log content duplication in ParseJsonNativeProcessor #1295
- [public] [both] [fixed] [backport]fix memory leak in ProcessorParseDelimiterNative #1299
- [public] [both] [fixed] [backport] fix: reset checkpoint dump time after dir registration completes to avoid log duplication #1301
- [public] [both] [fixed] [backport]Fix timestamp parsing compatibility issue for apsara log #1302
- [public] [both] [fixed] Fix: remove content if native processor parse failed #1304
- [public] [both] [fixed] Fix issue where container IP is occasionally empty for Kubernetes pods in HostNetwork mode #1280
Doc
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-1.8.4.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-1.8.4.linux-amd64.tar.gz.sha256 |
ilogtail-1.8.4.linux-arm64.tar.gz | Linux | arm64 | ilogtail-1.8.4.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.8.4
v1.8.3 release
1.8.3
Changes
All issues and pull requests are here.
Features
Fixed
- [public] [both] [fixed] fix log duplication problem arised from #1216
- [public] [both] [fixed] fix plugin crash due to nil in container info
- [public] [both] [fixed] resolve issue with ProcessorParseDelimiterNative capturing next line data
- [public] [both] [fixed] correct premature file read termination triggered by sender back pressure
- [public] [both] [fixed] implement panic recovery mechanism in plugin export
- [public] [both] [fixed] resolve crash issue during Apsara log parsing
- [public] [both] [fixed] correct parsing errors in ProcessorParseApsaraNative with large buffer input
Doc
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-1.8.3.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-1.8.3.linux-amd64.tar.gz.sha256 |
ilogtail-1.8.3.linux-arm64.tar.gz | Linux | arm64 | ilogtail-1.8.3.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.8.3
v1.8.1 release
1.8.1
Changes
All issues and pull requests are here.
Features
Fixed
- [public] [both] [fixed] fix prometheus send wrong sls logs when reuse string memory
- [public] [both] [fixed] fix dropping jvm metrics when collecting multiple skywalking instances's data
- [public] [both] [fixed] fix elasticsearch flusher authentication tls config and http config
- [public] [both] [fixed] fix profiling wrong type when the different profiling type having same stack.
- [public] [both] [fixed] fix send local buffer failed when upgrade iLogtail from version earlier than 1.3.
- [public] [both] [fixed] change strptime_ns to parse %c format from "%x %X" to "%a %b %d %H:%M:%S %Y" for consistent behavior with striptime.
- [public] [both] [fixed] fix topic key does not support underscore.
- [public] [both] [fixed] fix jmxfetch status error when exist multi jmxfetch config in the same machine.
- [public] [both] [fixed] fix increasing WSS memory issue in collected containers.
- [public] [both] [fixed] fix cannot log blacklist config error
Doc
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-1.8.1.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-1.8.1.linux-amd64.tar.gz.sha256 |
ilogtail-1.8.1.linux-arm64.tar.gz | Linux | arm64 | ilogtail-1.8.1.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.8.1
v1.8.0 release
1.8.0
Changes
All issues and pull requests are here.
Features
- [public] [both] [added] refactoried C++ process pipeline
- [public] [both] [added] support use accelerate processors with go processors
- [public] [both] [added] add new logtail metric module
- [public] [both] [added] ddd JSON flatten protocol, data can be flattened and then brushed into storage such as Kafka and ES.
- [public] [both] [added] use env
LOGTAIL_LOG_LEVEL
to control ilogtail log level - [public] [both] [added] add global host path blacklist
- [public] [both] [updated] support continue/end regex patterns to split multiline log
- [public] [both] [updated] support reader flush timeout
- [public] [both] [updated] flusher Kafka V2: support send the message with headers to kafka
- [public] [both] [updated] sls flusher supports nanosecond timestamp
- [public] [both] [updated] update gcc version to 9.3.1
- [public] [both] [updated] add make flag WITHOUTGDB
- [public] [both] [updated] cache incomplete line in memory to avoid repeated read system call
- [public] [both] [updated] unify metrics format
- [public] [both] [updated] verify host ip against NIC to get more accurate host ip
- [public] [both] [added] support plugin ProcessorParseTimestampNative
- [public] [both] [added] support plugin processor_otel_metric
- [public] [both] [updated] skywalking plugin support to capture
db.connection_string
tag - [public] [both] [added] support plugin ProcessorParseApsaraNative
- [public] [both] [added] support plugin ProcessorParseDelimiterNative
- [public] [both] [added] support plugin ProcessorFilterNative
- [public] [both] [added] support plugin ProcessorDesensitizeNative
- [public] [both] [added] add resource tag for env created logtail config
Fixed
- [public] [both] [fixed] remove the ip restrict of config server address, allow domain and hostname
- [public] [both] [fixed] add APSARA_LOG_TRACE to solve the problem of not being able to find LOG_TRACE.
- [public] [both] [fixed] fix collection duplication for statefulset sidecar that may mount same volume on different nodes
- [public] [both] [fixed] fix multiline is splitted if not flushed to disk together
- [public] [both] [fixed] fix line is truncated if \0 is in the middle of line
- [public] [both] [fixed] container cannot exit for file reopened by checkpoint
- [public] [both] [fixed] fix filename being mismatched to the deleted file if the deleted file size is 0 and their inode is same
- [public] [both] [fixed] fix config server panic caused by concurrent read and write shared object
- [public] [both] [fixed] add expand array switch in json processor for backward compatibility
- [public] [both] [fixed] timezone adjust not working with apsara_log
- [public] [both] [fixed] fix parse json without newline may result in partial parsing
- [public] [both] [fixed] fix json does not split correctly if beginning buffer contains invalid json
Doc
Download
Filename | OS | Arch | SHA256 Checksum |
---|---|---|---|
ilogtail-1.8.0.linux-amd64.tar.gz | Linux | x86-64 | ilogtail-1.8.0.linux-amd64.tar.gz.sha256 |
ilogtail-1.8.0.linux-arm64.tar.gz | Linux | arm64 | ilogtail-1.8.0.linux-arm64.tar.gz.sha256 |
Docker Image
Docker Pull Command
docker pull sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.8.0