forked from rsyslog/rsyslog
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
15869 lines (15835 loc) · 894 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
----------------------------------------------------------------------------------------
Scheduled Release 8.2302.0 (aka 2023.02) 2023-02-??
- 2023-01-27: core/template: implement negative position.to
This will easily permit to drop the last n characters from a property
without the need to know the exact length of the string. This is
especially useful as the exact length is most often not known
beforehand.
- 2023-01-18: Introduce --enable-libcap-ng configure option
The option allows to drop the capabilities to only
the necessary set, to minimize security exposure in
case there was ever a mistake in a networking
plugin or some other input resource. Moreover, it adds
ability to change uid and gid while retaining the
previously specified capabilities.
Add ability to change uid and gid while retaining the
capabilities previously specified.
closes https://github.com/rsyslog/rsyslog/issues/4986
Thanks to Attila Lakatos for the patch.
- 2023-01-16:
- omfile: add action parameters "rotation.*"
Add new action parameters
- rotation.sizeLimit
- rotation.sizeLimitCommand
provide automatic output file rotation functionality feature-wise
equivalent to legacy $outchannel. This finally permits to use
this feature set in rscript.
- core substring function: enhancement and hardening
Now, length can have a negative value -n to denote that the
substring should be build between startpos and the character
-n chars from the end. This is a shortcut for stripping charactes
on "both ends" of the string. See doc for details on the enhanced
semantics.
Also, some hardening against invalid startpos and length has
been added.
- core bugfix: wrong type conversion in internal string class could lead to segfault
This could only happen with very unusually large strings
Thanks to Flos Lonicerae for the patch.
- QA: changed to CodeQL scanning on github as LGTM replacement
- bugfix: wrong version number on daily stable builds
- CI: use newer version of zookeeper (needed modernization)
- ffaup bugfix : memory corruption with concurrent workers
The ffaup function fails to work properly when it is used with multiple workers.
The faup_handler_t struct is not supposed to be shared between threads.
This may have caused memory corruptions and race conditions when used
inside of actions.
Thanks to Thibaud Cartegnie for the fix.
- openssl bugfix: undefined reference error on OpenSSL 1.1 or higher.
This could have prevented ossl components from being loaded/used.
- 2023-01-02: core bugfix: template system may generate invalid json
When
- a list template
- is created with option.jsonf="on"
- and the last list element is a property with onEmpty="skip"
- and that property is actually empty
invalid JSON is generated.
The JSON string in this case ends with ", " instead of "}\n". This
patch fixes the issue.
closes https://github.com/rsyslog/rsyslog/issues/5050
----------------------------------------------------------------------------------------
Scheduled Release 8.2212.0 (aka 2022.12) 2022-12-06
- 2022-12-05: testbench: make python http server based tests more reliable
Harden them against races during server port assignment. Prevents
testbench flakes.
- 2022-12-05: omprog bugfix: invalid status handling at called program startup
There is a bug when external program *startup* does not return "OK". This
can also lead to a misadressing with potentially a segfault (very unlikely).
Note that no problem exists once the initializiation phase of the external
program is finished and regular message transfer runs.
The problem basically is that for a startup failure, the control data for
that external program instance is freed on error. Unfortunately, that state
data is needed later on to detect a suspended instance. We now keep the control
data even on init failure (as we then need to do normal control options).
closes https://github.com/rsyslog/rsyslog/issues/4967
- 2022-11-29: testbench bugfix: wrong message injection object of instance 1
In some client-server test cases, messages are supposed to be injected into
the instance 2(client), but they are actually injected into instance 1(server),
which may lead to false negative results. This patch fixed it by replacing
'injectmsg' with 'injectmsg2', and dealt with some minor issues.
Thanks to Guodong Zhu for the patch.
- 2022-11-21: rsyslog.conf man page bugfix: description of selectors
Document historic difference to BSD syslog selectors.
- 2022-11-18: imtcp bugfix: legacy config directives did no longer work
Many "$InputTCPServer..." config directives did no longer work
and were completely ignored (e.g. "$InputTCPServerStreamDriverMode").
This was a regression from a08591be5d9 (May, 5th 2021).
closes https://github.com/rsyslog/rsyslog/issues/5021
- 2022-11-16: ksi bugfix: sending of too many signing requests fixed.
As there is a bug in libksi where too many signing requests may have bene sent
out the amount of signing requests will be limited by KSI module until the fix
is implemented.
Thanks to Taavi Valjaots for the patch.
- 2022-11-14: bugfix: prevent potential segfault when switchung to queue emergency mode
When switching to Disk queue emergency mode, we destructed the in-memory
queue object. Practice has shown that this MAY cause races during
destruction which themselfs can lead to segfault. For that reason, we
now keep the disk queueu object. This will keep some ressources,
including disk space, allocated. But we prefer that over a segfault.
After all, it only happens after a serious queue error when we are
already at the edge of hard problems.
see also: https://github.com/rsyslog/rsyslog/issues/4963
- 2022-11-08: ksi bugfix: Segmentation fault in async mode fixed
Thanks to Taavi Valjaots for the patch.
- 2022-11-02: imjournal: add second fallback to _COMM
If SYSLOG_IDENTIFIER is not present in the journal message,
then lookup the _COMM field, which stands for the name
of the process the journal entry originates from. This is
needed in order to be in compliance with the journalctl
output.
Thanks to Attila Lakatos for the patch.
- 2022-10-25: core bugfix: local hostname invalid if no global() config object given
The local hostname is invalidly set to "[localhost]" on rsyslog startup
if no global() config object is present in rsyslog.conf. Sending a HUP
corrects the hostname.
This is a regression from ba00a9f25293f
closes https://github.com/rsyslog/rsyslog/issues/4975
closes https://github.com/rsyslog/rsyslog/issues/4825
- 2022-10-25: testbench bugfix: fixed timing issue that sometimes lead to test failure
Timing caused a race in test tool sync and could lead to premature termination of
tools, which in turn caused test failure
----------------------------------------------------------------------------------------
Scheduled Release 8.2210.0 (aka 2022.10) 2022-10-18
- 2022-10-13: fix NetBSD build issue
On NetBSD, time_t has for a long time now been __int64_t.
On 32-bit CPUs, the compiler is not obliged to define
__sync_bool_compare_and_swap_8, so instead this ends up
as an undefined symbol when linking rsyslog. This makes
the code fall back to the pthread / locking method on these
systems, but at least lets the program build.
Thanks to Havard Eidnes for the patch.
- 2022-10-12: omrabbitmq: Add TLS support
Thanks to github user 21stcavenan for the patch.
- 2022-09-14: config: add "abortOnFailedQueueStartup" global config parameter
similiar to "abortONUncleanConfig", this parameter aborts rsyslog
when a queue has problems during startup. Some users perfer rsyslog
to terminate in this case. By default, nothing changes.
closes https://github.com/rsyslog/rsyslog/issues/4902
- 2022-09-07: cor bugfix: leak in helper function SetString
A part of rsyslog runtime, SetString(), had a small memory leak when a value was
assigned multiple times. While this could potentially consume larger amounts of
memory, this did not happen in practice. The reason is that multiple assignments
to the same object occur very seldom.
Thanks to github user seuzw930 for the patch.
closes: https://github.com/rsyslog/rsyslog/issues/4961
- 2022-09-07: core bugfix: correct local host name after config processing
rsyslog.conf may affect the host's local name. These changes were
so far only activated after the first HUP. This patch now ensures
that the configured local host name is applied correctly throughout
all processing, including early startup.
This patch causes a slight change of behaviour. However, the behaviour
was inconsitent before. Now it is consistent and according to the config.
Please note: this patch also exposes a global entry point via "regular"
dynamic loading as this makes things much easier to do. This is in-line
with ongoing simplification effort.
Finally, we also remove a CI test that we do no longer need because
the problem covered is now addressed differently and the original issue
can no longer occur.
closes https://github.com/rsyslog/rsyslog/issues/4975
- 2022-08-31: imtcp: add option notifyonconnectionopen
Add this both as module an input parameter. Complements already-existing
config param notifyonconnectionclose and mirrors the similar feature from
imptcp.
The module parameter acts as default, similarly to notifyonconnectionclose.
Note that in contrast to imptcp, we emit IP addresses and not host
names. This sticks with the traditional semantics of imtcp.
Note that we also fixed a mislading error message in the case when a
disallowed sender tried to connect.
Thanks to John Chivian for suggesting the addition.
- 2022-08-26: openssl TLS driver: add mechanism to include extra CA files parameter
This change allows to include extra CA files so that no "unable to get issuer
certificates" issue is obtained when using chained cert files. New parameter name is
"NetstreamDriverCAExtraFiles".
Thanks to Sergio Arroutbi for the patch.
closes: https://github.com/rsyslog/rsyslog/issues/4851
- 2022-08-19: fix compile issue with older gcc compilers
Thanks to Julien Thomas for the contribution.
----------------------------------------------------------------------------------------
Scheduled Release 8.2208.0 (aka 2022.08) 2022-08-09
- 2022-08-09: ksi bugfix: request cache size and send timeout issue fixed.
Async service send timeout is not configurable and request cache size is too
small to handle large amount of signing requests with small amount of permitted
requests per aggregation round. For example user with max_requests = 4 results
cache size 5 * max_requests or at least 256. When signing 300 log files cache
will be too small resulting several unsigned blocks. When signing 200 log file
cache will be adequate, but with rate of 4 signatures per second, it is only
possible to sign 4 * 10 blocks before all requests that are not sent out will
timeout.
Fix for the issue is to make send timeout configurable and make the size of the
cache depend on the value of send timeout. New configuration value
sig.block.signtimeout="time, s" introduced that defines the time window wherein
the block has to be signed. The size of the request cache is increased to
3 * max_requests * sign_timeout or at least 256.
Thanks to Taavi Valjaots for the patch.
- 2022-08-09: imjournal bugfix: segmentation fault in close journal
Thanks to github user t-feng for the patch.
- 2022-08-09: net subsystem: support sha256 for StreamDriverAuthMode="x509/fingerprint"
Thanks to github user codemaker219 for the patch.
- 2022-08-05: imfile bugfix: message loss/duplication when monitored file is rotated
When a to-be-monitored file is being rotated, some messages may be lost or
duplicated. In case of duplication, many file lines may be duplicated
depending on actual timing. The whole bug was primarily timing depenedent
in general. It most often was visible in practice when the monitored
file was very frequently rotated (we had some report with every few
seconds).
Note that while we try hard to not lose any messages, input file
rotation always has some loss potential. This is inevitable if
the monitored file is being truncated.
Also note that this bugfix affects imfile, only. It has nothing to do
and no relation to rsyslog output files being rotated on HUP.
closes: https://github.com/rsyslog/rsyslog/issues/4797
- 2022-08-05: ksi bugfix: optimize processing of signer queue to fix delays.
There is a worker queue where rsyslog KSI module collects events and signing
requests. When queue is processed thread is periodically put to sleep. Previous
implementation handles signature requests well but sleeps every time after
handling new file open / close event. When several log files are opened or
closed simultaneously process is significantly slowed down. Another issue is
that thread always sleeps 1000ms that may be 2x longer than aggregation round.
This slows down overall signing process.
Fix for the issue is to simply not sleep after file open / close event if there
are next items to be processed. To speed up the signing process, rsyslog uses
KSI aggregator conf. to obtain the aggregation period that is used for the sleep
time configuration.
Thanks to Taavi Valjaots for the patch.
- 2022-08-04: ksi bugfix: possible crash fixed when several log files are opened.
KSI module in async mode used to request aggregator conf. every time a log
file was opened. When several log files were opened simultaneously
corresponding amount of pointless concurrent conf. requests were posted.
Concurrent conf. requests lead to a bug in libksi, where internal count of
pending requests was not decremented correctly causing system to crash.
Fix for the issue is to optimize the frequency of conf. requests so that only
one conf. requests is handled at once. Instead of checking conf. every time
log file is opened, conf is requested periodically after conf timeout. This will
affect both sync and async mode.
New option for KSI module introduced - sig.confinterval="time, s".
Thanks to Taavi Valjaots for the patch.
- 2022-08-04: openssl: add support to split tls commands by semicolon
- Add support to split tls commands by semicolon.
- Changed one test with multiple tls commands to use semicolon as
separator instead of newline.
closes: https://github.com/rsyslog/rsyslog/issues/4852
- 2022-08-04: openssl subsystem bugfix: build issue on Solaris
Needed header file was added. Platforms other than Solaris did not actually need it,
so this bug was discovered late.
Thanks to Jakub Kulík for the patch.
Import <strings.h> when index() is used.
- 2022-08-04: openssl: add more details to error messages
- Avoid LogMsg outputs osslEndSess on successfull terminated
connection. Only LogMsg if the connection was terminated
unsuccessfully.
- Handle SSL_ERROR_SYSCALL in both Send / osslRecordRecv,
do not log as error if underlaying socket was terminated
(ECONNRESET). Log as information instead.
closes: https://github.com/rsyslog/rsyslog/issues/4946
- 2022-08-04: omclickhouse: capture additional exceptions
- DB::NetException
- DB::ParsingExceptions
Thanks to Victor Kustov for the patch.
- 2022-08-04: mmanon bugfix: Simplified and fixed IPv4 digit detection.
- Fixed an issue with numbers above int64 in syntax_ipv4.
Numbers that were up to 256 above the max of an int64
could incorrectly be detected as valid ipv4 digit.
- Simplified the IPv4 digit detection function and renamed
to isPosByte.
- added testcasse for malformed IPvc4 addresses
closes: https://github.com/rsyslog/rsyslog/issues/4940
- 2022-07-21: imptcp: slight tuning
- reduce indirect addressing to obtain more speed
- also a fix for an annoying typo
- minor other optimizations
- modernization of one test
- 2022-07-20: template procesing/json: performance optimization
- 2022-07-19: core bugfix: memory leak when free action worker data table
During free action worker data table when action destruct, worker instance in worker
data table were not null. It resulted in memory leak.
Thanks to github user seuzw930 for the patch.
- 2022-07-13: omfile: support for zstd compression
The zstd library provides better and faster compression than zlib.
This patch integrates zstd as a dynamically-loadable functionality.
As such, no further dependencies need to be added to the rsyslog
base package.
Due to the increased performance, usage of zstd is highly recommended
for high-volume use cases.
This patch also refactor zlib compression in order to unify handling
in both compression cases.
- 2022-07-07: stream cleanup: move error message to debug log, only
This error message is most probably rooted in a kernel problem. At
least knowbody knows how it can happen. It's definitely not a
rsyslog issue. We also can recover from it for a long time now
so there is no reason to irritate users by emitteing this
"error" message.
- 2022-07-04: mmdblookup bugfix: Don't crash Rsyslog on mmdb file errors
Thanks to Théo Bertin (frikilax) for the patch.
- 2022-06-28: build error fix: libbson requires out-of-date language constructs
- 2022-06-27: OpenSSL: fix depreacted API issues for OpenSSL 3.x
- OpenSSL error strings are loaded automatically now
- Debug Callback has changed
- See for more:
https://www.openssl.org/docs/manmaster/man7/migration_guide.html
closes: https://github.com/rsyslog/rsyslog/issues/4912
----------------------------------------------------------------------------------------
Scheduled Release 8.2206.0 (aka 2022.06) 2022-06-14
- 2022-05-25: omelastisearch: allow omitting _type field
Allow omitting the _type field by setting it to an empty string.
Setting this field has been deprecated since 6.0, and support will
be removed in 8.0
Also add testbench test for empty searchType with ES 7.0
This checks for messages in the deprecation log and also provides
avoids deprecation messages from usage of transport.tcp.port in the
test configuration
Thanks to Jarkko Oranen for the patch.
- 2022-05-18: tcpsrv/imtcp: slight performance improvements
This change slightly improves performance for tcpsrv-based servers.
This affects imtcp and imgssapi as well as some helpers.
No other functional change is included in this change.
- 2022-05-12: imptcp bugfix: worker thread starvation on extreme traffic
When connectes were totally busy, without any pause, the assigened worker
did never terminate its reading loop. As such, it could not service any
other conenctions. If this happened multiple time and to all configured
workers, all other connections could not be processed at all. This extreme
scenario is very unlikely, as the whole issue is relatively unlikely.
In practice, the issue could lead to somewhat degraded performance and
resolved itself after some time (in practice no connection is 100% busy
for an extended period of time).
Note that this patch sets a fixed limit of 16 iterations for very busy
connections. This sounds like a good compromise between non-starvation
and performance. The exact number may be made configurable if there
is really need to.
- 2022-05-11: omelasticsearch: several support option for ElasticSearch 8
- config params searchIndex and documentType can be empty
- support for Data Stream API
Thanks to github user EHerzog76 for these changes.
- new config param esVersion.major
- 2022-05-09: tcp receiver bugfix: delay/potential hang on some error conditions
Error were not correctly handled in some cases for imtcp and imgssapi. This could
lead to a temporary stall of some connections. For ultry-low traffic systems, this
stall could stay for a long period of time. In most cases, it was resolved very quickly.
Note that imptcp was not affected.
Thanks to Iwan Timmer for the fix.
- 2022-05-05: net bugfix: potential buffer overrun
there is heap buffer overflow vulnerability in rsyslog tcp reception components.
This can only happen in octet-counted mode, which is enabled by default.
Affected components: imtcp, imptcp, imhttp, imgssapi, imdiag when octet-counted
framing was enabled.
If the receiver ports are exposed to the public Internet AND are used
without authentication, this can lead to remote DoS and potentially to
remote code execution. It is unclear if remote code execution is
actually possible. If so, it needs a very sophisticated attack.
When syslog best practices with proper firewalling and authentication
is used, thean attack can only be carried out from within the Intranet
and authorized systems. This limits the severity of the vulnerability
considerably (it would obviously require an attacker already to be
present inside the internal network).
Credits to Peter Agten for initially reporting the issue and working
with us on the resolution.
fixes CVE-2022-24903
Advisory:
https://github.com/rsyslog/rsyslog/security/advisories/GHSA-ggw7-xr6h-mmr8#advisory-comment-72243
- 2022-05-05: imptcp: set OS worker thread name
We now set the worker thread names to "imptcp/<thrd nbr>" where
<thrd nbr> is the numerical index (0, 1, ...) of the worker thread.
This enables to distinguish individual worker threads in OS tools like
htop. That is useful for performance testing and system monitoring.
The choosen name format is consistant with other similar thread
names inside rsyslog. For imptcp, worker threads were not yet
given individual names.
Note: "in:imptcp" is imptcp's "main" thread, which also is used
as a worker in some scenarios. This name was not modified.
- 2022-04-26: mmanon bugfix: shortened IPv6 form not always anonymized
If the IPv6 is in non-recommended form followed by a 5 digit port number, it
is not anonymized.
A reproducer for this is: 1a00:c820:1180:c84c::ad3f:d991:ec2e:49255
closes https://github.com/rsyslog/rsyslog/issues/4856
- 2022-04-22: mmdblookup fix: wrong copy of buffer
...following parse of libmaxminddb's return after a successful search sometimes
failed to return specific field from data.
Thanks to Théo Bertin for the patch.
- 2022-04-22: mmdblookup: several enhancements
- support arrays in MMDB entry
- support escaped quotes '"' in MMDB entry
- support '<' characters in MMDB entry, when in a field
- support '}' characters in MMDB entry, when in a field
Thanks to Théo Bertin for the patch.
----------------------------------------------------------------------------------------
Scheduled Release 8.2204.1 (aka 2022.04) 2021-05-05
- security bugfix: potential buffer overrun in imptcp, imtcp, imgssapi and others
This addresses CVE-2022-24903
see also https://github.com/rsyslog/rsyslog/security/advisories/GHSA-ggw7-xr6h-mmr8
----------------------------------------------------------------------------------------
Scheduled Release 8.2204.0 (aka 2022.04) 2021-04-19
- 2022-04-18: gnutls bugfix: possibility of infinite loop
There was a rare possibility that the E_AGAIN/E_INTERRUPT handling
could cause an infinite loop (100% CPU Usage), for example when a TLS
handshake is interrupted at a certain stage.
* After gnutls_record_recv is called, and E_AGAIN/E_INTERRUPT error
occurs, we need to do additional read/write direction handling
with gnutls_record_get_direction.
* After the second call of gnutls_record_recv (Expand buffer)
we needed to also check the eror codes for E_AGAIN/E_INTERRUPT
to do propper errorhandling.
* Add extra debug output based on ossl driver.
* Potential fix for 100% CPU Loop Receiveloop after gtlsRecordRecv
in doRetry call.
closes https://github.com/rsyslog/rsyslog/issues/4834
closes https://github.com/rsyslog/rsyslog/issues/4818
closes https://github.com/rsyslog/rsyslog/issues/4638
- 2022-04-17: core/bugfix: errorfile could grow over max configures size
When action.errorfile.maxsize configuration option is enabled and error file
already has a certain size smaller than max size configured, it is increasing
higher than configured max size as the error file is considered to be zero in code.
This fix reads current error file size and limits the size to the maximum
size configured.
Thanks to Sergio Arroutbi for the patch.
fixes https://github.com/rsyslog/rsyslog/issues/4821
- 2022-04-17: omkafka bugfix: potential misadressing
The `failedmsg_entry` expects a null-terminated string in `key`, but
here we allocate with malloc and copy a string-with-length-n into only
the first n bytes. If the final byte is null, this is by coincidence
only.
This was observed by means of seeing random binary data appended to
keys submitted to kafka apparently at random. This could also result
in more severe problems, inclusing a segfault.
Thanks to David Buckley for the patch.
- 2022-04-06: added new "FullJSONFmt" standard template (with addtl fields)
This comes handy for a number of use cases, especially with ElasticSearch.
Thanks to Art O Cathain for the patch.
- 2022-04-04: imfile: potential processing delay
This was mentioned by Mikko Kortelainen without exact details on what exactly
this could cause in practice. But we were confident enough that it is worth
merging (though it does not look like something that brought real problems in
practice, as we do not know any related reports).
see also: https://github.com/rsyslog/rsyslog/pull/4445
Thanks to Mikko Kortelainen for the patch.
- 2022-04-04: bugfix: cosmetic data races
there was a more or less cosmetic data race which could happen when children
processes died in quick sequence. Even then, no real harm happened, as all
children were reaped eventually.
A similar data race exists for HUP processing.
However, these races polluted TSAN test runs, and so we fixed them
- 2022-04-01: add property options to support ISO week/year number
Thanks to Mattia Barbon for the patch.
- 2022-04-01: core bugfix: "action suspended" message was emitted even when turned off
Most messages were diasabled, but there was one part of the code that ignored the
user configuration.
Thanks to Deyneko Aleksey for the patch.
- 2022-03-31: testbench: add more tests for rscript comparison operations
- 2022-03-31: core bugfix: make internal logs emitted during HUP procesing appear quicker
After call doHUP(), probably there is a internal log in the list. However, it
will not be wrote out immediately, because the mainloop will be blocked at
pselect in wait_timeout() until a long timeout or next message occur.
More deadly, the log may be lost if the deamon exits unexpectedly.
We might as well put processImInternal() after doHUP(), so that the message
will be flushed out immediately.
Fixes: 723f6fdfa6(rsyslogd: Fix race between signals and main loop timeout)
Thanks to Yun Zhou for the patch.
- 2022-03-20: refactor: Move the parser directive to the main config
Thanks to Attila Lakatos for the patch.
- 2022-03-16: refactor: ake the main message queue part of the config
The intent of this patch is to make the main message queue part of the main config.
It will help us to proceed towards dynamic configuration reload.
- regression bugfix: rsyslog may segfault during startup
glblGetMaxLine() might be called even before the main configuration file exists
resulting unexpected behavior, most probably segmentation fault. This is addressed
by re-introducing the old default of 8KiB. The problem was introduced earlier in
2022.
- regression fix: script string comparison did not work correctly
In rscript, comparison operations on strings did not work correctly
and returned false results. This is cause by a regression in commit
5cec5dd634e0. While it fixed number comparisons, it introduced new
problems in string comparisons, which were not present before. Note
that most items in rsyslog are strings, so this can actually cause
some problems.
----------------------------------------------------------------------------------------
Scheduled Release 8.2202.0 (aka 2022.02) 2022-02-15
- 2022-02-14: imfile bugfix: remove cause for "internal error message" (not causing harm)
When any message is output into a renamed input file, rsyslogd output the following:
message.
imfile: internal error? inotify provided watch descriptor 7 which we could not find
in our tables - ignored
When rsyslogd detects the inode change, it deletes the entry from wdmap[]. But,
the watch descriptor is not removed. Some application like sssd outputs some messages
(like "HUP signal was received!!") after HUP signal is received and before switching
into the new log file. And, the above messages can be output every log rotation.
This situation is now resolved.
Thanks to Masahiro Matsuya for the patch.
- 2022-02-04: rscript bugfix: literal numbers were not compared correctly
This problem occurred when numbers were used in rsyslog.conf in
the set statement, e.g.
set $nbr = 1234;
In this case, during comparisons, the number was actually interpreted
as a string with digits. Thus numerical comparisons lead to unexpected
results. Even more so, as in other places of the code they were
treated as native numbers.
This is now fixed. We cannot outrule that this causes, in border cases,
change of behavior to existing configs. But it is unlikely and the
previous behaviour was a clear bug and very unintuitive. This in our
opinion it is justified to risk a breaking change for an expected
very minor subset of installations, if any such exists at all.
closes https://github.com/rsyslog/rsyslog/issues/4770
- 2022-02-04: omelasticsearch bugfix: indexSuccess impstats counter in bulkmode wrong
When bulkmode is enabled, and a batch was processed without any
failures (errors is false), the code that increments the indexSuccess
impstats counter was never reached.
closes: https://github.com/rsyslog/rsyslog/issues/4794
- 2022-01-17: imkmsg bugfix: effectively disabled input on error reading kmsg
Due to a program bug, imkmsg could not recover from an kmsg read error.
Note that recovering is possible and was intended.
Thanks to Kailash Sethuraman for the patch.
- 2022-01-17: imtcp bugfix: worker threads were not properly terminated
Graceful shutdown of Rsyslog could lead to segmentation faults when
multiple imtcp inputs were being used. That is because the rest of the
tcpsrv threads are left behind running, while their underlying objects
are being disposed by the main thread as part of the module
de-initialization.
closes: https://github.com/rsyslog/rsyslog/issues/4776
Thanks to Gabor Orosz <[email protected]> for the analysis and patch.
- 2022-01-07: omlibdbi bugfix: use-after-free bug
This occurred in when sqllite driver was used. Depending on circumstances, this had
no visible issues (often) up to rsyslog segfault. The busier rsyslog is, the more
likely a bad outcome.
- 2022-01-06: omhttp bugfix: memory leak in lokirest batchmode
A JSON object was created (valueObj) but not used and also not released causing a
memory leak. Over time, this could lead to memory overcomittent.
closes: https://github.com/rsyslog/rsyslog/issues/4766
----------------------------------------------------------------------------------------
Scheduled Release 8.2112.0 (aka 2021.12) 2021-12-16
- 2021-12-14: refactor:Deallocate outchannel resources in rsconf destructor
Thanks to Attila Lakatos for the patch.
- 2021-12-14: refactor: use runConf instead of loadConf in ratelimiting during runtime
Thanks to Attila Lakatos for the patch.
- 2021-11-22: new contribtion: URL parser module function using libfa
Thanks to Théo Bertin for the patch.
- 2021-11-18: mmanon: relax IPv6 detection - improve anonymization
We so far tried to ensure a value is really an IPv6 address, in order
to avoid to mangle with just similar-looking information elements.
However, this lead to misdetection for unusual formats, e.g. when a
port is appended to a numerical IPv6 adress given without braces [].
This has been changed now. In a sense, we now prefer to err on the
side of privacy.
BEHAVIOR CHANGE:
Previously, a suspect value was not anonymized, and thus some other
elements (like some MAC addresses) preserved. Now the opposite is
true, and we anonymize anything that looks close enough to be an
IPv6 address. This improves anonymization.
closes https://github.com/rsyslog/rsyslog/issues/4725
- 2021-11-10: ruleset bugfix: ruleset queue was incorrectly named
The ruleset was incorrectly and unusably named. This was a regeression
from 4a63f8e9629c3c9481a8b6f9d7787e3b3304320b.
Many thanks to github user digirati82 for alerting us.
closes https://github.com/rsyslog/rsyslog/issues/4730
- 2021-11-10: omsnmp: update module to current IP best practices
The omsnmp module uses the inet_addr() function to convert the Internet host address
from IPv4 numbers-and-dots notation into binary data in network byte order. If the input
is invalid, INADDR_NONE (usually -1) is returned. Use of this function is problematic
because -1 is a valid address (255.255.255.255). We should avoid its use in favor of
inet_aton(), inet_pton(3), or getaddrinfo(3), which provide a cleaner way to indicate
error return [1].
This is just a request to satisfy covscan, so no error is reported at all.
Thanks to Attila Lakatos for the patch.
- 2021-10-27: ommysql: fix threading bug
When the MariaDB connection was (re)established, old or NULL handle
could be used. This is fixed now.
We need to synchronize access to the mysql handle, because multiple threads
use it and we may need to (re)init it during processing. This could lead to
races with potentially wrong addresses or NULL accesses. If this really
matters mostly depends on the MariaDB/MySQL client library. It looks like
they guard against fatal failuers. Anyhow, logging errors inside rsyslog
could happen in any case.
- 2021-10-25: testbench: false positive when impstats was not built
Test omfwd_fast_imuxsock failed when impstats was not built. This
has been corrected, test is now only executed when impstats is
present.
- 2021-10-25: imtcp: add support for permittedPeers setting at input() level
The permittedPeers settig was actually forgotten during the refactoring
of TLS input() level settings. This functionality is now added.
closes: https://github.com/rsyslog/rsyslog/issues/4706
----------------------------------------------------------------------------------------
Scheduled Release 8.2110.0 (aka 2021.10) 2021-10-19
- 2021-10-13: config bugfix: global(security.abortonidresolutionfail=) did not work
when used with rscript based configuration, it was not checked.
- 2021-10-13: config bugfix: global param $privDropToUser did not work correctly
The parameter was not implemented for rscript based configuration and
did not properly apply to legacy configuration. In essence, it almost always
did not work as expected.
see also: https://github.com/rsyslog/rsyslog/issues/4642
see also: https://github.com/rsyslog/rsyslog/commit/cbcaf2c7e5b67e5465e47bc7cc67af2eae47bd31
- 2021-10-12: rscript bugfix: ruleset called async when ruleset had queue.type="direct"
The call rscript statement is able to call a rule set either synchronously or
asynchronously. We did this, because practice showed that both modes
are needed. For various reasons we decided to make async
calls if the ruleset has a queue assigned and sync if not.
To know if a "queue is assigned" we just checked if queue parameters were
given. It was overlookeded the case of someone explicitly specifying a
"direct queue", aka "no queue". As such, queue="direct" triggered async
calls. That in turn meant that when a write operation to a variable was
made inside that rule set, other rulesets could or could not see the
write. While if was often not seen, this was a data race where the
change could also be seen by the outside.
This is now fixed. No matter if queue.type="direct" is specified or
left out, the call will always by synchronous. Any values written to
variables will also be seen by the "outside world" in later processing
stages.
Note that this has some potential to BREAK EXISTING CONFIGURATIONS.
We deem this acceptable because:
1. this was racy at all, so unexpected behaviour could alwas occur
2. it is actually unlikely that someone used the triggering conditions
in practice. But we can not outrule this, especially when the
configuration was auto-generated.
Potential compatibility issues can be solved by defining a small
array-memory queue on the ruleset in question instead of specifying
direct type.
Again, we expect that almost all users will never experience any
problems. If you do, however, please let us know: we may add an
option to re-enable the bug.
- 2021-10-12: ksi bugfix: locking bug fixed in rsksiCtxOpenFile
Thanks to Taavi Valjaots for the patch.
- 2021-10-11: core bugfix: fix typo in error message
Thanks to github user jkschulz for the patch.
- 2021-10-11: tcpsrv bugfix: compilation without exceptions
tcpsrv.c:992:1: error: label at end of compound statement
finalize_it:
^~~~~~~~~~~
Quoting from pthread.h:
pthread_cleanup_push and pthread_cleanup_pop are macros and must always
be used in matching pairs at the same nesting level of braces.
Amends commit bcdd220142ec9eb106550195ba331fd114adb0bd.
Thanks to Orgad Shaneh for the patch.
- 2021-10-11: mkubernetes bugfix: no connection retry to kubernetes APP
When connection to the kubernates API was not possible, mmkubernetes
did not retry. This does now happen via regular rsyslog retry
mechanism.
Thanks to github user jayme-github for the analysis and patch.
closes https://github.com/rsyslog/rsyslog/issues/4669
- 2021-10-11: openssl bugfix: Correct gnutlsPriorityString (custom ciphers) behaviour
- Only apply default anon ciphers if gnutlsPriorityString is NULL and
Authentication Mode is set to anon. Otherwise we do not set them
as they overwrite custom Ciphers.
- Added two tests for custom cipher configuration (anon/certvalid mode).
- Add call for applyGnutlsPriorityString if gnutlsPriorityString changes.
- Merged openssl init code from Connect into osslInitSession
closes: https://github.com/rsyslog/rsyslog/issues/4686
- 2021-10-11: build issue: handle undefined MAXPATHLEN, PATH_MAX
While we handled missing PATH_MAX, we did not handle missing MAXPATHLEN.
This happens under GNU/Hurd, because there is no official limit. However,
extremely long pathes are extremely uncommon, so we do not want to
use slow dynamic alloc each time we need to build pathes. So we
impose a limit of 4KiB, which should be fairly enough. Note that
this obviously increases stack requirements in GNU/Hurd.
As suggested by Michael Biebl, we have now implemented a generic
approach to handle this via autoconf.
- 2021-09-12: openssl: extended output information on connection failure
Now includes the remote client/server IP address in the log output.
- 2021-09-12: imhttp enhancements - query parameter ingestion & basic auth support
- Basic Authentication support & tests
* configured via imhttp option "basicAuthFile". This option should be configured
to point to your htpasswd file generated via a standard htpasswd tool.
tests:
* imhttp-post-payload-basic-auth.sh
* imhttp-post-payload-basic-auth-vg.sh
- Query parameter ingestion capability & tests
use t `addmetadata` option to inject query parameters into
metadata for imhttp input.
DISTRO PACKAGERS BEWARE: NEW DEPENDENCY FOR IMHTTP:
libaprutil (libaprutil1-dev on debian'ish, apr-util-devel on Red Hat)
Thanks to Nelson Yen for the patch.
- 2021-09-07: testbench bugfix: privdrop tests under root user did not work
When running under root, the privdrop tests did not properly work. This
patch fixes the issue and skips test where necessary.
This also includes some modernization of the related tests.
closes https://github.com/rsyslog/rsyslog/issues/4619
- 2021-09-07: core/ratelimiting: fix rate limiting for already parsed messages
Rate limiting may not have worked if the considered message had already
been parsed (not having NEEDS_PARSING in msgFlags).
This affects also imuxsock in its default configuration
(useSpecialParser="true" and ratelimit.severity="1")
- 2021-09-07: core bugfix: use of property $wday terminates string
When $wday is used inside a template, all template parts after it
are ignored. For exmaple:
template(name="json_filename" type="string" string="/var/log/%$wday%.log")
would generate something like "/var/log/0" - the ".log" part would be
missing. For the same reason, $wday can not reliably checked in script
filters.
Thanks to Alain Thivillon for reporting the bug and providing an
excellent analysis, which essentiellay was exactly this fix here.
closes https://github.com/rsyslog/rsyslog/issues/4670
- 2021-09-07: core/queue bugfix: potential misadressing when queue discarded messages
When a discard mark was set, the queue was very busy and discarded messages, a
NULL pointer access could happen. Depending on circumstances, several problems
could occur, including a SEGFAULT. This is now fixed.
closes: https://github.com/rsyslog/rsyslog/issues/4437
- 2021-09-07: imdiga bugfix: iOverallQueueSize calculation could be incorrect
This issue only affects testbench and rsyslog development debugging. The active
messages counter, used for synchronizing test steps, went wrong when the queue
discarded messages on it's consumer thread. Now fixed.
- 2021-09-06: gnutls driver: SAN priority did not work correctly on server side
PrioritizeSAN was not propagated when accepting a new connection, this is now fixed.
Thanks to Attila Lakatos for the patch.
- 2021-08-24: config: implement script-equavalent for $PrivDrop* statements
closes https://github.com/rsyslog/rsyslog/issues/891
----------------------------------------------------------------------------------------
Scheduled Release 8.2108.0 (aka 2021.08) 2021-08-17
- 2021-08-16: openssl tls: Improved error message output on tls failures.
closes: https://github.com/rsyslog/rsyslog/issues/4645
- 2021-08-16: impstats: add percentile metrics tracking functionality
Brief overview:
TO configure tracking percentile metrics in rainerscript:
User would need to define:
- which percentile to track, such as [p50, p99, etc.]
- window size - note, this correlates directly with memory usage to
track the percentiles.
To track a value, user would call built-in function `percentile_observe()` in their configurations to
record an integer value, and percentile metrics would be emitted every
impstats interval.
Thanks to Nelson Yen for the patch.
- 2021-08-12: imfile: add parameter "ignoreolderthanoption"
instructs imfile not to ingest a file that has not been modified in the
specified number of seconds.
Thanks to github user yanjunli76 for the patch (submitted from Nelson Yen)
- 2021-08-10: imklog bugfix: invalid memory adressing, could cause abort
This is a regeression from commit 94c4a87. It introduced a free() call
using an object that was no longer valid (the main pointer to the
to-be-freed object) was already freed at time of use. This could
cause various issues, including a segfault.
Note: this bug was triggerred only during late phase of rsyslog
shutdown, so it did not affect regular operation.
Special thanks to github user wxiaoguang for analyzing the issue
and providing a draft fix proposal, on which this patch builds.
see also https://github.com/rsyslog/rsyslog/pull/4629
closes https://github.com/rsyslog/rsyslog/issues/4625
- 2021-08-09: imfile bugfix: deleteStateOnFileDelete missed some state files
When the log file is deleted, imfile would attempt to delete the statefile but it
was missing the file_id part of the statefile name. This means the statefiles were
only removed in the log file was less than 512 characters, because for very small
files the file ID hash is not created. This lead to some state files not being
deleted.
Thanks to pearseimperva for the patch.
- 2021-08-09: imfile bugfix: hash char invalidly added in readmode != 0
If imfile is ingesting log files with readMode set to 2 or 1, the resulting
messages all have a '#' character at the end. This patch corrects the behaviour.
Note: if some external script "supported" the bug of extra hash character at
the end of line, it may be necessary to update them.
closes https://github.com/rsyslog/rsyslog/issues/4491
- 2021-08-09: omelasticsearch bugfix: errorFile mutex was not consistently locked
Lock the file during SIGHUPs to avoid issues with concurrent accesses by
writeDataError().
Thanks to François Poirotte for the patch.
- 2021-08-09: imudp: add socket type (IPv4 vs. 6) to input name
Most importantly, the input name is used for stats counter names as
well. Previously, the same name was used for IPv4 and IPv6, so we had
two counters with an equal name. That left users puzzled.
Unfortunately, this change can potentially require changes to existing
analysis scripts, as the name is now slightly different.
closes https://github.com/rsyslog/rsyslog/issues/4364
- 2021-08-06: omfwd: add capability for action-specific TLS certificate settings
This permits to override the global definitions for TLS certificates
at the action() level.
- 2021-08-06: imfile bugfix: file handle leak if "freshStartTail" was turned on
- 2021-08-05: imtcp: permit to use different certificate files per input/action
This completes the ability to override global/default TLS settings at the imtcp
input() level. Support for using multiple CAs/Certs per Connection is now provided.
- 2021-08-04: imptcp bugfix: keep alive interval was incorrectly set
The interval was accidentally set to keep alive interval. This has been
corrected.
closes https://github.com/rsyslog/rsyslog/issues/4609
- 2021-07-08: openssl network driver bugfix: small memory leak
Fixes a static, non-growing memory leak which existed when parameter
"GnutTLSPriorityString" was used. This was primarily a cosmetic issue,
but caused some grief during development in regard to memory leak
detectors.
Note: yes, this is for openssl -- the parameter name is historical.
- 2021-07-07: psrv bugfix: abort if no listener could be started
Modules (like imtcp and imdiag) which use tcpsrv could abort or
otherwise malfunction if no listener for a specific input could
be started.
Found during implementing a new feature, no report from practice.
But could very well happen.
- 2021-07-07: mmkubernetes bugfix: apiserver error handling
- Added graceful handling of apiserver errors with unexpected responses,
i.e., anything other than 200, 404, or 429. Idea is that apiserver
transient error state will recover. We don't want mmkubernetes to miss
metadata resolution for containers that don't have cached metadata.
During these transient error states, mmkubernetes will provide basic
container file path based resolution of namespace and pod metadata for
new pods whose metadata is not yet cached. After this error state
recovers, mmkubernetes is expected to resume its metadata resolution as
expected.
- Added a unit test case for apiserver return 500 with changes to mock server
- Fixed existing unit test that was failing due to missing expected results file
- Added mmkubernetes unit tests to testbench
Thanks to Abdul Waheed for the patch (submitted from Nelson Yen).
- 2021-07-07: ommongodb bugfixes
- Fix Segmentation fault when server is down
- Add server connexion check while resuming
Thanks to Kevin Guillemot for the patch.
- 2021-06-28: omkafka improvements
- drain librdkafka queues and retry later during rsyslog restart or hup. This
re-injects messages into rsyslog's native queues.
- add statsname on per kafka instance for better visibility
- omkafka - count errors related ssl as "errors_ssl"
Thanks to Nelson Yen for the patch.
- 2021-06-23: some CI/QA improvements, Travis-CI disabled
For the time being, Travis CI is disabled because it was outdated and Travis also
changed their system. We will re-evaluate if we re-enable it. Since quite a while
the Travits tests were redundant with the rest of CI, so this does not reduce
coverage.
- 2021-06-23: omhttp bugfix: dynrestpath param in batch mode invalid
When batchmode was used, the templates could not be used to
expand dynrestpath. We are now storing the restpath param
within the batch data if we are in batch mode.
When we are in batch mode, and the restpath value changes, the
batch is submitted and reinitialized
closes: https://github.com/rsyslog/rsyslog/issues/4567
- 2021-06-17: add predefined template RSYSLOG_SyslogRFC5424Format
This is essentially the same as RSYSLOG_SyslogProtocol23Format with
a better name and a fix to remove the unnecessary LF at the end of
the message.
The different name also enables us to fix the LF issue without
any concern about backwards compatibility.
closes https://github.com/rsyslog/rsyslog/issues/4384
- 2021-06-17: impstats/bugfix: _sender_stats reports integer counter as string
Note that this introduces a small backwards incompatibility: in previous output
the field was of string type, now it is integer (as intended). We discussed this
on the mailing list and the overwhelming thought was that this is not a problem
because almost all analysis backends are able to cover that format change. This made
the bugfix essentially costmetic.
HOWEVER, if you still experience issues, please let us know. We can add an option
to provide the previous format, and just spared to do so because there was no
evidence it was needed.
----------------------------------------------------------------------------------------
Scheduled Release 8.2106.0 (aka 2021.06) 2021-06-15
NOTE: the prime new feature is support for TLS and non-TLS connections
via imtcp in parallel. Furthermore, most TLS parameters can now be overriden
at the input() level. The notable exceptions are certificate files, something
that is due to be implemented as next step.
- 2021-06-14: new global option "parser.supportCompressionExtension"
This permits to turn off rsyslog's single-message compression extension
when it interferes with non-syslog message processing (the parser
subsystem expects syslog messages, not generic text)
closes https://github.com/rsyslog/rsyslog/issues/4598
- 2021-05-12: imtcp: add more override config params to input()
It is now possible to override all module parameters at the input() level. Module
parameters serve as defaults. Existing configs need no modification.
- 2021-05-06: imtcp: add stream driver parameter to input() configuration
This permits to have different inputs use different stream drivers
and stream driver parameters.
closes https://github.com/rsyslog/rsyslog/issues/3727
- 2021-04-29: imtcp: permit to run multiple inputs in parallel
Previously, a single server was used to run all imtcp inputs. This
had a couple of drawsbacks. First and foremost, we could not use
different stream drivers in the varios inputs. This patch now
provides a baseline to do that, but does still not implement the
capability (in this sense it is a staging patch).
Secondly, we now ensure that each input has at least one exclusive
thread for processing, untangling the performance of multiple
inputs from each other.
see also: https://github.com/rsyslog/rsyslog/issues/3727
- 2021-04-27: tcpsrv bugfix: potential sluggishnes and hang on shutdown
tcpsrv is used by multiple other modules (imtcp, imdiag, imgssapi, and,
in theory, also others - even ones we do not know about). However, the
internal synchornization did not properly take multiple tcpsrv users
in consideration.
As such, a single user could hang under some circumstances. This was
caused by improperly awaking all users from a pthread condition wait.
That in turn could lead to some sluggish behaviour and, in rare cases,
a hang at shutdown.
Note: it was highly unlikely to experience real problems with the
officially provided modules.
- 2021-04-22: refactoring of syslog/tcp driver parameter passing
This has now been generalized to a parameter block, which makes it much cleaner and
also easier to add new parameters in the future.
- 2021-04-22: config script: add re_match_i() and re_extract_i() functions
This provides case-insensitive regex functionality.
closes https://github.com/rsyslog/rsyslog/issues/4429
----------------------------------------------------------------------------------------
Scheduled Release 8.2104.0 (aka 2021.04) 2021-04-20
- 2021-04-19: new contributed module imhiredis
Thanks to Théo Bertin (frikilax) for the patch.
- 2021-04-19: new built-in function get_property() to access property vars
Provides ability to evaluate a rsyslog variable using dynamically
evaluated parameters.
1st param is the rsyslog param, 2nd param is a key, can be an array
index or key string.
Useful for accessing json sub-objects, where a key
needs to be evaluated at runtime. Can be used to access arrays as well.
Thanks to Nelson Yen for contributing this module.
- 2021-04-19: mmdblookup: add support for mmdb DB reload on HUP
Thanks to Théo Bertin (frikilax) for the patch.
- 2021-04-19: script bugfix: empty array in foreach() improperly handled
When running a foreach() loop inside a ruleset, if the json array/object iterated
over is empty but valid, the foreach will make the message processing in the
ruleset abort operation, no following operation (such as actions) will be
executed after this.
Thanks to Théo Bertin (frikilax) for the patch.
- 2021-04-19: imjournal bugfixes (handle leak, empty file)
Flush the FILE* buffer before rename & fsync in order
to not end up syncing an empty file.
Also, close WorkDir on fsync in order to prevent
file descriptor leakage.
Thanks to github user gerd-rausch for the fix.
- 2021-04-06: new contributed function module fmunflatten
This commit adds a new rainerscript function to unflatten keys in a JSON tree. It
provides a way to expand dot separated fields.
<result> = unflatten(<source-tree>, <key-separator-character>);
It allows for instance to produce this: { "source": { "ip": "1.2.3.4", "port": 443 } }
from this source data: { "source.ip": "1.2.3.4", "source.port": 443 }
Thanks to Julien Thomas for the contribution.
- 2021-02-22: test bugfix: some tests did not work with newer TLS library versions
Newer versions provide TLS versions that cannot be disabled in older versions as they
are unknown there. This is solved by setting restrictions in multiple steps. For
older library versions, the final step will error out, but the other one be applied.
This permits to achieve proper test results.
closes: https://github.com/rsyslog/rsyslog/issues/4534
- some improvements to project CI
----------------------------------------------------------------------------------------
Scheduled Release 8.2102.0 (aka 2021.02) 2021-02-16
- 2021-02-15: omfwd: add stats counter for sent bytes
Thanks to John Chivian for suggesting this feature.
- 2021-02-15: omfwd: add error reporting configuration option
RSyslog on a plain TCP cannot guarantee the message delivery
without using RELP protocol. Besides that the logs may be
flooded with connection errors making the rest of messages
difficult to find. To alleviate the problem (see issue 3910),
this patch adds a configuration option that enables to reduce
the number of network errors logged and reported.
For example, if each 10th network error message should be logged,
the rsyslog configuration has to be updated as follows.
action(type="omfwd" Target="<IP_ADDR>" Port="<PORT>" Protocol="tcp" ConErrSkip="10")
Thanks to Libor Bukata for the patch.
- 2021-02-15: action stats counter bugfix: failure count was not properly incremented
In some cases the counter was not incremented, most notably with transaction-enabled
actions.
Thanks to github user thinkst-marco for the patch.
- 2021-02-15: action stats counter bugfix: resume count was not incremented
And so it always stayed at zero.
Thanks to github user thinkst-marco for the patch.
- 2021-02-15: omfwd bugfix: segfault or error if port not given
If omfwd is configured via RainerScript config format and the "port"
parameter is not given, a segfault will most likely happen on
connection establishment for TCP connections. For UDP, this is
usually not the case.
Alternatively, in any case, errors may happen.
Note that the segfault will usually happen right on restart so this
was easy to detect.
We did not receive reports from practice. Instead, we found the bug
while conducting other work.
- 2021-01-29: lookup table bugfix: data race on lookup table reload
A data race could happen when a lookup table was reloaded. We found
this while moving to newer version of TSAN, but have no matching
report from practice. However, there is a potential for this to cause
a segfault under "bad circumstances".
- 2021-01-18: testbench modernization
Bump dependency versions, use newer distro versions for some tests.
Make kafka distcheck separate to help diagnose flaky kafka tests.
- 2021-01-16: testbench: fix invalid sequence of kafka tests runs
kafka tests can not run well in parallel (mostly due to ressource
constraints on CI machines). Accidentally, this was not enforced for
one of the tests. That could lead to random failures and false positives.
- 2021-01-14: testbench: fix kafkacat issues
The kafkacat tool has an upper limit of how many messages it can send
at once. Going over that limit causes messages loss. The exact limit
seems to depend on the environment. This causes testbench false positives.
This commit fixes two related issues:
- errors during kafkacat run were not detected - this has been added
- we now have a "max messages at once" setting, after which kafkacat
is restarted for the next batch of messages. It currently is set
to 25,000 msgs per incarnation. All tests loop now to send the
required number of messages. This has been fixed at the testbench
framework level, so no need to adjust individual tests.
- 2021-01-14: testbench: fix year-dependendt clickhouse test
A test had the year value hardcoded and as such failed whenever the
year changed. This patch corrects that.
----------------------------------------------------------------------------------------
Scheduled Release 8.2012.0 (aka 2020.12) 2020-12-08
- 2020-12-07: testbench bugfix: some tests did not work in make distcheck
- certificate file missing in dist tarball
- some test cases did not properly specify path to cert file
Thanks to Michael Biebl for alerting us and providing part of
the fix.
closes https://github.com/rsyslog/rsyslog/issues/4446
- 2020-12-07: immark: rewrite with many improvements
- mark message text can now be specified
- support for rulesets
- support for using syslog API vs. regular internal interface
- support for output template system
- ability to specify is mark message flag can be set
- minor changes and improvements
- 2020-11-30: usability: re-phrase error message to help users better understand cause
see also https://github.com/rsyslog/rsyslog/issues/3910
- 2020-11-10: add new system property $now-unixtimestamp
Among others, this may be used as a monotonic counter
for doing load-balancing and other things.
Thanks to Nicholas Brown for suggesting this feature.
- 2020-11-04: omfwd: add new rate limit option
Adding new rate limit option to omfwd for rate limiting
syslog messages sent to the remote server
ratelimit.interval:
Specifies the rate-limiting interval in seconds.
Default value is 0, which turns off rate limiting.
ratelimit.burst
Specifies the rate-limiting burst in number of messages.
closes https://github.com/rsyslog/rsyslog/issues/4423
Thanks to Dinesh-Ramakrishnan for the patch.