Features
- Release of the
kubernetes_events_monitor
plugin with collects Kubernetes Events and pushes them into Scalyr. See the monitor documentation for more information.
Bugs
- Fix issue preventing log configs with glob patterns to override exact match patterns. This is required to help override redaction and parsers rules for Docker.
- Fix issue in mysql_monitor preventing error message from being logged
Features
- Add in v2 K8s support. You must configure a k8s cluster name to start using the new v2 support. See Kubernetes Agent Install directions.
Bugs
- Only allow non-secure connections to Scalyr if explicit override option is sset.
- Redesign of k8s cache to better handle large number of pods
- Allow the k8s api server address to be set via a configuration option.
- Parallelized container metric fetches to avoid not being able to collect metrics in time.
Bugs
- Fix bad class name for wrapped responses in Docker and Kubernetes monitor.
- Fix high CPU consumption in Kubernetes monitor due to encoding detection.
- Add defensive code to Kubernetes cache to prevent cache death spiral.
Bugs
- Fix urgent incompability issue with python version < 2.7 in call to string decoding.
Bugs
- Adjusted unicode decoding to be more tolerant of bad input.
- Added support for optimized json parsing library. Changed k8s agent to use optimized version.
Features
- Added ability to specify default values for environment variables imported in configuration files.
Bugs
- Added temporary hack to new Kubernetes support to upload Daemonsets as Deployments.
Features
- Finalized new support for Kubernetes. Not turned on by default yet. Will be generally available in November.
Bugs
- Fix journald monitor to system default path.
Bug fixes
- Remove / rework debug log statements that were consuming too much CPU.
- Add ability to disable the Kubernetes monitor from parsing the json log lines on the client
- Do not write out the full checkpoint.json file every iteration. Instead, write out only the active log files on every iteration with a consolidation every one minute.
Features:
- Added support in the
kubernetes_monitor
for annotations-based configuration. You can now specify parsers and whether or not a pod's logs can be sent to Scalyr via k8s annotations. See documentation onkubernetes_monitor
for more details.
Bug fixes:
- Added fixes that should eliminate memory leak.
Features:
- Added
journald_monitor
for collecting logs from a localjournald
service. Requires installation of the Pythonsystemd
library. - Added
garbage_monitor
for debugging memory consumption issues in the agent. - Added
verify_k8s_api_queries
option to thekubernetes_monitor
to allow disabling SSL verification on local API calls.
Bug fixes:
- Fixed issues causing the
linux_process_metrics_monitor
to throw exceptions when the monitored process dies. - Additional debug flags to aide in investigating memory issues with the agent.
Features:
- Relaxed configuration rules to allow for any config option to be specified in a configuration fragment in the
agent.d
directory.
Bug fixes:
- Fixed bug in
linux_process_metrics
that would generate errors when a monitored process died - Fixed issue in the docker and kubernetes monitors that would prevent the agent from sending logs from short lived containers
- Fixed issue in the docker and kubernetes monitors that could cause the tail end of a containers logs to not be uploaded after it exits
- Fixed issue in docker monitor where the
metrics_only
config option was not being obeyed. - Added various options to help investigate memory leak
Bug fixes:
- Added various options to help investigate memory leak
Features:
- Added kubernetes support with introduction of new kubernetes monitor along with kubernete image published on dockerhub.com under scalyr/scalyr-k8s-agent. See online documentation for more information.
- Added
network_interface_suffix
option tolinux_system_metrics
monitor to control regular expression used to validate interfaces names. Additionally, widened existing rule to accept letters at the end of the interface name. - Added
aggregate_multiple_processes
tolinux_process_metrics
which allows multiple processes to be matched by process matching rule. The reported metrics will include the statistics from all matching processed. - Added
include_child_processes
tolinux_process_metrics
to include all child processes from the matched process in the reported metrics.
Features:
- Changed default for
max_line_size
to 9900 to match new largermessage
field support on the server.
Bug fixes:
- Fixed bug causing the Docker plugin to
leak
file descriptors as new containers were added while using thedocker_api
mode.
Features:
- Support to parse log lines written as a JSON object to extract line content before sending to Scalyr. You may turn this on by setting
parse_lines_as_json
totrue
in your log configuration stanza for a particular file. This is useful when uploading raw Docker logs. - Read the Scalyr api key from an environment variable
- Add support for
PUT
requests to thehttp_monitor
. - Add support for replacing redacted values with a hash of the actual value. See the redaction documentation for more details.
Bug fixes:
- Fix win32 build such that it correctly pulls packages from the third party repository. This fixes a bug resulting in SOCK5 proxy support not working under Windows.
New features:
- Upgraded
requests
library to support new proxy protocols such asSOCKS5
. To useSOCKS5
, use either thesocks5
orsocks5h
protocol when specifying the proxy address (socks5
resolves DNS addresses locally, whereassocks5h
resolves addresses at theSOCKS5
server).
Bug fixes:
- Fix bug preventing multiple instances of the
syslog_monitor
from logging to separate log files.
New features:
- Added new option
report_container_metrics
todocker_monitor
to allow disabling of gathering and reporting metrics for each Docker container.
Bug fixes:
- Fix recent breakage causing the
run_monitor
code to fail. - Fix error / warning logging in
syslog_monitor
to better capture issues with Docker support.
New features:
- Added feature to allow completely disabling the use of the Docker socket in certain Docker configurations.
- New
max_log_size
andmax_log_rotations
configuration options that can be used to set the maximum length an agent-generated log file can grow before it is rotated and the maximum number of rotations to keep.
Bug fixes:
- Fix bug preventing rotated Docker files from being deleted
- Fix bug that resulted in
https_proxy
and ``http_proxy` configuration options being ignored.
New features:
- Docker integration: Old docker log files will be deleted after unused for 1 day (by default).
- Change to pidfile format in preparation for better
systemd
support
Bug fixes:
- Syslog/Docker fix that was causing logs being uploaded with default parser instead of user-specified parser
- Minor fixes for BMP unicode
New features:
- New
rename_logfile
option to change the file name and path for a log file when uploading it to Scalyr. - New
copy_from_start
option to instruct the agent to copy a log file from its start when it first matches. - New metric in
linux_system_metrics
to record number of CPUs used by a machine. Metric name issys.cpu.count
. - New
strip_domain_from_default_server_host
configuration option to remove any domain name fromserverHost
when using the default value.
Bug fixes:
- Syslog/Docker fix that generated a
port already in use
error when reading new configuration file - Syslog/Docker fix that caused dropped log lines when write rate exceeded certain threshold
- Syslog CPU performance improvements
Major overhaul of Docker support:
- New approach to improve stability across multiple versions of Docker
- Relies on Docker's
syslog
logging plugin - Automatic collection of Docker metrics for all containers running on host
- Improved Dockerfiles to ease creating images with custom Scalyr Agent configuration
- Improved methods for configuring Scalyr Agent running within the container
- Customize where container log files are written
- See the Docker installation documentation for more details.
Additional features:
- New
exclude
option for writing log matching rules. When specifying which logs to collect using a glob pattern, you can exclude any of the matching logs using another glob. The field value should be an array of glob patterns. - The
import_vars
feature has been extended to work in all files in theagent.d
directory. The variables imported by a file are only applied to that file. - Both the
api_key
andscalyr_server
fields may now be set in any file in theagent.d
directory. - Initial support for compressing upload payloads
Bug fixes:
- Removed spurious exceptions in the
linux_process_metrics
module due to reading blank lines from/prod/pid/io
- Prevent log upload being wedged when encountering invalid utf-8 characters
- Prevent errors in Windows when file descriptors were attempted to be close
- More diagnostic output and defensive code for the
linux_process_metrics
module to help investigate issue
Bug fixes:
- Fixed bug in
linux_process_metrics
causing errors when monitored process disappears
Features:
- Added ability to set HTTP and HTTPS proxies via configuration file. Use new configuration variables
http_proxy
andhttps_proxy
. You must also setuse_requests_lib
to true. - Report the number of open file descriptors held by a process for
linux_process_metrics
. The metric name isapp.io.fds
. - The assigned the
scalyrAgentLog
parser to the agent log - Added ability to skip the Scalyr connectivity check at agent start up using
--no-check-remote-server
.
Bug fixes:
- Added third-party library to fix issue with
postgres_monitor
. - Added extra logging to track issue reported with Windows agent of silent shutdown.
- Fixed issue with
haltBefore
line grouper that prevented logs from being uploaded for several minutes. - Changed default for
line_completion_wait_time
from 300 secs to 5 secs. - Always delete the
*.pyc
files when doing an upgrade or uninstall
Features:
- Added feature to prevent tracking stale log files on a per-directory basis. Reduces checkpoint size due to large directories.
Bug fixes:
- Fix bug causing agent to fail startup due to
'module' object has no attribute 'UUID'
. - Improved log line format for
snmp_monitor
. You may disable using new format by setting the monitor config optionuse_legacy_format
to true.
Features:
- Release of new
snmp_monitor
plugin, used to monitor SNMP devices
Bug fixes:
- Fix bug in
windows_process_metrics
when matching by commandline. - Fix invalid character bug when parsing unicode characters with decimal value > 2^16
- Fix bug in
shell_monitor
plugin resulting in defunct processes lingering - Fix bug in
url_monitor
plugin resulting in not emitting metric in some failure cases
Features:
- Integration of new request library, allowing routing through target specified by
proxy
environment variable. New library turned off by default for now. To turn on, setuse_requests_lib
.
Bug fixes:
- Fix
windows_system_metrics
monitor to no longer report disk usage on empty drives - Fix
mysql_monitor
to recreate connection to db on failures - Fix metrics names rather than throw errors when invalid metrics names are emitted by modules such as graphite.
Features:
- Windows Event Log monitor now supports the EvtLog API implemented by Windows Vista and newer system. This allows for more expressive queries, as well as selecting events by severity level such as "Critical".
Bug fixes:
- Fix UTF-8 truncation bug in redis plugin.
- New option in redis plugin that controls logging of UTF-8 converstion errors. See the
utf8_warning_interval
for more details. - Fix bug where
network_interface_prefixes
option was broken when listing multiple interfaces. - Fix
no attribute _tunnel_host
bug with python 2.4 ssl library
Features:
- Added plugin for reading SLOWLOG from redis
- Heavily optimized common path for copying logs to increase throughput
Features:
- New sequence id and number implementation to help prevent events being added to the database multiple times
- Create new configuration option
max_existing_log_offset_size
to set a limit on how far back in a log file we are willing to go for a log file we have been previously copying. - Enabled
syslog_monitor
for Windows platforms - Preview release of
windows_event_log_monitor
, a monitor for copying the Windows event log to Scalyr. - Preview release of docker support. E-mail [email protected] for more details
- Added new option
network_interface_prefixes
to linux_system_metrics monitor to override the prefix for monitored network interfaces.
Bug fixes:
- Fixed some race conditions in writing the pidfile on POSIX systems.
- Fixed bug causing agent to stop copying logs when previously copied logs are removed
- Disabled checking the command of the running agent process to guard against pid re-use since this was leading to some false negatives in the
is the agent running
checks. - Remove exception logging when bad response received from server
- Print more informative message when server responds with
server too busy
- Fix Utf-8 encoding bug with redaction rules
Bug fixes:
- Added some extra diagnostic information to help investigate some customer issues
New features:
- Enable line groupers on the agent. Each log file may specify line grouping rules in its configuration. The agent can use this to group consecutive log lines together into a single logic log line. All future sampling and redaction rules will work on this single line instead of the individual log lines.
Bug fixes:
- Close file handles for files being scanned that have not had any activity in the last hour. Should fix too many open files bug.
- Reduce the default
max_line_size
value to 3400 to avoid truncation issues introduced by the server.
New features:
- Allow any monitor's metric log's rate limiter to be modified via configuration, along with flush aggregation.
- Provide
--upgrade-without-ui
onscalyr-agent-2-config
to allow upgrading the agent without UI (Windows). - Write user-friendly error message to event log when a major configuration problem is seen during start up (Windows)
Bug fixes:
- Do not fail Scalyr Agent service start if configuration file registry is not set (Windows).
- Remove rate limiter on metric log for graphite monitor and add in flush aggregation.
- Fix that prevented graphite monitor thread from starting when accepting both text and pickle protocols
Bug fixes:
- Prevent syslog monitor from writing syslog message to agent log.
- Performance improvement for syslog monitor to prevent it from flushing the traffic log file too often
- Fix bug in error reporting from client that caused exceptions to be thrown while logging an exception
New features:
- Added syslog monitor for receiving logs via syslog protocol.
Bug fixes:
- Fix bug causing agent to temporarily stop copying logs if the underlying log file disappeared or read access was removed for Windows.
New features:
- Make log processing parameters such as
max_log_offset_size
adjustable via the configuration file.
Bug fixes:
- Fix bug preventing turning off default system and agent process monitors.
- Fix bugs preventing release of open file handles
Bug fixes:
- Relax metric name checking to allow including dashes.
- Fix bug where linux_system_metrics was ignoring the configured sample interval time.
New features:
- Add a new option
--set-server-host
in scalyr-agent-2-config to set the server host from the commandline. - Add new instance variables
_log_write_rate
and_log_max_write_burst
to ScalyrMonitor to allow monitor developers to override the rate limits imposed on their monitor's log. See the comments in test_monitor.py for more details. - Add new parameter
emit_to_metric_log=True
to force a log line to go to the metric log instead of the agent log for monitors. See the comments in test_monitor.py for more details. - Added new configuration parameters to easily change any or all monitor's sample rate. Use
global_monitor_sample_interval
in the general configuration to set it for all monitors, orsample_interval
in an individual monitor's config. The value should be the number of seconds between samples and can be fractional.
Bug fixes:
- Fix failing to accept unicode characters in metric values
New features:
- Support for Windows. Many changes to support this platform.
Bug fixes:
- Fix excessive CPU usage bug when monitoring thousands of logs that are not growing.
New features:
- The
run_monitor.py
takes a new option-d
to set the debug level.
Bug fixes:
- Fix false warning message about file contents disappearing.
- Assign a unique thread id for each log file being copied to mimic old agent's behavior.
New features:
- Added the MySQL, Apache, and Nginx monitor plugins.
- New functions for defining and adding documentation for Scalyr Plugin Monitor configuration options, log entries, and metrics.
- Added support for variable substitution in configuration files.
- New
print_monitor_doc.py
tool for printing documentation for a monitor. This tool is experimental.
Bug fixes:
- Fixed bug that prevented having release notes for multiple releases in CHANGELOG.md
- Fixed bug with CR/LF on Windows preventing log uploads
Documentation fixes:
- Updated MySQL monitor documentation to note deprecated metrics.
Internal:
- Refactored code to extract platform-specific logic in preparation for Windows work.
New core features:
- Complete rewrite of the Scalyr Agent in Python!
- No more Java dependency.
- Easier to configure.
- Easier to diagnosis issues.
- Monitor plugin support.
- Detailed status page.
Bugs fixed since beta and alpha testing:
- Slow memory "leak".
- graphite_module not accepting connections from localhost when only_accept_local is false.
- Ignore old pidfile if the process is no longer running. Added pid collison detection as well.