Skip to content

Latest commit

 

History

History
968 lines (686 loc) · 53 KB

changelog.rst

File metadata and controls

968 lines (686 loc) · 53 KB
Raw

Changelog

Note

Changes in the devel branch, but not released yet are marked as DRAFT.

Ksconf 0.13

Switching to use Python's namespace packages. This is an internal change that should make future expansions easier but should have no impact on normal users.

New namespaces:

  • ksconf
  • ksconf.commands
  • ksconf.plugins

Renames:

  • ksconf/__init__ -> ksconf.version
  • ksconf/commands/__init__.py -> ksconf/command.py

Ksconf v0.13.9 (2024-01-04)

  • Fix app deployment bug that can occur if a previously deployed path changes from a file to a directory.

  • Internal layering improvements. * Add/improve layer caching mechanisms.

    Packaging large apps saw a 2-3x performance improvement during development.

    • Add ability to filter layer operations base on based path names (not just layer names).
    • Created concept of layer types, currently supporting implicit vs explicit layers.
    • Cleanup some internally confusing terminology.

Ksconf v0.13.8 (2023-12-18)

More internal Laying improvements for the ansible collections (and general readability improvements). No improvements for CLI users.

  • Additional Layer handling improvements for AppPackager and LayerCombiner. Primary benefits of these internal improvements are for the ksconf_package command in the `cdillc.splunk`_ collection, and anyone who has to look at the code.
  • Replace all LayerRoot* terminology with LayerCollection to reduce confusion. The term _root_ was used multiple places within the layering system to both a top-level folder, as well as a container of layers.
  • Replace confusing nested Layer class structure for a more traditional flat one.
  • Add a convenience function for LayerCollection creation (that handles various layering schemes).
  • Add :py:method:~`ksconf.layer.LayerFilter.add_rules` to simplify frequent use case.

Ksconf v0.13.7 (2023-12-12)

  • Update some app builder internal caching mechanisms that are now re-used by the ksconf_package command in the `cdillc.splunk`_ collection. That's it. Everyone else can just skip this version.

Ksconf v0.13.6 (2023-12-06)

  • Add new --empty-stanza to :ref:`ksconf_cmd_filter`. Thanks to Manoj Baba for the idea.
  • Fix unarchive bug when dealing with local files.
  • Improve handling of UTF8-BOM in some situations. As always, the marker (BOM) will be silently discarded.

API Changes:

Ksconf v0.13.5 (2023-10-20)

  • Fix bug in ksconf diff` when using the --format=json option. This has been broken for a number of releases (added a unit test to keep that from repeating). Fixed the newly caused import issue caused by a missed file during the v0.13.4 release, as well as earlier Enum-related bug: TypeError: Object of type DiffVerb is not JSON serializable.
  • :py:class:`~ksconf.app.manifest.AppManifest`: Add filter_files() method for generic filtering capabilities, and drop_ds_autogen() to handle the specific use case of removing that "autogenerated" file created by Deployment Server.
  • Linter and typing fixes

Ksconf v0.13.4 (2023-10-13)

API Changes (only):

Ksconf v0.13.3 (2023-10-12)

  • Created a new public interface :py:mod:`ksconf.version`. This provides a consistent means of getting the version information without any extra modules being loaded, now that ksconf is a namespace package. Note that for v0.13.0 - 0.13.2, this was called ksconf._ksconf. Hopefully in that short time frame nobody got too attached to that terrible name.
  • Changes for :ref:`ksconf_cmd_rest-publish`:
    • Implement --insecure parameter. Previously, SSL/TLS validation was not enforced as expected.
    • Fix bug with acl endpoint URL.
    • Fix issue where sharing namespace settings could be ignored. More work may be needed. (Unit testing desperately needed.)
    • Fix some incorrect/misleading CLI argument help.
    • Internal improvements for long-term maintenance.
  • Shell scripts: If you need the ksconf version from a shell script, it's possible to use python -m ksconf.version to spit out shell variables to get version information. This previously worked with ksconf._version, but that was never publicly documented and should be avoided in the future.

Ksconf v0.13.2 (2023-10-10)

  • Add --token to allow bearer authentication for rest-publish.

Ksconf v0.13.1 (2023-10-05)

  • Removed overlooked debug message at startup.

Ksconf v0.13.0 (2023-10-05)

  • Switching to python package namespaces for for ksconf and ksconf.commands and created ksconf.plugins. This allows for more flexible packaging of various ksconf components including optional subcommands and plugins. Unless you are a python developer, you should never notice a difference. Splunk App users should re-install to avoid any confusion.

Ksconf 0.12

Highlights:

  • Add new ksconf subcommands for very basic, but previously missing, :ref:`ksconf_cmd_attr-get` and :ref:`ksconf_cmd_attr-set`. These commands allow for easy target information for capture or update.
  • Pre-commit hooks have been moved into their own ksconf-pre-commit repo. To allow time for migration to the new repo, the existing hooks will remain for a few release before being removed. To migrate, simply add -pre-commit to the end of the repo field, and update rev to v0.12.0 or later.

Packaging changes:

  • Dropped hard lxml from requirements. This is still handled automatically when using the pre-commit hooks (from the new repository). But this may be missing. To get access to all CLI functionality, run pip install ksconf[thirdparty], or for the full experience use pip install ksconf[fully-loaded]
  • Remove the use of the endpoints python package and shift to using importlib.metadata (or the equivalent backport), as it suggested by the original author of that package. The original necessity of this library was to workaround performance issues in pkg_resources (and the fact that it's no present in Splunk's embedded python. This move reduces code complexity but it does mean some additional runtime dependencies on older versions of Python. In many cases, this really isn't a new dependency, since pluggy requires it as well.

Ksconf v0.12.3 (2023-10-05)

  • Removed overlooked debug message at startup.

Ksconf v0.12.2 (2023-10-05)

  • App building with Ksconf: Added a convenience method to allow running ksconf commands more easily during the build process. You can now invoke ksconf using :py:meth:`~ksconf.build.BuildStep.run_ksconf` method which allows direct execution of a ksconf command. Previously this was accomplished by using :py:meth:`~ksconf.build.BuildStep.run`, using Python interpreter internal path as the executable, launching the ksconf in "module" mode. So this approach is simpler and in the future it may be invoked internally, removing the need for launching an additional Python process.
  • Enhanced plugin error handling.
  • Many little doc build fixes.

Ksconf v0.12.1 (2023-10-03)

  • Introducing :ref:`ksconf_cmd_attr-get` and :ref:`ksconf_cmd_attr-set` - the newest and simplest ksconf commands ever! Use this to quickly grab and/or update a specific stanzas, attribute combination from a conf file.
  • Add new attribute-level matching logic to ksconf filter. Use --attr-matches and/or --attr-not-matches to match specific attribute and value combinations for stanza matching. This can be used to find props with a specific KV_MODE, find saved search containing a specific search command, or list indexes not using volume: designation. See the :ref:`ksconf_cmd_filter` docs for example usage. Thanks to yohonet for inspiring this change, along with the new attr-get command.
  • Add --in-place processing behavior for :ref:`ksconf_cmd_merge` to simplify the process of merging new content into an existing conf file.
  • Docs Improvements: * Fixed documentation generation bug that prevented command line options from showing up in the per-command doc pages. (Broken since v0.10) * Fixed docs embedded in the Splunk app (and possibly causing some other display issues on the main rtd site)
  • Fixed some CLI file handling bug that resulted in broken use of - (stdin) and/or fancy shell commands involving <(some command) syntax, which can be a helpful trick to reduce the number of temporary files.

Ksconf v0.12.0 (2023-09-27)

Ksconf 0.11

Highlights:

  • Ksconf is beginning to treat Splunk apps more holistically and not just as a collection of .conf files.
  • Significant portions of this new code base is directly leveraged by the Ansible modules located in the `cdillc.splunk`_ collection, a sibling project to Ksconf. some of the code code there has made it's way into the core ksconf project in this release.

API Changes

Ksconf v0.11.9 (2023-09-26)

  • Splunk app packaging changes only. No need to update the package for CLI usage.
  • Fix packaging bug impacting the Ksconf Splunk app. This issue was introduced in v0.11.6. Thanks to yohonet for brining this to my attention.
  • Note that this took two release cycles to fully resolve. (Ignore ksconf v0.11.8)

Ksconf v0.11.7 (2023-09-20)

  • Support disabling of plugins by name via KSCONF_PLUGIN_DISABLE environment variable. This expects a space separated lists of plugin names.
  • Add new plugins documentation.

Ksconf v0.11.6 (2023-09-20)

  • Introducing plugin functionality using `pluggy`_ plugin management system. This adds a small, single-package dependency that can greatly increase customization potential of ksconf. The first demo of this can be seen in the ksconf-jinja-markdown package that enables .j2 payloads to be rendered by registering a custom Jinja filter named markdown2html.

Ksconf v0.11.5 (2023-08-25)

  • Use atomic file operations for more updates (such as most .conf writing and app packaging). This is enabled by the new context managers :py:func:`~ksconf.util.file.atomic_writer` and :py:func:`~ksconf.util.file.atomic_open`. Under the covers temporary files are written to and then renamed into place to ensure that the output file is either fully updated or not touched at all. This should reduce possible data loss and/or confusion during some difficult to handle corner cases (i.e., disk full, or interrupted execution).
  • Add new :py:func:`~ksconf.conf.parser.parse_string` function to enable simple parsing from a string. (I'm not sure why it took me so long to add this; it's so simple; and I've looked for this function dozens of times over the years, and always came up with a new StringIO workaround.)
  • App Manifest changes:
    • App manifest file format was updated to v2 which reports file mode as as familiar octal (string) instead of an integer. This is easier to read in most cases.
    • Add API level improvements to support reading/writing manifests when the archive is using a temporary filename.
  • Improved various error messages, minor optimizations, and some minor security improvements.

Ksconf v0.11.4 (2023-06-09)

  • Updated Jinja2 support to allow variables to be passed in for the combine and package operations. To use Jinja2 rendering feature, use the --enable-handler=jinja option. Forcing users to op-in to this behavior seems to be safest option moving forward. To feed variables into the Jinja2 rendering process, use the --template-vars option. Either pass a literal value or @filename. Currently JSON literals as well as .json, .yaml and .conf files.
  • Minor output fixes for combine (failed variable substitution)
  • Rename LayerConfig to :py:class:`~ksconf.layer.LayerContext`. I doubt anyone is using this, but just in case.

Ksconf v0.11.3 (2023-05-17)

  • Initial support for rendering Jinja2 templates in app layers, which can be used for app packaging and combine operations. This isn't really exposed via the CLI yet. Notice that if you have *.j2 files in your apps that are NOT Jinja2 templates, this could cause problems for you. There's no way to disable this functionality as of yet.
  • Move .conf and .spec combine handlers into handler functions. This makes it easier to supporting additional file types with special merge-handling logic in the future and simplifies the code into smaller units.
  • More internal embracing of pathlib and typing.

Ksconf v0.11.0 (2023-05-13)

  • First release of all ksconf.app.* functionality!
  • The unarchive command has been updated to use the new functionality.
  • More embracing of Python 3, f-strings, and pathlib!

NOTE: If you don't need for the new Splunk app functionality, there's little value in upgrading to v0.11. There's no new CLI features exposing this new functionality yet.

Disregard version v0.11.1, and v0.11.2 has minor internal fixes and shouldn't be broadly used. They were also released on 2023-05-13.

Ksconf 0.10

Highlights:

  • Ksconf now requires Python 3.7 or newer.
  • The Python package was renamed ksconf.

API Changes

Ksconf v0.10.2 (2023-05-13)

  • Fix an unarchive bug triggered by trailing slashes in --app-name. Trailing slashes are removed automatically. Any other / present will trigger an error and require the user to clarify.

Ksconf v0.10.1 (2023-03-07)

  • Fix for pre-commit hook installation. Pre-commit now uses an alternate shallow clone that fails to fetch the actual tag pointed to by rev. The ksconf python packaging process relies on git metadata. This results in an error message InvalidVersion: Invalid version: 'cec3615' in the pre-commit.log file. See pre-commit #2610 for additional background.

Ksconf v0.10.0 (2023-03-03)

Highlights:

  • The official Python package was renamed ksconf. The kintyre-splunk-conf package will continue to be released in parallel at least until version 1.0. You can continue updating and using the kintyre-splunk-conf package but eventually startup warnings will be added to remind users to switch.
  • Ksconf v0.10 and later requires Python 3.7 or newer. If you need Python 2.7 or 3.6 support, please stick with the latest 0.9.x release of kintyre-splunk-conf.
  • The KSCONF acronym has taken on a new meaning. Originally, meaning Kintyre's Splunk CONFiguration tool, now becomes a recursive acronym: Ksconf Splunk CONFiguration tool. Kintyre has been acquired by CDI LLC, and this option seemed least intrusive.
  • Add support Dashboard Studio dashboards. The JSON blobs inside of Simple XML payloads can now be formatted too. Multiline searches are still difficult to diff, but there's no way to fix that while using JSON.

More changes:

  • Remove six built-in dependency.
  • Refactor the combine logic into more reusable classes. This simplifies the CLI logic for both the combine and package functionality. The combine CLI and functionality remains unchanged.
  • Updated package to use new combine new layer classes rather than making internal CLI calls to "combine". This has the potentially to be more efficient and allow for easier functionality expansions in the future.
  • Deprecated the --layer_method=auto choice from the package command. This will be an error in the next release (v0.11).
  • Add new layer-related dynamic variables for the package command. Use {{layers_list}} to capture what unique layer names made their way into a tarball, and use {{layers_hash}} when that list get too long to be manageable.

Bug fixes:

  • Fixed sort bug where the user was incorrectly told that a file with errors was unexpectedly also successfully replaced. The contradictory output messages have been cleaned up. For clarity, this only occurred for inline replacement mode, and was purely a reporting issue, not a file handling problem.
  • Fixed compatibility issues with rest-publish command and the splunk-sdk library around data type expectations. A big thanks to bayeslearner (#95) for the fix. If you run into any issues, try upgrading your version of splunk-sdk.

Ksconf 0.9

Highlights:

  • Last version to support Python 2! It's time.

API Changes

Ksconf v0.9.3 (2022-02-26)

  • Added internal caching for :py:class:`~ksconf.package.AppVarMagic` (ksconf package command) to reduce repeated variable expansion work. This will likely go unnoticed by most, but it does speed up some operations in the cdillc.splunk.ksconf_package Ansible module.
  • Minor docs corrections.

Ksconf v0.9.2 (2022-03-04)

  • The filter command can now include/exclude stanzas based on the boolean value of disabled using the new --enabled-only or --disabled-only arguments. The default behavior remains the same, that is, the disabled attribute is completely ignored. Thanks to John B Splunker for inspiring this feature!

Ksconf v0.9.1 (2022-03-03)

  • Ksconf now tries harder to preserve file modification times. This is supported in merge, combine and package commands. Specifically, merged .conf files and concatenated files will keep the most recent modification time in the destination. This should make the output of combine and package (by extension) more deterministic in many scenarios.

Ksconf v0.9.0 (2021-08-12)

Features & Enhancements:

  • Add new --keep-existing option for ksconf combine to preserve certain files that exist within the target directory but not within any source. Similarly the new --disable-cleanup option will prevent any files from being removed. This is useful, for example if using ksconf combine to write apps into deployment-apps where Splunk automatically creates a local app.conf file, and the deletion and recreation of the file can result in unnecessary app re-deployments. These new options can be used together; for example, one useful pattern is to use --disable-cleanup to block all removals while perfecting/testing --keep-existing patterns.
  • Add support for previewing stanza changes with ksconf promote by combining --stanza X and --summary options at the same time. Thanks to guilhemmarchand for the suggestion. (#89)
  • New CLI args for ksconf diff. (1) New --detail option to specify how to handle certain 'replace' levels which impacts the way certain changes are represented. (2) New --format json for a more parsable output format. Note: This json format shouldn't be considered stable at this time. If you have ideas about how this could be used, please reach out.
  • Allow enabling/disabling TTY colors via environmental variable. The new --disable-color option will disable color, or to disable more widely, add something like export KSCONF_TTY_COLOR=off to your bashrc profile or Windows environment variables.

Bug fixes:

  • Fixed layer detection bugs for dir.d mode for layers. (1) Layers that weren't immediately under the source directory were not detected, and (2) layers existing beyond a symlink were not detected. This change targeted for ksconf combine but may fix other similar issues.
  • Fixed #91. where ksconf diff wouldn't correctly handle empty stanzas in the second input file (Reversing the order would sometimes worked to avoid the issue). This was resolved by enabling some improved empty stanza handling in the conf comparison algorithms that were updated back in 0.7.10, but never globally applied. This has been resolved.

Documentation improvements

  • New git tip: Use a gitdir: pointer to relocate the .git dir to avoid replicating it when a directory like master-apps is a git working copy.
  • Additional quick use case in the cheatsheet page. Demonstrate how ksconf could be used to list all "apps" present on a deployment server from the serverclass.conf file.

API Change:

  • Replaced use of match_bwlist() with the :py:class:`~ksconf.filter.FiltedListSplunkGlob` class, which allows old code to be cleaned up and technically, there's some expanded capabilities because of this (like many filters now supporting file://filter.txt type syntax, but this hasn't been documented and may be left as an Easter egg; because who reads changelogs?)
  • Dropped tty_color() which had already been replaced with the TermColor class.

Ksconf 0.8

Highlights:

  • New command :ref:`ksconf_cmd_package` is designed for both Splunk developers and admins * New module :py:mod:`ksconf.builder` helps build Splunk apps using a pipeline; or when external Python libraries are bundled into an app
  • Legit layer support with built-in layer filtering capabilities is available in several commands
  • Python 3! Head's up: We'll be dropping support for Python 2 in an upcoming release

Note

Come chat about ksconf on GitHub discussions even if it's to say we should use some other forum to stay in touch.

What's new:

  • The new ksconf package command supports the creation of Splunk app .spl files from a source directory. The package command can be used by admins to transfer apps around an organization, while keeping the local folder intact, or by a developer who wants local to be automatically merged into default. The app version can be set based on the latest git tag by simply saying --set-version={{git_tag}}.
  • The ksconf.builder Python module is a API-only first for ksconf! This build library allow caching of expensive deterministic build operations, and has out-of-the-box support for frequent build steps like adding Python modules locally using pip. As the first feature with no CLI support, I'm exceeded to get input from the broader community on this approach. Of course this is just an experimental first release. As always, feedback welcome!
  • Native support for layers! It's official, layers are now a proper ksconf feature, not just an abstract concept that you could throw together yourself given enough time and effort. This does mean that ksconf has to be more opinionated, but the design supports switching layer methods, which can be extended over time to support new different strategies as they emerge and are embraced by the community. Supports layers filtering as a native feature. This has always been technically possible, but awkward to implement yourself. Layer support is currently available in :ref:`ksconf_cmd_combine` and :ref:`ksconf_cmd_package` commands.
  • Moving to Python 3 soon. In preparation for the move to Python 3, I've added additional backport libraries to be installed when running Python 2. Support for Python 2 will be dropped in a future release, and anyone still on Splunk 7 who can't get a Python 3 environment will have to use an older version of ksconf. Also note that when jumping to Python 3, we will likely be requiring Python 3.6 or newer right out of the gate. (This means dropping Python 2.7, 3.4 and 3.5 all at the same time.) Whoohoo for f-strings!
  • CLI option abbreviation has been disabled. This could be a breaking change for existing scripts. Hopefully no one was relying on this already, but in order to prevent long-term CLI consistency issues as new CLI arguments are added, this feature has been disabled for all version of Python. This feature is only available, and was enabled by default, starting in Python 3.5.
  • Removed insensitive language. Specifically the terms 'whitelist' and 'blacklist' have been replaced, where possible. Fortunately, these terms were not used in any CLI arguments, so there should be no user-facing changes as a result of this.
  • Removed support for building a standalone executable (zipapp). This packaging option was added in v0.4.3, and deprecated in v0.6.0 once the Splunk app install option became available. I'm pretty sure this won't be missed.

API Changes

Developer changes:

  • Formatting via autopep8 and isort (enforced by pre-commit)
  • Better flake8 integration for bulk checking (run via: tox -e flake8,flake8-unittest)

Ksconf v0.8.7 (2020-04-29)

  • Support combining *.conf.spec files in ksconf combine, thus allowing README.d to be it's own layer.
  • Fixed potential unarchive issue with older version of git where git add --all DIR is more explicit, but equivalent to the modern day, git add DIR.

Ksconf v0.8.6 (2020-04-20)

  • Fixed install.py Splunk app CLI install helper script to support referencing a specific version of Python. This is needed on Splunk 8.0 if you'd like to use Python 3 (or Splunk 8.1 if you want to use Python 2.7, but please don't.) I suppose this would also work with using a custom Python interpreter other than the ones Splunk ships with, but then why not install with pip, right? (Thanks to guilhem.marchand for bringing this issue to my attention.)
  • Updated docs regarding changes to the use of install.py and fixed a bunch of spelling mistakes and other minor doc/comment tweaks.
  • Fixed ASCII art issue.

Ksconf v0.8.5 (2020-04-07)

  • Fixed packaging issue where external dependencies were missing. This doesn't impact the Splunk package install, or anyone running Python 3.6 or later.

Ksconf v0.8.4 (2020-03-22)

  • CLI change: Replaced short option for --allowlist to be -a, before it was -w. I assume this was left over early development where the argument was initial called --whitelist, but at this point -w is just confusing. Normally, I'd keep -w for a period of time and issue a deprecation warning. However, given that 0.8.0 was released less than a week ago, and that ksconf package is an "alpha" feature, I'm going to make this change without prior warning.
  • Add some safety checks to the package command to check for app naming issues (where the app folder doesn't match [package] id value in app.conf), and hidden files and directories.
  • Add new {{app_id}} variable that's usable with the ksconf package command.
  • Added a new optional argument to :py:func:`~ksconf.builder.steps.copy_files` called target for additional control over the destination path of artifacts copied into the build folder.
  • Minor tweak to unhandled exceptions. The name of the exception class is now show, and may be helpful in some situations.
  • When using make_missing in :py:class:`~ksconf.conf.parser.update_conf`, missing directories will now be created too.
  • Additional fixes to the Ksconf for Splunk App build.py script: Now explicitly creating a top-level ksconf folder. It's likely that this was the root cause of several other issues.

Ksconf v0.8.3 (2021-03-20)

  • Fixed bugs created by v0.8.2 (yanked on pypi)
  • Properly resolved issues with Splunk app building process.
  • Open issue uncovered where ksconf package can produce a tarball that's unusable by Splunkbase.

Ksconf v0.8.1 (2021-03-20)

  • Fixed some build issues with the Splunk app. (The splunk app is now built with ksconf package and the ksconf.builder)
  • Minor doc fix up; you know, the stuff typically found minutes after any new release :-)

Ksconf v0.8.0 (2021-03-19)

In addition to the 0.8 summary above, 0.8.0 specifically includes the following changes:

  • Add automatic layer support. Currently the two supported layer schemes are (1) explicit layers (really this will disable automatic layer detection), and (2) the dir.d format which uses the default.d/##-layer-name style directory support, which we previously promoted in the docs, but never really fully supported in a native way. This new dir.d directory layout support also allows for multiple *.d folders in a single tree (so not just default.d), and if your apps have different layer-points in different apps, it's all handled transparently.
  • Layer selection support was added to the combine command. This allows you to --include and --exclude layers as you see fit. See the docs for more details and examples of this new functionality. This works for both the new dir.d directories and the explicit layers, though moving to the dir.d format is highly encouraged.
  • New cheatsheet example: Using ksconf package and splunk install app together.
  • Updated the combine behavior to optimize for the situation where there is only a single conf input file provided. This behavior leaves any .conf or .meta file untouched so there's no sorting/normalizing or banner. See #64.
  • Eliminated an "unknown command" error when one of the ksconf python modules has a SyntaxError. The new behavior isn't perfect (you may still see "unrecognized arguments"), but overall it's still a step in the right direction.

Ksconf 0.7.x

New functionality, massive documentation improvements, metadata support, and Splunk app install fixes.

Release v0.7.10 (2021-03-19)

  • Fixed bug where empty stanzas in the local file could result in deletion in default with ksconf promote. Updated diff interface to improve handling of empty stanzas, but wider support is still needed across other commands; but this isn't a high priority.

Release v0.7.9 (2020-09-23)

  • Fixed bug where empty stanzas could be removed from .conf files. This can be detrimental for capability::* entries in authorize.conf, for example. A big thanks to nebffa for tracking down this bug!

Release v0.7.8 (2020-06-19)

  • New automatic promote mode is now available using CLI arguments! This allows stanzas to be selected for promotion from the CLI in batch and interactive modes. This implementation borrows (and shares code) with the ksconf filter command so hopefully the CLI arguments look familiar. It's possible to promote a single stanza, a stanza wildcard, regex or invert the matching logic and promote everything except for the named stanza (blocklist). Right now --stanza is the only supporting matching mode, but more can be added as needed. A huge thanks to mthambipillai for providing a pull-request with an initial implementation of this feature!
  • Added a new summary output mode (ksconf promote --summary) that will provide a quick summary of what content could be promoted. This can be used along side the new --stanza filtering options to show the names of stanzas that can be promoted.
  • Replaced insensitive terminology with race-neutral terms. Specifically the terms 'blacklist' and 'whitelist' have been replaced. NOTE: This does not change any CLI attributes, but in a few cases the standard output terminology is slightly different. Also terminology in .conf files couldn't be updated as that's controlled by Splunk.
  • Fixed bug in the unarchive command where a locale folder was blocked as a local folder and where a nested default folder (nested under a Python package, for example) could get renamed if --default-dir was used, now only the top-most default folder is updated. Also fixed an unlikely bug triggered when default/app.conf is missing.
  • Fixed bug with minimize when the required --target argument is not given. This now results in a reminder to the user rather than an unhandled exception.
  • Splunk app packaging fix. Write access to the app was previously not granted due to a spelling mistake in the metadata file.

Release v0.7.7 (2020-03-05)

  • Added new --follow-symlink option to the combine command so that input directory structures with symbolic links can be treated the same as proper directories.
  • Corrected Windows issue where wildcard (glob) patterns weren't expanded by for check and sort. This is primarily a difference in how a proper shells (e.g., bash, csh, zsh) handle expansion natively vs CMD on Windows does not. However, since this is typically transparently handled by many CLI tools, we'll follow suite. (BTW, running ksconf from the GIT Bash prompt is a great alternative.) Only the most minimalistic expansion rules will be available, (so don't expect {props,transforms,app}.conf to work anytime soon), but this should be good enough for most use cases. Thanks to SID800 for reporting this bug.
  • Fixed issues with the unarchive command when git is not installed or an app is being unarchived (installed/upgrade) into a location not managed by Git. Note that additional output is now enabled when the KSCONF_DEBUG environmental variable is set (in lieu of a proper verbose mode). Bug report provided by SID800.
  • Enhanced ksconf --version output to include Git executable path and version information; as well as a platform dump. (Helpful for future bug reporting.)
  • Added feature to disable the marker file (safety check) automatically created by the combine command for use in automated processing workflows.
  • Updated pre-commit documentation and sample configurations to use rev rather than sha as the means of identifying upstream tags or revisions. Recent releases of pre-commit will warn you about this during each run.
  • Fixed a temporary file cleanup issue during certain in-place file replacement operations. (If you found any unexpected *.tmp files, this could have been the cause.)

Release v0.7.6 (2019-08-15)

  • Fresh review and cleanup of all docs! (A huge thank you to Brittany Barnett for this massive undertaking)
  • Fixed unhandled exception when encountering a global stanza in metadata files.
  • Expand some error messages, sanity checks, and added a new session token (--session-key) authentication option for rest-publish.

Release v0.7.5 (2019-07-03)

  • Fixed a long-term bug where the diff output of a single-line attribute change was incorrectly represented in the textual output of 'ksconf diff' and the diff output in other commands. This resolves a combination of bugs, the first half of which was fixed in 0.7.3.
  • Allow make_docs script to run on Windows, and other internal doc build process improvements.

Release v0.7.4 (2019-06-07)

  • Inline the six module to avoid elusive bootstrapping cases where the module couldn't be found. This primarily impacts pre-commit users. The ksconf.ext.* prefix is being used for this, and any other inlined third party modules we may need in the future.
  • Other minor docs fixes and internal non-visible changes.

Release v0.7.3 (2019-06-05)

  • Added the new :ref:`ksconf_cmd_xml-format` command.
    • The ksconf xml-format command brings format consistency to your XML representations of Simple XML dashboards and navigation files by fixing indentation automatically adding <![CDATA[ ... ]]> blocks, as needed, to reduce the need for XML escaping, resulting in more readable source.
    • Additionally, a new pre-commit hook named :ref:`ksconf-xml-format <ksconf_pre_commit>` was added to leverage this new functionality. It looks specifically for xml views and navigation files based on path. This may also include Advanced XML, which hasn't been tested; So if you use Advanced XML, proceed with caution.
    • Note that this adds lxml as a packaging dependency which is needed for pre-commit hooks, but not strictly required at run time for other ksconf commands. This is NOT ideal, and may change in the future in attempts to keep ksconf as light-weight and standalone as possible. One possible alternative is setting up a different repo for pre-commit hooks. Python packaging and distribution tips welcome.
  • Fixed data loss bug in promote (interactive mode only) and improved some UI text and prompts.
  • Fixed colorization of ksconf diff output where certain lines failed to show up in the correct color.
  • Fixed bug where debug tracebacks didn't work correctly on Python 2.7. (Enable using KSCONF_DEBUG=1.)
  • Extended the output of ksconf --version to show the names and version of external modules, when present.
  • Improved some resource allocation in corner cases.
  • Tested with Splunk 7.3 (numeric similarity in version numbers is purely coincidental)

Attention!

API BREAKAGE

The DiffOp output values for DIFF_OP_INSERT and DIFF_OP_DELETE have been changed in a backwards-compatible breaking way. The values of a and b were previously reversed for these two operations, leading to some code confusion.

Release v0.7.2 (2019-03-22)

  • Fixed bug where filter would crash when doing stanza matching if global entries were present. Global stanzas can be matched by searching for a stanza named default.
  • Fixed broken pre-commit issue that occurred for the v0.7.1 tag. This also kept setup.py from working if the six module wasn't already installed. Developers and pre-commit users were impacted.

Release v0.7.1 (2019-03-13)

  • Additional fixes for UTF-8 BOM files which appear to happen more frequently with local files on Windows. This time some additional unit tests were added so hopefully there are few regressions in the future.
  • Add the ignore-missing argument to :ref:`ksconf_cmd_merge` to prevent errors when input files are absent. This allows bashisms Some_App/{{default,local}}/savedsearches.conf to work without errors if the local or default file is missing.
  • Check for incorrect environment setup and suggest running sourcing :file:`setSplunkEnv` to get a working environment. See #48 for more info.
  • Minor improvements to some internal error handling, packaging, docs, and troubleshooting code.

Release v0.7.0 (2019-02-27)

Attention!

For anyone who installed 0.6.x, we recommend a fresh install of the Splunk app due to packaging changes. This shouldn't be an issue in the future.

General changes:

  • Added new :ref:`ksconf_cmd_rest-publish` command that supersedes the use of rest-export for nearly every use case. Warning: No unit-testing has been created for this command yet, due to technical hurdles.
  • Added :doc:`cheatsheet` to the docs.
  • Massive doc cleanup of hundreds of typos and many expanded/clarified sections.
  • Significant improvement to entrypoint handling and support for conditional inclusion of 3rd party libraries with sane behavior on import errors, and improved warnings. This information is conveniently viewable to the user via ksconf --version.
  • Refactored internal diff logic and added additional safeties and unit tests. This includes improvements to TTY colorization which should avoid previous color leaks scenarios that were likely if unhandled exceptions occur.
  • New support for metadata handling.
  • CLI change for rest-export: The --user argument has been replaced with --owner to keep clean separation between the login account and object owners. (The old argument is still accept for now.)

Splunk app changes:

  • Modified installation of python package installation. In previous releases, various .dist-info folders were created with version-specific names leading to a mismatch of package versions after upgrade. For this reason, we suggest that anyone who previously installed 0.6.x should do a fresh install.
  • Changed Splunk app install script to install.py (it was bootstrap_bin.py). Hopefully this is more intuitive.
  • Added Windows support to install.py.
  • Now includes the Splunk Python SDK. Currently used for rest-publish but will eventually be used for additional functionally unique to the Splunk app.

Ksconf 0.6.x

Add deployment as a Splunk app for simplicity and significant docs cleanup.

Release v0.6.2 (2019-02-09)

Release v0.6.1 (2019-02-07)

  • (Trivial) Fixed some small issues with the Splunk App (online AppInspect)

Release v0.6.0 (2019-02-06)

  • Add initial support for building ksconf into a Splunk app.
    • App contains a local copy of the docs, helpful for anyone who’s working offline.
    • Credit to Sarah Larson for the ksconf logos.
    • No ksconf functionality exposed to the Splunk UI at the moment.
  • Docs/Sphinx improvements (more coming)
    • Begin work on cleaning up API docs.
    • Started converting various document pages into reStructuredText for greatly improved docs.
    • Improved PDF fonts and fixed a bunch of sphinx errors/warnings.
  • Refactored the install docs into 2 parts. With the new ability to install ksconf as a Splunk app it’s quite likely that most of the wonky corner cases will be less frequently needed, hence all the more exotic content was moved into the “Advanced Install Guide”, tidying things up.

Ksconf 0.5.x

Add Python 3 support, new commands, support for external command plugins, tox and vagrant for testing.

Release v0.5.6 (2019-02-04)

  • Fixes and improvements to the filter command. Found issue with processing from stdin, inconsistency in some CLI arguments, and finished implementation for various output modes.
  • Add logo (fist attempt).

Release v0.5.5 (2019-01-28)

  • New :ref:`ksconf_cmd_filter` command added for slicing up a conf file into smaller pieces. Think of this as GREP that’s stanza-aware. Can also allow or block attributes, if desirable.
  • Expanded rest-export CLI capabilities to include a new --delete option, pretty-printing, and now supports stdin by allowing the user to explicitly set the file type using --conf.
  • Refactored all CLI unittests for increased readability and long-term maintenance. Unit tests now can also be run individually as scripts from the command line.
  • Minor tweaks to the snapshot output format, v0.2. This feature is still highly experimental.

Release v0.5.4 (2019-01-04)

  • New commands added:
    • :ref:`ksconf_cmd_snapshot` will dump a set of configuration files to a JSON formatted file. This can be used used for incremental "snapshotting" of running Splunk apps to track changes overtime.
    • :ref:`ksconf_cmd_rest-export` builds a series of custom curl commands that can be used to publish or update stanzas on a remote instance without file system access. This can be helpful when pushing configs to Splunk Cloud when all you have is REST (splunkd) access. This command is indented for interactive admin not batch operations.
  • Added the concept of command maturity. A listing is available by running ksconf --version
  • Fix typo in KSCONF_DEBUG.
  • Resolving some build issues.
  • Improved support for development/testing environments using Vagrant (fixes) and Docker (new). Thanks to Lars Jonsson for these enhancements.

Release v0.5.3 (2018-11-02)

  • Fixed bug where ksconf combine could incorrectly order directories on certain file systems (like ext4), effectively ignoring priorities. Repeated runs may resulted in undefined behavior. Solved by explicitly sorting input paths forcing processing to be done in lexicographical order.
  • Fixed more issues with handling files with BOM encodings. BOMs and encodings in general are NOT preserved by ksconf. If this is an issue for you, please add an enhancement issue.
  • Add Python 3.7 support
  • Expand install docs specifically for offline mode and some OS-specific notes.
  • Enable additional tracebacks for CLI debugging by setting KSCONF_DEBUG=1 in the environment.

Release v0.5.2 (2018-08-13)

  • Expand CLI output for --help and --version
  • Internal cleanup of CLI entry point module name. Now the ksconf CLI can be invoked as python -m ksconf, you know, for anyone who’s into that sort of thing.
  • Minor docs and CI/testing improvements.

Release v0.5.1 (2018-06-28)

  • Support external ksconf command plugins through custom entry_points, allowing for others to develop their own custom extensions as needed.
  • Many internal changes: Refactoring of all CLI commands to use new entry_points as well as pave the way for future CLI unittest improvements.
  • Docs cleanup / improvements.

Release v0.5.0 (2018-06-26)

  • Python 3 support.
  • Many bug fixes and improvements resulting from wider testing.

Ksconf 0.4.x

Ksconf 0.4.x switched to a modular code base, added build/release automation, PyPI package registration (installation via pip install and, online docs.

Release v0.4.10 (2018-06-26)

  • Improve file handling to avoid “unclosed file” warnings. Impacted parse_conf(), write_conf(), and many unittest helpers.
  • Update badges to report on the master branch only. (No need to highlight failures on feature or bug-fix branches.)

Release v0.4.9 (2018-06-05)

  • Add some missing docs files

Release v0.4.8 (2018-06-05)

  • Massive cleanup of docs: revamped install guide, added ‘standalone’ install procedure and developer-focused docs. Updated license handling.
  • Updated docs configuration to dynamically pull in the ksconf version number.
  • Using the classic ‘read-the-docs’ Sphinx theme.
  • Added additional PyPi badges to README (GitHub home page).

Release v0.4.4-v0.4.7 (2018-06-04)

  • Deployment and install fixes (It’s difficult to troubleshoot/test without making a new release!)

Release v0.4.3 (2018-06-04)

  • Rename PyPI package kintyre-splunk-conf
  • Add support for building a standalone executable (zipapp).
  • Revamp install docs and location
  • Add GitHub release for the standalone executable.

Release v0.4.2 (2018-06-04)

  • Add readthedocs.io support

Release v0.4.1 (2018-06-04)

  • Enable PyPI production package building

Release v0.4.0 (2018-05-19)

  • Refactor entire code base. Switched from monolithic all-in-one file to clean-cut modules.
  • Versioning is now discoverable via ksconf --version, and controlled via git tags (via git describe --tags).

Module layout

  • ksconf.conf.* - Configuration file parsing, writing, comparing, and so on
  • ksconf.util.* - Various helper functions
  • ksconf.archive - Support for decompressing Splunk apps (tgz/zip files)
  • ksconf.vc.git - Version control support. Git is the only VC tool supported for now. (Possibly ever)
  • ksconf.commands.<CMD> - Modules for specific CLI functions. I may make this extendable, eventually.

Ksconf 0.3.x

First public releases.

Release v0.3.2 (2018-04-24)

  • Add AppVeyor for Windows platform testing
  • Add codecov integration
  • Created ConfFileProxy.dump()

Release v0.3.1 (2018-04-21)

  • Setup automation via Travis CI
  • Add code coverage

Release v0.3.0 (2018-04-21)

  • Switched to semantic versioning.
  • 0.3.0 feels representative of the code maturity.

Ksconf legacy releases

Ksconf started in a private Kintyre repo. There are no official releases; all git history has been rewritten.

Release legacy-v1.0.1 (2018-04-20)

  • Fixes to blocklist support and many enhancements to ksconf unarchive.
  • Introduces parsing profiles.
  • Lots of bug fixes to various subcommands.
  • Added automatic detection of ‘subcommands’ for CLI documentation helper script.

Release legacy-v1.0.0 (2018-04-16)

  • This is the first public release. First work began Nov 2017 (as a simple conf ‘sort’ tool, which was imported from yet another repo.) Version history was extracted/rewritten/preserved as much as possible.
  • Mostly stable features.
  • Unit test coverage over 85%
  • Includes pre-commit hook configuration (so that other repos can use this to run ksconf sort and ksconf check against their conf files.