Skip to content

Commit

Permalink
Move to .net standard (#63)
Browse files Browse the repository at this point in the history 
* Initial commit with all tests passing for the first time

* Added separation between es5,6 tests to es7 tests

* Fixed build location for tests projects

* Fixed integration tests

* Fixed test for es6

* Configured nuspec to use all frameworks

* Removed obsolete authentication from the appender

* Better csproj structure

* Updated dependencies

* Added ILRepack to reduce dependencies

* Fixed readme and removed unused files

* Fixed test for Custom Id logging

* Updated libraries, since there is no use for low versions with ILRepack

* Fixed a typo

* Better csproj structure

* Fixed CR comments from @urielha

* Better csproj structure for test projects

* Removed unnecessary files
  • Loading branch information
Eran Gil authored Apr 12, 2020
1 parent 3b9798c commit 5decf4a
Show file tree
Hide file tree
Showing 53 changed files with 364 additions and 1,696 deletions.
281 changes: 142 additions & 139 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,18 @@ log4stash

log4stash is a [log4net](http://logging.apache.org/log4net/) appender to log messages to the [ElasticSearch](http://www.elasticsearch.org) document database. ElasticSearch offers robust full-text search engine and analyzation so that errors and messages can be indexed quickly and searched easily.

log4stash provide few logging filters similar to the filters on [logstash](http://logstash.net).
log4stash provides a few logging filters similar to the filters on [logstash](http://logstash.net).

The origin of log4stash is [@jptoto](https://github.com/jptoto)'s [log4net.ElasticSearch](https://github.com/jptoto/log4net.ElasticSearch) repository.

log4stash is based on RestSharp and Newtonsoft.Json but uses ILRepack to avoid nuget dependencies.

**log4stash is fully open source, MIT licensed.**

### Features:
* Supports .NET 4.0+ and .NET Core
* Supports .NET 4.5.2+ and .NET Core 2.0+
* Easy installation and setup via [Nuget](https://nuget.org/packages/log4stash/)
* Ability to analyze the log event before sending it to elasticsearch using built-in filters and custom filters similar to [logstash](http://logstash.net/docs/1.4.2/).
* Ability to analyze the log event before sending it to ElasticSearch using built-in filters and custom filters similar to [logstash](http://logstash.net/docs/1.4.2/).

### Breaking Changes:
Navigate to breaking changes page [here](https://github.com/urielha/log4stash/blob/master/docs/breaking.md). See also [Version notes](https://github.com/urielha/log4stash/blob/master/docs/version_notes.md) page.
Expand All @@ -37,7 +41,7 @@ I do my best to reply to issues or questions ASAP. Please use the [ISSUES](https

### Configuration Examples:

Almost all the parameters are optional, to see the default values check the [c'tor](https://github.com/urielha/log4stash/blob/master/src/log4stash/ElasticSearchAppender.cs#L52) of the appender and the c'tor of every filter.
Almost all the parameters are optional, to see the default values check the [c'tor](https://github.com/urielha/log4stash/blob/master/src/log4stash/ElasticSearchAppender.cs#L86) of the appender and the c'tor of every filter.
You can also set any public property in the appender/filter which didn't appear in the example.

##### Simple configuration:
Expand All @@ -55,144 +59,142 @@ You can also set any public property in the appender/filter which didn't appear
##### (Almost) Full configuration:
```xml
<appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
<Server>localhost</Server>
<Port>9200</Port>
<!-- optional: in case elasticsearch is located behind a reverse proxy the URL is like http://Server:Port/Path, default = empty string -->
<Path>/es5</Path>
<IndexName>log_test_%{+yyyy-MM-dd}</IndexName>
<!-- optional: type support was removed in ElasticSearch 7, so if not defined in configuration there won't be a type in the request -->
<Server>localhost</Server>
<Port>9200</Port>
<!-- optional: in case elasticsearch is located behind a reverse proxy the URL is like http://Server:Port/Path, default = empty string -->
<Path>/es5</Path>
<IndexName>log_test_%{+yyyy-MM-dd}</IndexName>
<!-- type support was removed in ElasticSearch 7, so if not defined in configuration there won't be a type in the request -->
<IndexType>LogEvent</IndexType>
<Bulksize>2000</Bulksize>
<BulkIdleTimeout>10000</BulkIdleTimeout>
<IndexAsync>False</IndexAsync>
<DocumentIdSource>IdSource</DocumentIdSource> <!-- obsolete! use IndexOperationParams -->
<!-- Serialize log object as json (default is true).
<Bulksize>2000</Bulksize>
<BulkIdleTimeout>10000</BulkIdleTimeout>
<IndexAsync>False</IndexAsync>
<DropEventsOverBulkLimit>False</DropEventsOverBulkLimit>

<!-- Serialize log object as json (default is true).
-- This in case you log the object this way: `logger.Debug(obj);` and not: `logger.Debug("string");` -->
<SerializeObjects>True</SerializeObjects>
<SerializeObjects>True</SerializeObjects>

<!-- optional: elasticsearch timeout for the request, default = 10000 -->
<ElasticSearchTimeout>10000</ElasticSearchTimeout>

<!-- optional: elasticsearch timeout for the request, default = 10000 -->
<ElasticSearchTimeout>10000</ElasticSearchTimeout>
<!-- optional: ssl connection -->
<Ssl>False</Ssl>
<AllowSelfSignedServerCert>False</AllowSelfSignedServerCert>

<!--You can add parameters to the request to control the parameters sent to ElasticSearch.
for example, as you can see here, you can add a routing specification to the appender.
<!--You can add parameters to the request to control the parameters sent to ElasticSearch.
for example, as you can see here, you can add a custom id source to the appender.
The Key is the key to be added to the request, and the value is the parameter's name in the log event properties.-->
<IndexOperationParams>
<Parameter>
<Key>_routing</Key>
<Value>%{RoutingSource}</Value>
</Parameter>
<Parameter>
<Key>_id</Key>
<Value>%{IdSource}</Value>
</Parameter>
<Parameter>
<Key>key</Key>
<Value>value</Value>
</Parameter>
</IndexOperationParams>

<!-- for more information read about log4net.Core.FixFlags -->
<FixedFields>Partial</FixedFields>

<Template>
<Name>templateName</Name>
<FileName>path2template.json</FileName>
</Template>

<!--Only one credential type can used at once-->
<!--Here we list all possible types-->
<AuthenticationMethod>
<!--For basic authentication purposes-->
<Basic>
<Username>Username</Username>
<Password>Password</Password>
</Basic>
<!--For AWS ElasticSearch service-->
<Aws>
<Aws4SignerSecretKey>Secret</Aws4SignerSecretKey>
<Aws4SignerAccessKey>AccessKey</Aws4SignerAccessKey>
<Aws4SignerRegion>Region</Aws4SignerRegion>
</Aws>
<!-- For Api Key (X-Pack) authentication -->
<AuthenticationMethod>
<ApiKey>
<!-- ApiKeyBase64 takes precedence over Id/ApiKey -->
<ApiKeyBase64>aWQ6YXBpa2V5</ApiKey>
<!-- Or -->
<Id>id</Id>
<ApiKey>apikey</ApiKey>
</ApiKey>
</AuthenticationMethod>
</AuthenticationMethod>

<!-- all filters goes in ElasticFilters tag -->
<ElasticFilters>
<Add>
<Key>@type</Key>
<Value>Special</Value>
</Add>

<!-- using the @type value from the previous filter -->
<Add>
<Key>SmartValue</Key>
<Value>the type is %{@type}</Value>
</Add>

<Remove>
<Key>@type</Key>
</Remove>

<!-- you can load custom filters like I do here -->
<Filter type="log4stash.Filters.RenameKeyFilter, log4stash">
<Key>SmartValue</Key>
<RenameTo>SmartValue2</RenameTo>
</Filter>

<!-- converts a json object to fields in the document -->
<Json>
<SourceKey>JsonRaw</SourceKey>
<FlattenJson>false</FlattenJson>
<!-- the separator property is only relevant when setting the FlattenJson property to 'true' -->
<Separator>_</Separator>
</Json>

<!-- converts an xml object to fields in the document -->
<Xml>
<SourceKey>XmlRaw</SourceKey>
<FlattenXml>false</FlattenXml>
</Xml>

<!-- kv and grok filters similar to logstash's filters -->
<Kv>
<SourceKey>Message</SourceKey>
<ValueSplit>:=</ValueSplit>
<FieldSplit> ,</FieldSplit>
</kv>

<Grok>
<SourceKey>Message</SourceKey>
<Pattern>the message is %{WORD:Message} and guid %{UUID:the_guid}</Pattern>
<Overwrite>true</Overwrite>
</Grok>

<!-- Convert string like: "1,2, 45 9" into array of numbers [1,2,45,9] -->
<ConvertToArray>
<SourceKey>someIds</SourceKey>
<!-- The separators (space and comma) -->
<Seperators>, </Seperators>
</ConvertToArray>

<Convert>
<!-- convert given key to string -->
<ToString>shouldBeString</ToString>

<!-- same as ConvertToArray. Just for convenience -->
<ToArray>
<SourceKey>anotherIds</SourceKey>
</ToArray>
</Convert>
</ElasticFilters>
<IndexOperationParams>
<Parameter>
<Key>_id</Key>
<Value>%{IdSource}</Value>
</Parameter>
<Parameter>
<Key>key</Key>
<Value>value</Value>
</Parameter>
</IndexOperationParams>

<!-- for more information read about log4net.Core.FixFlags -->
<FixedFields>Partial</FixedFields>

<Template>
<Name>templateName</Name>
<FileName>path2template.json</FileName>
</Template>

<!--Only one credential type can be used at once-->
<!--Here we list all possible types-->
<AuthenticationMethod>
<!--For basic authentication purposes-->
<Basic>
<Username>Username</Username>
<Password>Password</Password>
</Basic>
<!--For AWS ElasticSearch service-->
<Aws>
<Aws4SignerSecretKey>Secret</Aws4SignerSecretKey>
<Aws4SignerAccessKey>AccessKey</Aws4SignerAccessKey>
<Aws4SignerRegion>Region</Aws4SignerRegion>
</Aws>
<!-- For Api Key (X-Pack) authentication -->
<ApiKey>
<!-- ApiKeyBase64 takes precedence over Id/ApiKey -->
<ApiKeyBase64>aWQ6YXBpa2V5</ApiKey>
<!-- Or -->
<Id>id</Id>
<ApiKey>apikey</ApiKey>
</ApiKey>
</AuthenticationMethod>

<!-- all filters goes in ElasticFilters tag -->
<ElasticFilters>
<Add>
<Key>@type</Key>
<Value>Special</Value>
</Add>

<!-- using the @type value from the previous filter -->
<Add>
<Key>SmartValue</Key>
<Value>the type is %{@type}</Value>
</Add>

<Remove>
<Key>@type</Key>
</Remove>

<!-- you can load custom filters like I do here -->
<Filter type="log4stash.Filters.RenameKeyFilter, log4stash">
<Key>SmartValue</Key>
<RenameTo>SmartValue2</RenameTo>
</Filter>

<!-- converts a json object to fields in the document -->
<Json>
<SourceKey>JsonRaw</SourceKey>
<FlattenJson>false</FlattenJson>
<!-- the separator property is only relevant when setting the FlattenJson property to 'true' -->
<Separator>_</Separator>
</Json>

<!-- converts an xml object to fields in the document -->
<Xml>
<SourceKey>XmlRaw</SourceKey>
<FlattenXml>false</FlattenXml>
</Xml>

<!-- kv and grok filters similar to logstash's filters -->
<Kv>
<SourceKey>Message</SourceKey>
<ValueSplit>:=</ValueSplit>
<FieldSplit> ,</FieldSplit>
</kv>

<Grok>
<SourceKey>Message</SourceKey>
<Pattern>the message is %{WORD:Message} and guid %{UUID:the_guid}</Pattern>
<Overwrite>true</Overwrite>
</Grok>

<!-- Convert string like: "1,2, 45 9" into array of numbers [1,2,45,9] -->
<ConvertToArray>
<SourceKey>someIds</SourceKey>
<!-- The separators (space and comma) -->
<Seperators>, </Seperators>
</ConvertToArray>

<Convert>
<!-- convert given key to string -->
<ToString>shouldBeString</ToString>

<!-- same as ConvertToArray. Just for convenience -->
<ToArray>
<SourceKey>anotherIds</SourceKey>
</ToArray>
</Convert>
</ElasticFilters>
</appender>
```

Expand All @@ -215,7 +217,8 @@ The inspiration to the filters and style had taken from [elasticsearch/logstash]

### Build status:

The CI will be soon to Azure DevOps and will be tested against ElasticSearch 5,6,7 every time.
The CI will be move soon to Azure DevOps and will be tested against ElasticSearch 5,6,7 every time.
Support for lower ElasticSearch versions is no longer maintained.

| Status | Provider |
| ------ | -------- |
Expand All @@ -237,6 +240,6 @@ The CI will be soon to Azure DevOps and will be tested against ElasticSearch 5,6
[docs-filters-remove]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Remove.md
[docs-filters-rename]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Rename.md

[erangil2]:https://github.com/erangil2
[ignasv]:https://github.com/ignasv
[erangil]:https://github.com/erangil2
[mfpalladino]:https://github.com/mfpalladino
5 changes: 5 additions & 0 deletions docs/breaking.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ description: List of breaking changes
redirect_from: "/breaking/"
--- -->

### __Upgrading to 3.0.0__
* BasicAuthUsername and BasicAuthPassword no longer exists as parameters and now needs to be configured through an authentication method.
* DocumentIdSource no longer exists as a parameter and now needs to be configured through IndexOperationParams
* Stopped support of .net 4.0


### __Upgrading to 2.0.4__
BasicAuthUsername and BasicAuthPassword moved under AuthenticationMethod.
Expand Down
3 changes: 3 additions & 0 deletions docs/version_notes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ description: List of version notes
redirect_from: "/version_notes/"
--- -->

### Versions 3.*
* :green_book: log4stash 3.0.0 targets .net standard natively, removed RestSharp dependency and added API Key authentication

### Versions 2.*

* :green_book: log4stash 2.0.4 has new filters see [filters section][filters-section] for more information. Added support for AWS Version 4 authorization header thanks to [@Marcelo Palladino][mfpalladino]. Support configuring multiple elastic nodes. Add document id source so you can decide your own doc id (instead of getting generated one from the elastic). Add Timeout option for requests. Many thanks to [@eran gil][erangil] for the efforts and the pull requests.
Expand Down
23 changes: 8 additions & 15 deletions log4stash.nuspec
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,34 +16,27 @@
</description>
<releaseNotes>
* Added option for API Key authentication with Elasticsearch

* Targets .net standard natively
Thanks a lot to @erangil2 for his tremendous contributions :)

And thanks to: @zhengziying, @triarc, @prime8, @henkish, @mfpalladino, @ignasv, @wpgh, @Thecentury, @pnumainville and @Aleksandre for their contributions.
And thanks to: @zhengziying, @triarc, @prime8, @henkish, @mfpalladino, @ignasv, @wpgh, @Thecentury, @pnumainville, @Aleksandre and @dannyyy for their contributions.
</releaseNotes>
<copyright>Copyright 2019</copyright>
<copyright>Copyright 2020</copyright>
<tags>log logging exception elasticsearch log4net appender logstash filters</tags>
<dependencies>

<group targetFramework=".NETFramework4.0">
<dependency id="log4net" version="2.0.8.0" />
<dependency id="RestSharp" version="105.2.3" />
<dependency id="Microsoft.Bcl.Async" version="1.0.168" />
</group>
<group targetFramework=".NETFramework4.5">
<group targetFramework=".NETFramework4.5.2">
<dependency id="log4net" version="2.0.8.0" />
<dependency id="RestSharp" version="105.2.3" />
</group>
<group targetFramework=".NETStandard2.0">
<dependency id="log4net" version="2.0.8.0" />
<dependency id="RestSharp" version="106.1.0" />
<dependency id="System.Security.Permissions" version="4.4.0" />
</group>
</dependencies>
</metadata>
<files>
<file src="lib\**\log4stash*.dll" target="lib"/>
<file src="lib\**\log4stash*.pdb" target="lib"/>
<file src="lib\net452\log4stash*.dll" target="lib\net452"/>
<file src="lib\net452\log4stash*.pdb" target="lib\net452"/>
<file src="lib\netstandard2.0\log4stash*.dll" target="lib\netstandard2.0"/>
<file src="lib\netstandard2.0\log4stash*.pdb" target="lib\netstandard2.0"/>
<file src="readme.txt" target="" />
<file src="icon.png" target="" />
</files>
Expand Down
Loading

0 comments on commit 5decf4a

Please sign in to comment.