Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

doc: update readme to markdown #96

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

doc: update readme to markdown #96

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 91 additions & 107 deletions README.rst → README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,16 @@
**Note: Maintainers needed : Those that committed in the past code to this repo or are presenting new PRs and have experience and interest on helping to maintain repos & python libraries (code quality, testing, integration, etc). If you are interested on getting our PR's through and helping others to contribute to the library, please get in touch.**

===============
CMRESHandler.py
===============
----

| |license| |versions| |status| |downloads|
| |ci_status| |codecov| |gitter|
[![downloads](https://img.shields.io/pypi/dd/CMRESHandler.svg)](https://pypi.python.org/pypi/CMRESHandler)
[![versions](https://img.shields.io/pypi/pyversions/CMRESHandler.svg)](https://pypi.python.org/pypi/CMRESHandler)
[![status](https://img.shields.io/pypi/status/CMRESHandler.svg)](https://pypi.python.org/pypi/CMRESHandler)
[![license](https://img.shields.io/pypi/l/CMRESHandler.svg)](https://pypi.python.org/pypi/CMRESHandler)
[![ci_status](https://travis-ci.org/cmanaha/python-elasticsearch-logger.svg?branch=master)](https://travis-ci.org/cmanaha/python-elasticsearch-logger)
[![codecov](https://codecov.io/github/cmanaha/python-elasticsearch-logger/coverage.svg?branch=master)](http://codecov.io/github/cmanaha/python-elasticsearch-logger?branch=master)
[![gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/cmanaha/python-elasticsearch-logger?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)


**Note: Maintainers needed : Those that committed in the past code to this repo or are presenting new PRs and have experience and interest on helping to maintain repos & python libraries (code quality, testing, integration, etc). If you are intereted on getting our PR's through and helping others to contribute to the library, please get in touch.**


Python Elasticsearch Log handler
********************************
# Python Elasticsearch Log handler

This library provides an Elasticsearch logging appender compatible with the
python standard `logging <https://docs.python.org/2/library/logging.html>`_ library.
Expand All @@ -20,58 +19,62 @@ The code source is in github at `https://github.com/cmanaha/python-elasticsearch
<https://github.com/cmanaha/python-elasticsearch-logger>`_


Installation
============
# Installation

Install using pip::

pip install CMRESHandler

Requirements Python 2
=====================
# Requirements Python 2

This library requires the following dependencies
- elasticsearch
- requests
- enum


Requirements Python 3
=====================
# Requirements Python 3
This library requires the following dependencies
- elasticsearch
- requests

Additional requirements for Kerberos support
============================================
Additionally, the package support optionally kerberos authentication by adding the following dependecy
# Additional requirements for Kerberos support

Additionally, the package support optionally kerberos authentication by adding the following dependency
- requests-kerberos

Additional requirements for AWS IAM user authentication (request signing)
=========================================================================
Additionally, the package support optionally AWS IAM user authentication by adding the following dependecy
# Additional requirements for AWS IAM user authentication (request signing)

Additionally, the package support optionally AWS IAM user authentication by adding the following dependency
- requests-aws4auth

Using the handler in your program
==================================
To initialise and create the handler, just add the handler to your logger as follow ::
# Using the handler in your program

from cmreslogging.handlers import CMRESHandler
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200}],
auth_type=CMRESHandler.AuthType.NO_AUTH,
es_index_name="my_python_index")
log = logging.getLogger("PythonTest")
log.setLevel(logging.INFO)
log.addHandler(handler)
To initialize and create the handler, just add the handler to your logger as follow ::

```python
from cmreslogging.handlers import CMRESHandler
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200}],
auth_type=CMRESHandler.AuthType.NO_AUTH,
es_index_name="my_python_index")
log = logging.getLogger("PythonTest")
log.setLevel(logging.INFO)
log.addHandler(handler)
```

You can add fields upon initialisation, providing more data of the execution context ::

from cmreslogging.handlers import CMRESHandler
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200}],
auth_type=CMRESHandler.AuthType.NO_AUTH,
es_index_name="my_python_index",
es_additional_fields={'App': 'MyAppName', 'Environment': 'Dev'})
log = logging.getLogger("PythonTest")
log.setLevel(logging.INFO)
log.addHandler(handler)
```python
from cmreslogging.handlers import CMRESHandler
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200}],
auth_type=CMRESHandler.AuthType.NO_AUTH,
es_index_name="my_python_index",
es_additional_fields={'App': 'MyAppName', 'Environment': 'Dev'})
log = logging.getLogger("PythonTest")
log.setLevel(logging.INFO)
log.addHandler(handler)
```

This additional fields will be applied to all logging fields and recorded in elasticsearch

Expand All @@ -82,18 +85,20 @@ To log, use the regular commands from the logging library ::
Your code can also dump additional extra fields on a per log basis that can be used to instrument
operations. For example, when reading information from a database you could do something like::

start_time = time.time()
database_operation()
db_delta = time.time() - start_time
log.debug("DB operation took %.3f seconds" % db_delta, extra={'db_execution_time': db_delta})
```python
start_time = time.time()
database_operation()
db_delta = time.time() - start_time
log.debug("DB operation took %.3f seconds" % db_delta, extra={'db_execution_time': db_delta})
```

The code above executes the DB operation, measures the time it took and logs an entry that contains
in the message the time the operation took as string and for convenience, it creates another field
called db_execution_time with a float that can be used to plot the time this operations are taking using
Kibana on top of elasticsearch

Initialisation parameters
=========================
# Initialization parameters

The constructors takes the following parameters:
- hosts: The list of hosts that elasticsearch clients will connect, multiple hosts are allowed, for example ::

Expand All @@ -118,54 +123,56 @@ The constructors takes the following parameters:
- es_doc_type: A string with the name of the document type that will be used ``python_log`` used by default
- es_additional_fields: A dictionary with all the additional fields that you would like to add to the logs

Django Integration
==================
# Django Integration

It is also very easy to integrate the handler to `Django <https://www.djangoproject.com/>`_ And what is even
better, at DEBUG level django logs information such as how long it takes for DB connections to return so
they can be plotted on Kibana, or the SQL statements that Django executed. ::

from cmreslogging.handlers import CMRESHandler
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': './debug.log',
'maxBytes': 102400,
'backupCount': 5,
},
'elasticsearch': {
'level': 'DEBUG',
'class': 'cmreslogging.handlers.CMRESHandler',
'hosts': [{'host': 'localhost', 'port': 9200}],
'es_index_name': 'my_python_app',
'es_additional_fields': {'App': 'Test', 'Environment': 'Dev'},
'auth_type': CMRESHandler.AuthType.NO_AUTH,
'use_ssl': False,
},
```python
from cmreslogging.handlers import CMRESHandler
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': './debug.log',
'maxBytes': 102400,
'backupCount': 5,
},
'elasticsearch': {
'level': 'DEBUG',
'class': 'cmreslogging.handlers.CMRESHandler',
'hosts': [{'host': 'localhost', 'port': 9200}],
'es_index_name': 'my_python_app',
'es_additional_fields': {'App': 'Test', 'Environment': 'Dev'},
'auth_type': CMRESHandler.AuthType.NO_AUTH,
'use_ssl': False,
},
'loggers': {
'django': {
'handlers': ['file','elasticsearch'],
'level': 'DEBUG',
'propagate': True,
},
},
'loggers': {
'django': {
'handlers': ['file','elasticsearch'],
'level': 'DEBUG',
'propagate': True,
},
}
},
}
```

There is more information about how Django logging works in the
`Django documentation <https://docs.djangoproject.com/en/1.9/topics/logging//>`_


Building the sources & Testing
------------------------------
## Building the sources & Testing

To create the package follow the standard python setup.py to compile.
To test, just execute the python tests within the test folder

Why using an appender rather than logstash or beats
---------------------------------------------------
## Why using an appender rather than logstash or beats

In some cases is quite useful to provide all the information available within the LogRecords as it contains
things such as exception information, the method, file, log line where the log was generated.

Expand All @@ -177,29 +184,6 @@ using `SysLogHandler <https://docs.python.org/3/library/logging.handlers.html#sy
`logstash syslog plugin <https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html>`_.


Contributing back
-----------------
Feel free to use this as is or even better, feel free to fork and send your pull requests over.


.. |downloads| image:: https://img.shields.io/pypi/dd/CMRESHandler.svg
:target: https://pypi.python.org/pypi/CMRESHandler
:alt: Daily PyPI downloads
.. |versions| image:: https://img.shields.io/pypi/pyversions/CMRESHandler.svg
:target: https://pypi.python.org/pypi/CMRESHandler
:alt: Python versions supported
.. |status| image:: https://img.shields.io/pypi/status/CMRESHandler.svg
:target: https://pypi.python.org/pypi/CMRESHandler
:alt: Package stability
.. |license| image:: https://img.shields.io/pypi/l/CMRESHandler.svg
:target: https://pypi.python.org/pypi/CMRESHandler
:alt: License
.. |ci_status| image:: https://travis-ci.org/cmanaha/python-elasticsearch-logger.svg?branch=master
:target: https://travis-ci.org/cmanaha/python-elasticsearch-logger
:alt: Continuous Integration Status
.. |codecov| image:: https://codecov.io/github/cmanaha/python-elasticsearch-logger/coverage.svg?branch=master
:target: http://codecov.io/github/cmanaha/python-elasticsearch-logger?branch=master
:alt: Coverage!
.. |gitter| image:: https://badges.gitter.im/Join%20Chat.svg
:target: https://gitter.im/cmanaha/python-elasticsearch-logger?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge
:alt: gitter
## Contributing back

Feel free to use this as is or even better, feel free to fork and send your pull requests over.
4 changes: 3 additions & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
here = path.abspath(path.dirname(__file__))

# Get the long description from the README file
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
with open(path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()

dependencies = [
Expand All @@ -38,6 +38,8 @@

description='Elasticsearch Log handler for the logging library',
long_description=long_description,
long_description_content_type="text/markdown",


# The project's main homepage.
url='https://github.com/cmanaha/python-elasticsearch-logger',
Expand Down