Merge pull request #28 from airbnb/docs-conf-authors-contribs-reqs

[docs] Add Authors, Contributors, and additional doc changes

AUTHORS.rst

@@ -0,0 +1,10 @@
+Below are a list of both core members and contributors to StreamAlert.
+
+Core Members:
+  * `Jack Naglieri <https://www.github.com/jacknagz>`_
+  * `Mime Frame <https://www.github.com/mime-frame>`_
+  * `Jamie Brim <https://www.github.com/strcrzy>`_
+  * `Austin Byers <https://github.com/austinbyers>`_
+  * `Chunyong Lin <https://github.com/chunyong-lin>`_
+
+Contributors:

CONTRIBUTING.rst

@@ -0,0 +1,128 @@
+Contributing
+============
+
+Installing Dependencies
+-----------------------
+
+Make sure you are running Python 2.7::
+
+  $ python -V
+  Python 2.7.x
+
+Install:
+
+1. `Pip <https://pip.pypa.io/en/stable/installing/>`_
+2. `Terraform <https://www.terraform.io/intro/getting-started/install.html>`_
+
+Fork the StreamAlert repository from the UI, and then clone it locally::
+
+  $ git clone https://github.com/<your-username>/streamalert.git
+
+Change into the cloned StreamAlert repo, and install Python dependencies::
+  
+  $ cd streamalert
+  $ pip install -r requirements.txt
+
+Making Changes
+--------------
+
+Checkout a new branch::
+  
+  git checkout -b <branch-name>
+
+Make changes, add features, or fix bugs.
+
+When writing commit messages, make sure to prefix with one of the following tags::
+
+  [docs]              # changes to StreamAlert documentation
+  [cli]               # stream_alert_cli changes
+  [lambda][alert]     # stream_alert lambda changes
+  [lambda][output]    # stream_alert_output lambda changes
+  [terraform]         # terraform changes
+  [core]              # changes with core stream_alert classes used across both functions
+  [testing]           # changes with testing infrastructure or processes
+  [setup]             # StreamAlert development setup changes
+
+The first line of your commit message should be short.  Use newlines to explain further::
+
+  [tag] short description
+
+  * longer explanation of what the change is
+  * with added context
+
+.. note:: Please squash your similar commits into one.  This keeps the repositories commit history easy to read.
+
+Commit Squashing
+~~~~~~~~~~~~~~~~
+
+If all of your commits fall within the same tag, you can squash them during the pull request process via the Github UI.  If not, follow the steps below.
+
+Scenario: You have unstaged changes that you want to add into your last commit::
+
+  $ git add -u           # add all tracked files
+  $ git commit --amend   # commit these changes to the last commit
+
+Scenario: You have two commits you want to merge into one::
+
+  $ git log --oneline
+  c3dbbe9 [docs] add contributing guide
+  f5b038e [docs] add streamalert authors
+  04e52c1 [setup] add jinja2 to requirements.txt
+  ...
+  
+In this case, let's merge the two ``[docs]`` commits::
+
+  $ git rebase -i f5b038e~1
+  pick f5b038e [docs] add streamalert authors
+  squash c3dbbe9 [docs] add contributing guide
+  
+If your editor is `vim`, type `:wq` once you enter the above changes.
+
+This will now open a new window to modify your combined commit message.  Make your edits, and exit once again.
+
+After rebasing, you will need to force push your branch if it already exists upstream::
+
+  $ git push origin <mybranch> -f
+
+Tests
+-----
+
+Unit Testing
+~~~~~~~~~~~~
+
+StreamAlert contains unit tests for many parts of the code.  When making changes, you need to ensure that you do  not break existing functionality.  To run unit tests locally::
+
+  # run this from the repo root
+  $ nosetests -v test/unit
+
+Each test should end with ``... ok``, and finally you should see ``OK`` at the end.
+
+If you are making changes which require unit test refactoring, please do so.
+
+If you are adding features to existing classes with tests, you must add test cases to verify expected behavior.
+
+Integration Testing
+~~~~~~~~~~~~~~~~~~~
+
+To verify StreamAlert works from end-to-end, locally, follow the testing instructions `here <https://streamalert.io/rules.html>`_. 
+
+Pull Request
+------------
+
+Once your code is ready for review, push the branch to your forked repository, and make a pull-request to the main ``airbnb/streamalert`` repo.
+
+The title of your pull request should be a short description of your changes.
+
+In your pull request body, use the following template::
+
+  to @airbnb/streamalert-maintainers
+
+  size: small|medium|large
+  resolves: #1              # only add this if there's a relevant open issue related to this PR
+
+  * summary of changes 1
+  * summary of changes 2
+  
+All pull requests must pass continuous integration tests (nosetests) and receive a code review from one of the maintainers.
+
+After your branch has been accepted, you are free to merge into ``master``!

README.rst

@@ -1,15 +1,13 @@
-**********************************************************
 StreamAlert - Serverless, Realtime Data Analysis Framework
-**********************************************************
+==========================================================
 
 .. image:: https://travis-ci.org/airbnb/streamalert.svg?branch=master
     :target: https://travis-ci.org/airbnb/streamalert
 
 StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
 
-For more details, see our announcement post `here <https://medium.com/@airbnbeng/e8619e3e5043>`_
-
-*High-level*:
+High-level
+~~~~~~~~~~
 
 * Deployment is automated: simple, safe and repeatable for any AWS account
 * Easily scalable from megabytes to terabytes per day
@@ -20,8 +18,10 @@ For more details, see our announcement post `here <https://medium.com/@airbnbeng
 * Supports different types of data (ex: JSON, CSV, Key-Value, or Syslog)
 * Supports different use-cases like security, infrastructure, compliance and more
 
-*Links*:
+Links
+~~~~~
 
+* `Announcement Post <https://medium.com/@airbnbeng/e8619e3e5043>`_
 * `User Guide <https://streamalert.readthedocs.io/>`_
 * `Twitter <https://twitter.com/streamalert_io>`_ (unofficial)
 * `Slack <https://streamalert.herokuapp.com/>`_ (unofficial)

conf/sample_sources.json

@@ -7,13 +7,13 @@ $ ./stream_alert_cli.py terraform status
 */
 {
   "kinesis": {
-    "cluster-1--kinesis-stream": {
+    "prefix_cluster1_stream_alert_kinesis": {
       "logs": [
         "json_log_name",
         "csv_log_name"
       ]
     },
-    "cluster-2-kinesis-stream": {
+    "prefix_cluster2_stream_alert_kinesis": {
       "logs": [
         "json_log_name"
       ]

docs/source/account.rst

@@ -24,7 +24,7 @@ Open ``variables.json`` and set ``account_id`` to this value.
 prefix
 ~~~~~~
 
-Open ``variables.json`` and set ``prefix`` to your company name.
+Open ``variables.json`` and set ``prefix`` to your company or organization name.
 
 Administrator
 ~~~~~~~~~~~~~

docs/source/clusters.rst

@@ -7,7 +7,7 @@ StreamAlert will deploy separate infrastructure for each ``cluster`` (environmen
 
 What constitutes a ``cluster`` is up to you.
 
-Example: You could define ``IT``, ``PCI`` and ``Production`` clusters
+Example: You could define ``IT``, ``PCI`` and ``Production`` clusters.
 
 **Strategy**
 

docs/source/conf-datasources.rst

@@ -1,22 +1,27 @@
-Datasources
-===========
+Datasource Configuration
+========================
+
+For background on supported datasource types, read `datasources <datasources.html>`_.
 
 Overview
 --------
 
-Datasources are defined in ``conf/sources.json``
+* Datasources defined in ``conf/sources.json`` control which datasources can send to and be analyzed by StreamAlert.
+* Each datasource (``kinesis``, ``s3``) contains a mapping of specific datasource names (kinesis stream names, s3 bucket IDs) along with a list of logs coming from that source.
+* Logs are defined in ``conf/logs.json``
+* Each log in the list of ``logs`` dictates to StreamAlert how to parse incoming data from that entity.  Data will only be analyzed if their type is defined here.
 
 Example::
 
     {
-      "kinesis": {                 # define each kinesis stream w/respective logs
-        "abc_corporate_stream": {  # kinesis stream name
-          "logs": [                # expected log types
+      "kinesis": {                               # define each kinesis stream w/respective logs
+        "abc_corporate_stream_alert_kinesis": {  # kinesis stream name
+          "logs": [                              #  expected log types
             "box",
             "pan"
           ]
         },
-        "abc_production_stream": {
+        "abc_production_stream_stream_alert_kinesis": {
           "logs": [
             "inspec",
             "osquery"

docs/source/conf-schemas.rst

@@ -43,7 +43,7 @@ Here are the basics:
 * An empty hash ({}) implies zero or more elements
 * Schemas can be as tight or as loose as you want (see Example: osquery)
 
-JSON Example: inspec
+JSON Example: Inspec
 --------------------
 Schema::
 

docs/source/datasources.rst

@@ -1,5 +1,5 @@
 Datasources
-=============
+===========
 
 StreamAlert supports:
 
@@ -8,10 +8,12 @@ StreamAlert supports:
 * Log Forwarding Agents\*
 * Code/Applications\*
 
-\* *must send to a Kinesis Stream*
+.. note:: \* *Must send to a Kinesis Stream*
+
+To configure datasources, read `datasource configuration <conf-datasources.html>`_
 
 Amazon S3
-----------
+---------
 
 StreamAlert supports data analysis and alerting for logs written to configured Amazon S3 buckets.
 This is achieved via Amazon S3 Event Notifications looking for an event type of ``s3:ObjectCreated:*``
@@ -29,9 +31,30 @@ Example non-AWS use-cases:
 * Web Application logs (Apache, nginx, ...)
 * SaaS logs (Box, OneLogin, …)
 
+AWS Kinesis Streams
+-------------------
+
+StreamAlert also supports data analysis and alerting for logs written to AWS Kinesis Streams.
+By default, StreamAlert configures an AWS Kinesis stream per `cluster <clusters.html>`_.
+
+Optionally, StreamAlert can also utilize existing streams as an additional source
+by adding the following into your generated Terraform cluster file (found in ``terraform/cluster-name.tf``)::
+
+  // Enable a Kinesis Stream to send events to Lambda
+  module "kinesis_events_<cluster-name>" {
+    source                    = "modules/tf_stream_alert_kinesis_events"
+    lambda_staging_enabled    = true
+    lambda_production_enabled = true
+    lambda_role_id            = "${module.stream_alert_<cluster-name>.lambda_role_id}"
+    lambda_function_arn       = "${module.stream_alert_<cluster-name>.lambda_arn}"
+    kinesis_stream_arn        = "<add-kinesis-stream-ARN-here>"
+    role_policy_prefix        = "<cluster-name>"
+  }
+  
+There are several ways to send data into AWS Kinesis, as listed below.
 
 Log Forwarding Agents
-----------------------
+~~~~~~~~~~~~~~~~~~~~~
 
 StreamAlert utilizes AWS Kinesis Streams for real-time data ingestion.
 
@@ -42,25 +65,12 @@ Log forwarding agents that support AWS Kinesis Streams:
 * **aws-kinesis-agent**
 * **osquery**
 
-
 Code/Applications
-----------------------
+~~~~~~~~~~~~~~~~~
 
 StreamAlert utilizes AWS Kinesis Streams for real-time data ingestion.
 
 Your code can send data to an AWS Kinesis Stream via:
 
 * AWS SDK (Streams API)
 * KPL (Amazon Kinesis Producer Library)
-
-
-AWS Kinesis Streams
--------------------
-
-StreamAlert can utilize existing stream(s) and/or create/deploy new streams to support log forwarding agents or code/applications.
-
-
-
-
-
-

docs/source/getting-started.rst

@@ -1,16 +1,15 @@
 Getting Started
 ===============
 
-Perform the following on your laptop or development machine
+Perform the following steps on your laptop or development machine
 
 Dependencies
 ------------
 
-1. Install Python2.7
-2. Install `pip <https://pip.pypa.io/en/stable/installing/>`_
-3. Install `Terraform <https://www.terraform.io/intro/getting-started/install.html>`_
-4. Download StreamAlert: ``git clone https://github.com/airbnb/streamalert.git``
-5. Install dependencies: ``pip install -r requirements.txt``
+1. Install Python 2.7 and  `pip <https://pip.pypa.io/en/stable/installing/>`_
+2. Install `Terraform <https://www.terraform.io/intro/getting-started/install.html>`_
+3. Download StreamAlert: ``git clone https://github.com/airbnb/streamalert.git``
+4. Install dependencies: ``pip install -r requirements.txt``
 
 .. note:: For Mac OSX/Homebrew users, add the following to ~/.pydistutils.cfg:
 
@@ -19,16 +18,17 @@ Dependencies
    [install]
    prefix=
 
-Configuration
--------------
+Quick Start
+-----------
 
-1. Define your `account <account.html>`_
+1. Define your `AWS account <account.html>`_
 2. Define your `clusters <clusters.html>`_
 3. Define your `datasources <conf-datasources.html>`_
 4. Define your `schemas <conf-schemas.html>`_
-5. Define your `kinesis streams <kinesis-streams.html>`_
-6. Define your `kinesis firehose <kinesis-firehose.html>`_
-7. Define your `lambda settings <lambda.html>`_
-8. Define/deploy your `secrets <secrets.html>`_
+5. Configure your `kinesis streams <kinesis-streams.html>`_
+6. Configure your `kinesis firehose <kinesis-firehose.html>`_
+7. Configure your `lambda settings <lambda.html>`_
+8. Write your `rules <rules.html>`_
+9. Define/deploy your `secrets <secrets.html>`_
 
 Now it's time to `deploy <deployment.html>`_!