README

Weed-3                    \../    .`'`^.
A Sinatra Analytics API    \ \   (  @   )
                            \ `./   (__/,
                            .,____~~~~~-`

(Picture is unrelated)
=================================

Gathering real-time analytics for arbitrary factors is simple enough to
do at low levels of scale. However, once you are recording millions of
items and storing them in SQL, the aggregate queries become too slow to
realistically perform any kind of performant data analysis.

This library aims to solve this problem well enough for the average web
application to use in daily analytics activities by utilizing some data 
warehousing concepts and smart caching.

It's still very much in pre-alpha (aka 'fun' mode), so don't trust your
data in it. Please send tested patches if you find bugs.

Installation
------------
Weed3 is built on Sinatra, a simple ruby web framework, the installation
of which is beyond the scope of this file. 

For rubyists, it should behave fairly well as a Rack middleware, or even 
as a plugin, since the entire application is namespaced under the Weed module.

For everyone else, you should be able to start this application as a daemon
and proxy to it, or start it under your webserver (Apache 2 or Nginx) as
a Passenger application.

Your zeroth step is to create and load the database (later, modify the database 
settings in environment.rb). The application should create its own sqlite3
database in db/ but you'll want to load the schema.

    $ rake db:migrate
    
Now, check to see if the tests all pass on your system.

    $ rake test
    
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I"lib:test" "/Users/courtenay/.gem/ruby/1.8/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/application_test.rb" "test/stats_test.rb" 
    Loaded suite /Users/courtenay/.gem/ruby/1.8/gems/rake-0.8.7/lib/rake/rake_test_loader
    Started
    ..............
    Finished in 10.277229 seconds.
    
Start the server like this:

    $ ruby -r weed.rb -e 'Weed::Application.run!'

Access it in your browser at this URL: http://localhost:4567/ or test it with curl

    $ curl http://localhost:4567
    Weed!

    $ curl http://localhost:4567/stats
    {"count":7}

Weed attempts to be a good RESTful citizen, so, send a hit with a POST

    $ curl http://localhost:4567 -d"q[bucket_id]=10"
    $ curl http://localhost:4567/stats
    {"count":8}

    $ curl http://localhost:4567/stats/10
    {"count":1}

    $ curl http://localhost:4567/stats/10/month/2010/1
    {"count":1}
    
    $ curl http://localhost:4567/stats/10/week/2010/1/14/daily
    [0,0,0,1,0,0,0]

    For more information (there are more things you can do) see the 'weed.rb' file.
    
    # todo: flesh this out
    
You can also get trends back from it if you ask; like, this number has dropped to 50%
of last month's number (-50). I think the logic is wrong on this at the moment.

    #  {"date": [count, percent_from_previous_month]}
    #
    $ curl http://localhost:4567/trends/10/2009/1/monthly
    {"2009-12":[25,-50]},{"2009-11":[50,100]},{"2009-10":[0,null]}
    

How it works
------------

Weed3 records stats in the Stats table with a datestamp.

When you request an aggregate (hits this month), it creates an entry in the 
Weed::CachedStats table with the period scale ('month') the date (2009, 12)
and the sum of daily counts so that you never have to run that aggregate again.
(Note the daily counts are themselves aggregates, a count of all items that day)

In addition, because it knows about the hierarchy of dates (year - month - day)
if you do a sum of the year's dates, it will calculate the sum of the month's dates
(only 12 items scanned in the aggregate) rather than the days (365 items)

This part of the app is transparent to users.


note: this is fast only if there are existing records in the db, not on a fresh db
so it's up to you to keep the caches hot!



====================

todo: the stats_test takes too long because of by_total and is failing
todo: by_total / by_year / by_month -- maybe should run a count on the db if records are missing?
  rather than counting the subrecords
todo: assume no data is missing? (i.e. data is corrupted, regenerate indexes to run aggregates!)



====================



Copyright ©2010 Courtenay Gasking