Revert "bug fix"

This reverts commit de5b43e.

Conflicts:
	config/reporter.csv

README.md

@@ -1,13 +1,51 @@
-# TorQ-Finance-Starter-Pack
-An example production ready market data capture system, using randomly generated financial data.  This is installed on top of the base TorQ package, and includes a version of [kdb+tick](http://code.kx.com/wsvn/code/kx/kdb+tick).
+![TorQ Logo](../master/html/img/torq-logo.png)
 
-## Set Up 
+The framework forms the basis of a production kdb+ system by implementing some core functionality and utilities on top of kdb+, allowing developers to concentrate on the application business logic. It incorporates as many best practices as possible, with particular focus on performance, process management, diagnostic information, maintainability and extensibility. Wherever possible, we have tried to avoid re-inventing the wheel and instead have used contributed code from code.kx.com (either directly or modified). This framework will be suitable for those looking to create a new kdb+ system from scratch or those looking to add additional functionality to their existing kdb+ systems.
 
-1. Download a zip of the latest version of [TorQ](https://github.com/AquaQAnalytics/TorQ/archive/master.zip)
-2. Download a zip of [this starter pack](https://github.com/AquaQAnalytics/TorQ-Finance-Starter-Pack/archive/master.zip)
-3. Unzip TorQ
-4. Unzip the starter pack over the top (this will replace some files)
-5. Read [this document](https://github.com/AquaQAnalytics/TorQ-Finance-Starter-Pack/blob/master/AquaQTorQFinanceStarterPack.pdf?raw=true) for information on how to configure and get started
+[Have a skim through our brochure](../master/aquaq-torq-brochure.pdf?raw=true) for a bit more information.  The easiest way to get a production capture started is to download and install one of the [Starter Packs](https://github.com/AquaQAnalytics), or [read the manual](../master/AquaQTorQ.pdf?raw=true)
 
+## Quick Start
 
+To launch a process wrapped in the framework, you need to set the environment variables and give the process a type and name.  The type and name can be explicitly passed on the command line.  setenv.sh is an example of how to set the environment variables on a unix type system.  For a windows system, see http://www.computerhope.com/issues/ch000549.htm.  kdb+ expects all paths to be / (forward-slash) separated so all paths on all OSs should be forward-slash separated. 
 
+To avoid standard out/err being redirected, used the -debug flag
+``` 
+./setenv.sh         /- Assuming unix type OS
+q torq.q -proctype test -procname mytest -debug
+```
+
+To load a file, use -load
+```
+q torq.q -load mytest.q -proctype test -procname mytest -debug
+```
+This will launch the a process running within the framework with all the default values.  For the rest, read the document!
+
+## Release Notes
+
+- **1.0, Feb 2014**: 
+  * Initial public release of TorQ
+- **1.1, Apr 2014**: 
+  * Added compression utilities, HTML5 utilities, housekeeping process, file alerter process, kdb+tick quick start
+- **1.2, Sep 2014**:	
+  * Tested on kdb+ 3.2
+  * Added connections to external (non TorQ) processes using nonprocess.csv
+  * Modified file alerter with optional switch to move or not move a file if any function fails to process the file
+  * Discovery service(s) host:port(s) can be passed on the command line (.servers.DISCOVERY) to a process (this should enable complete bypassing of process.csv if required)
+  * Add custom hook (.servers.connectcustom) which is invoked whenever a new connection is made (allows, for example, subscription to a new process)
+  * Add optional application detail file ($KDBCONFIG/application.txt) to allow customisation of the start up banner (application version etc.)
+  * If required env. variables (KDBCODE, KDBCONFIG, KDBLOG) are not set they will default to $QHOME/code, $QHOME/config, $QHOME/logs respectively (previously the process failed and exited)
+- **2.0, Jan 2015**:  
+  * Added RDB process which extends r.q from kdb+ tick.
+  * Added WDB to write down data periodically throughout the day.  Extends w.q.
+  * RDB and WDB allow seemless end-of-day event (no data outage, no tickerplant back pressure)
+  * Added Reporting Process to run reports periodically and process the results
+  * Added Subscription code to manage multiple subscriptions to different data sources
+  * Added email library which uses libcurl.  Used to send emails from TorQ processes
+  * Added standard monitoring checks to the database code
+  * Added data loader script.  Utility functions to load a directory of data into a database in chunks, sort and part at the end
+  * Added tickerplant log recovery utilities to recover as many messages as possible from a log file rather than just stopping at the first bad message
+  * Added compression process to run and compress a given database
+  * Modified compression code to handle par.txt databases
+  * Modified compression code and housekeeping process to run with kdb+ 2.*
+  * Modified std out/err logging and usage logging to include process name and process type (the logmsg table had changed along with some of the functions in the .lg namespace so you might need to check in case you have overridden any of them)
+  * Removed launchtick scripts and some default configuration: to create a test system, install a starter pack

code/handlers/logusage.q

@@ -1,4 +1,4 @@
-/ log external (.z.p* & .z.exit) usage of a kdb+ session
+/ log external (.proc.cp[]* & .z.exit) usage of a kdb+ session
 
 // based on logusage.q from code.kx
 // http://code.kx.com/wsvn/code/contrib/simon/dotz/

config/application.txt

@@ -1,2 +0,0 @@
-
-TorQ Finance Starter Pack

config/compressionconfig.csv

@@ -1,2 +1,2 @@
 table,minage,column,calgo,cblocksize,clevel
-default,1,default,2,16,9
+default,10,default,1,15,0

config/housekeeping.csv

@@ -1,4 +1,2 @@
 function,path,match,exclude,age
-zip,{KDBLOG}/,*.log,,10
-rm,{KDBLOG}/,*.gz,,30
-zip,{KDBHDB}/,database20*,,1
+rm,{KDBLOG}/,*.log,,10

config/process.csv

@@ -1,16 +1 @@
 host,port,proctype,procname
-homer,310000,discovery,discovery1
-homer,311000,tickerplant,tickerplant1
-homer,312000,rdb,rdb1
-homer,313000,hdb,hdb1
-homer,313010,hdb,hdb2
-homer,314000,wdb,wdb1
-homer,314500,sort,sort1
-homer,300000,gateway,gateway1
-homer,301000,kill,killtick
-homer,302000,monitor,monitor1
-homer,303000,tickerlogreplay,tpreplay1
-homer,304000,housekeeping,housekeeping1
-homer,305000,reporter,reporter1
-homer,306000,compression,compression1
-homer,307000,feed,feed1

config/settings/compression.q

@@ -8,6 +8,6 @@
 
 \d .cmp
 inputcsv:`$getenv[`KDBCONFIG],"/compressionconfig.csv"				// compression config file to use
-hdbpath:`:hdb/database								// hdb directory
+hdbpath:`:hdb									// hdb directory
 maxage:365									// the maximum date range of partitions to scan
 exitonfinish:1b									// exit the process when compression is complete

config/settings/gateway.q

@@ -1,7 +1,7 @@
 // Default configuration for the gateway process
 
 \d .gw
-synccallsallowed:1b		// whether synchronous calls are allowed
+synccallsallowed:0b		// whether synchronous calls are allowed
 querykeeptime:0D00:30		// the time to keep queries in the
 errorprefix:"error: "		// the prefix for clients to look for in error strings
 

config/settings/monitor.q

@@ -2,7 +2,4 @@
 
 // Server connection details
 \d .servers
-
-// list of connections to make at start up
-// can't use `ALL as the tickerplant doesn't publish heartbeats
-CONNECTIONS:`discovery`rdb`hdb`wdb`sort`gateway`housekeeping`reporter`feed
+CONNECTIONS:`ALL		// list of connections to make at start up

config/settings/rdb.q

@@ -18,7 +18,7 @@ schema:1b			//retrieve the schema from the tickerplant
 tpconnsleepintv:10		//number of seconds between attempts to connect to the tp                                                                 
 gc:1b				//if true .Q.gc will be called after each writedown - tradeoff: latency vs memory usage
 sortcsv:`:config/sort.csv	//location of csv file
-reloadenabled:1b		//if true, the RDB will not save when .u.end is called but
+reloadenabled:0b		//if true, the RDB will not save when .u.end is called but
                			//will clear it's data using reload function (called by the WDB)
 parvaluesrc:`log		//where to source the rdb partition value, can be log (from tp log file name),
 				//tab (from the the first value in the time column of the table that is subscribed for)

config/settings/sort.q

@@ -33,7 +33,7 @@ partitiontype:`date                     // set type of partition (defaults to `d
 gmttime:1b                              // define whether the process is on gmttime or not
 getpartition:{(`date^partitiontype)$(.z.d;.z.D)gmttime}       // function to determine the partition value
 reloadorder:`hdb`rdb                    // order to reload hdbs and rdbs
-hdbdir:`:hdb/database			// move wdb database to different location
+hdbdir:`:hdb                            // move wdb database to different location
 sortcsv:`:config/sort.csv               // location of csv file
 permitreload:1b                         // enable reload of hdbs/rdbs
 compression:()                          // specify the compress level, empty list if no required

config/settings/wdb.q

@@ -33,7 +33,7 @@ partitiontype:`date			// set type of partition (defaults to `date, can be `date,
 gmttime:1b				// define whether the process is on gmttime or not
 getpartition:{(`date^partitiontype)$(.z.d;.z.D)gmttime}       // function to determine the partition value
 reloadorder:`hdb`rdb			// order to reload hdbs and rdbs
-hdbdir:`:hdb/database			// move wdb database to different location
+hdbdir:`:hdb				// move wdb database to different location
 sortcsv:`:config/sort.csv		// location of csv file
 permitreload:1b				// enable reload of hdbs/rdbs
 compression:()				// specify the compress level, empty list if no required

logs/dummy.log

@@ -0,0 +1 @@
+dummy log file