Skip to content

Commit

Permalink
565 update all aquaq references to data intellect (#574)
Browse files Browse the repository at this point in the history
* #565 updating all github references to new user account dataintellecttech

* #565 small typo

* #565 updating references from old repo to new repo
  • Loading branch information
AUnterrainer authored May 30, 2023
1 parent 3669d0c commit edad5b0
Show file tree
Hide file tree
Showing 9 changed files with 37 additions and 37 deletions.
22 changes: 11 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
![TorQ Logo](../master/html/img/DataIntellect-TorQ-logo.png)

[Read our documentation](https://AquaQAnalytics.github.io/TorQ/)
[Read our documentation](https://DataIntellectTech.github.io/TorQ/)

The framework forms the basis of a production kdb+ system by implementing some core functionality and utilities on top of kdb+, allowing developers to concentrate on the application business logic. It incorporates as many best practices as possible, with particular focus on performance, process management, diagnostic information, maintainability and extensibility. Wherever possible, we have tried to avoid re-inventing the wheel and instead have used contributed code from code.kx.com (either directly or modified). This framework will be suitable for those looking to create a new kdb+ system from scratch or those looking to add additional functionality to their existing kdb+ systems.

[Have a skim through our brochure](../master/aquaq-torq-brochure.pdf?raw=true) for a bit more information. The easiest way to get a production capture started is to download and install one of the [Starter Packs](https://github.com/AquaQAnalytics), or [read through our Github-Pages site](https://AquaQAnalytics.github.io/TorQ/). We also have a [Google Group for questions/discussions](https://groups.google.com/forum/#!forum/kdbtorq).
[Have a skim through our brochure](../master/aquaq-torq-brochure.pdf?raw=true) for a bit more information. The easiest way to get a production capture started is to download and install one of the [Starter Packs](https://github.com/DataIntellectTech), or [read through our Github-Pages site](https://DataIntellectTech.github.io/TorQ/). We also have a [Google Group for questions/discussions](https://groups.google.com/forum/#!forum/kdbtorq).

## Quick Start

Expand Down Expand Up @@ -62,7 +62,7 @@ Head to the address it gives you to check if your changes have worked. More info
* Data Access API bug fixes
* Improved attribute routing surrounding overlapping processes
- **4.1.0, Apr 2021**
* Data Access API Update [Blog post](https://www.aquaq.co.uk/torq-2/data-access-api/) and [Documentation](docs/dataaccess.md)
* Data Access API Update [Blog post](https://dataintellect.com/data-access-api/) and [Documentation](docs/dataaccess.md)
- **4.0.1, Mar 2021**
* Bug fixes
* Documentation improvements
Expand All @@ -78,10 +78,10 @@ Head to the address it gives you to check if your changes have worked. More info
* DQS,DQC and DQE upgrades
* PCAP functionality additions
- **3.7.0, May 2020**
* Added Data Quality System ([blog post](https://www.aquaq.co.uk/torq-2/torq-data-quality-system/) and [documentation](http://aquaqanalytics.github.io/TorQ/Processes/#torq-data-quality-system-architecture))
* Added Data Quality System ([Blog post](https://dataintellect.com/torq-data-quality-system/) and [documentation](http://dataintellecttech.github.io/TorQ/Processes/#torq-data-quality-system-architecture))
* Fix k4unit testing so that it works with kdb+ 4.0.
- **3.6.0, Apr 2020**:
* Added DataDog integration for monitoring: https://www.aquaq.co.uk/q/monitoring-a-torq-system-with-datadog/
* Added DataDog integration for monitoring: ([Blog post] (https://dataintellect.com/monitoring-a-torq-system-with-datadog/)
* Improved k4unit integration
* Tested with kdb+ 4.0
* Fix for reval calls via gateaway
Expand Down Expand Up @@ -117,11 +117,11 @@ Head to the address it gives you to check if your changes have worked. More info
* Changed the start script, to start or stop processes separately, in a batch or all at once. All the process configuration is now at one place in $KDBCONFIG/process.csv.
* Bug fix for erroneous brackets that were causing permissions config to be loaded in incorrect order
- **3.2.0, February 2018**:
* Added initial version of [analytics library](http://aquaqanalytics.github.io/TorQ/analyticslib/), a set of functions to aid data manipulation
* Added optional [dependency checker](http://aquaqanalytics.github.io/TorQ/gettingstarted/#application-dependency) for applications to check if they have minimal TorQ and kdb+ dependencies
* Added [kdb compression to filealerter](https://github.com/AquaQAnalytics/TorQ/commit/1b048d5e52fa26d537d6c88064b533f376ef08c6). Use case is for compression of tickerplant log files, so log files can be replayed directly rather than having separate uncompress step
* Added [-onelog command line](http://aquaqanalytics.github.io/TorQ/gettingstarted/#logging) option to write stdout/stderr to same log file
* Added [file pattern option](http://aquaqanalytics.github.io/TorQ/utilities/#dataloaderq) to data loading utility
* Added initial version of [analytics library](http://dataintellecttech.github.io/TorQ/analyticslib/), a set of functions to aid data manipulation
* Added optional [dependency checker](http://dataintellecttech.github.io/TorQ/gettingstarted/#application-dependency) for applications to check if they have minimal TorQ and kdb+ dependencies
* Added [kdb compression to filealerter](https://github.com/DataIntellectTech/TorQ/commit/1b048d5e52fa26d537d6c88064b533f376ef08c6). Use case is for compression of tickerplant log files, so log files can be replayed directly rather than having separate uncompress step
* Added [-onelog command line](http://dataintellecttech.github.io/TorQ/gettingstarted/#logging) option to write stdout/stderr to same log file
* Added [file pattern option](http://dataintellecttech.github.io/TorQ/utilities/#dataloaderq) to data loading utility
* Bug Fixes
- **3.1.0, May 2017**:
* added kafka which is a real time messaging system with persistent storage in message logs
Expand All @@ -131,7 +131,7 @@ Head to the address it gives you to check if your changes have worked. More info
- **3.0.0, January 2017**:
* Added a permissioning system, allowing granular control of access to users & groups
* Added LDAP support, allowing a user to authenticate against an LDAP server
* Improved documentation now available at http://aquaqanalytics.github.io/TorQ/
* Improved documentation now available at http://dataintellecttech.github.io/TorQ/
- **2.7.0, November 2016**:
* Tickerplant incorporated. Tickerplant has faster recovery for real time subscribers, and better timezone handling
* Filealerter uses a splayed table to store the table of already processed data. If it finds a table in current format (flat) it will change it to a splay
Expand Down
8 changes: 4 additions & 4 deletions docs/InstallGuide.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ Install Guide

In your Linux terminal run the following lines copying them in one by one:

wget https://raw.githubusercontent.com/AquaQAnalytics/TorQ/master/installtorqapp.sh
wget https://raw.githubusercontent.com/DataIntellectTech/TorQ/master/installtorqapp.sh

wget --content-disposition https://github.com/AquaQAnalytics/TorQ/archive/3.7.0.tar.gz
wget --content-disposition https://github.com/DataIntellectTech/TorQ/archive/3.7.0.tar.gz

wget --content-disposition https://github.com/AquaQAnalytics/TorQ-Finance-Starter-Pack/archive/v1.9.0.tar.gz
wget --content-disposition https://github.com/DataIntellectTech/TorQ-Finance-Starter-Pack/archive/v1.9.0.tar.gz

Then to launch the script.

Expand Down Expand Up @@ -53,7 +53,7 @@ Check if the stack is up
<tbody>
<tr>
<td class="tg-btxf">torq</td>
<td class="tg-btxf">Is a mandatory parameter that is the full path or relative path to the TorQ installation. It can either be a TorQ Directory where the version is already unzipped, that can be used when multiple TorQ Applications are used on the server for example and all point to a single TorQ main code. This will create a softlink to the relevant TorQ code. Or it can be a .tar.gz file of the TorQ installation for a fresh install. Example usage in the script:<br><br><span style="font-weight:bold;font-style:italic"> --torq /home/user/TorQ/TorQ-3.7.0</span><br><br>Where TorQ-3.7.0 is unzipped directory from the latest release .tar.gz file. Or <br><br><span style="font-weight:bold;font-style:italic">--torq /home/user/TorQ/TorQ-3.7.0.tar.gz </span><br><br>Which is the .tar.gz file from GitHub using: <br><br><span style="font-weight:bold;font-style:italic">wget --content-disposition https://github.com/AquaQAnalytics/TorQ/archive/3.7.0.tar.gz</span></td>
<td class="tg-btxf">Is a mandatory parameter that is the full path or relative path to the TorQ installation. It can either be a TorQ Directory where the version is already unzipped, that can be used when multiple TorQ Applications are used on the server for example and all point to a single TorQ main code. This will create a softlink to the relevant TorQ code. Or it can be a .tar.gz file of the TorQ installation for a fresh install. Example usage in the script:<br><br><span style="font-weight:bold;font-style:italic"> --torq /home/user/TorQ/TorQ-3.7.0</span><br><br>Where TorQ-3.7.0 is unzipped directory from the latest release .tar.gz file. Or <br><br><span style="font-weight:bold;font-style:italic">--torq /home/user/TorQ/TorQ-3.7.0.tar.gz </span><br><br>Which is the .tar.gz file from GitHub using: <br><br><span style="font-weight:bold;font-style:italic">wget --content-disposition https://github.com/DataIntellectTech/TorQ/archive/3.7.0.tar.gz</span></td>
</tr>
<tr>
<td class="tg-0pky">releasedir</td>
Expand Down
2 changes: 1 addition & 1 deletion docs/Processes.md
Original file line number Diff line number Diff line change
Expand Up @@ -784,7 +784,7 @@ A key tickerplant function is to timestamp the incoming data before it gets publ

This allows for more complex configurations, such as an Foreign Exchange data capture system which timestamps data in UTC but rolls at 5 PM EST.

The key variable used is `.eodtime.dailyadj` but more information on setting up a TorQ process in a different time zone can be found [here](https://aquaqanalytics.github.io/TorQ/utilities/#eodtimeq).
The key variable used is `.eodtime.dailyadj` but more information on setting up a TorQ process in a different time zone can be found [here](https://dataintellecttech.github.io/TorQ/utilities/#eodtimeq).

### Per Table Customisation

Expand Down
22 changes: 11 additions & 11 deletions docs/cheatsheet.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Cheat Sheet

The purpose of this cheatsheet is to provide a very condensed guide to the bits you need to know about TorQ to either debug a running process, or to extend a TorQ stack. It does not replace the main body of documentation, and ordering is presumed order-of-relevance. The below uses the default configuration of TorQ and TorQ Finance Starter Pack, though a lot of it is configurable. It's probably a good idea to read the [About](http://aquaqanalytics.github.io/TorQ/Overview/) section at least.
The purpose of this cheatsheet is to provide a very condensed guide to the bits you need to know about TorQ to either debug a running process, or to extend a TorQ stack. It does not replace the main body of documentation, and ordering is presumed order-of-relevance. The below uses the default configuration of TorQ and TorQ Finance Starter Pack, though a lot of it is configurable. It's probably a good idea to read the [About](http://dataintellecttech.github.io/TorQ/Overview/) section at least.
## Debugging Running Processes
Each TorQ process has several internal tables which are useful. Connect to these using standard tools (qcon, an IDE etc.). Default user:pass of admin:admin will usually work.

Expand All @@ -18,10 +18,10 @@ select from .usage.usage where time within ...

Note that this table is not especially helpful for gateway queries which are executed in an async call back manner. The gateway part of the request will (should) usually have a very short run time so the back end services should be interrogated to see what the slow parts are.

[More info on usage logs.](http://aquaqanalytics.github.io/TorQ/handlers/#logusageq)
[More info on usage logs.](http://dataintellecttech.github.io/TorQ/handlers/#logusageq)

### .timer.timer
.timer.timer shows information about the timer calls which are scheduled / have been run. Pay attention to the "active" field- if a timer call fails it will be removed from the timer (active=0b). To avoid this if required, wrap the function being executed by the timer in an error trap in the standard way. Use .timer.timer in combination with .usage.usage to work out if there are slow running/too frequent timers which are causing problems. [More info.](http://aquaqanalytics.github.io/TorQ/utilities/#timerq)
.timer.timer shows information about the timer calls which are scheduled / have been run. Pay attention to the "active" field- if a timer call fails it will be removed from the timer (active=0b). To avoid this if required, wrap the function being executed by the timer in an error trap in the standard way. Use .timer.timer in combination with .usage.usage to work out if there are slow running/too frequent timers which are causing problems. [More info.](http://dataintellecttech.github.io/TorQ/utilities/#timerq)

### Log Files
Log files are stored in the log directory specified by the environment variable KDBLOG. Each process creates 3 log files:
Expand All @@ -41,13 +41,13 @@ sort -nk1 out_log err_log | grep -B n ERR
```

### .clients.clients
This shows inbound connections (connections created into this process). It may have interesting information about connection open/close. If it has a lot of rows it means some clients are connecting and disconnecting frequently. [More info.](http://aquaqanalytics.github.io/TorQ/handlers/#trackclientsq)
This shows inbound connections (connections created into this process). It may have interesting information about connection open/close. If it has a lot of rows it means some clients are connecting and disconnecting frequently. [More info.](http://dataintellecttech.github.io/TorQ/handlers/#trackclientsq)

### .servers.SERVERS
This shows outbound connections (connections created by this process to other processes). It's useful for tracking connections which have died.

## Starting, Stopping and Debugging Processes
95% of TorQ installations in production run on Linux, and the below applies to Linux only. Use the torq.sh script to start/stop/debug processes. [Become familiar with torq.sh, it has some very handy utilities.](http://aquaqanalytics.github.io/TorQ/gettingstarted/#using-torqsh)
95% of TorQ installations in production run on Linux, and the below applies to Linux only. Use the torq.sh script to start/stop/debug processes. [Become familiar with torq.sh, it has some very handy utilities.](http://dataintellecttech.github.io/TorQ/gettingstarted/#using-torqsh)

```
newdeploy$ ./deploy/bin/torq.sh
Expand Down Expand Up @@ -125,11 +125,11 @@ q)

## Deploying

Deployments should be very simple on Linux [using the installation script](http://aquaqanalytics.github.io/TorQ/InstallGuide/).
Deployments should be very simple on Linux [using the installation script](http://dataintellecttech.github.io/TorQ/InstallGuide/).

```bash
# pull in latest TorQ FSP install script
wget https://raw.githubusercontent.com/AquaQAnalytics/TorQ-Finance-Starter-Pack/master/installlatest.sh
wget https://raw.githubusercontent.com/DataIntellectTech/TorQ-Finance-Starter-Pack/master/installlatest.sh
# execute it
bash installlatest.sh
# It will finish with a message like below:
Expand Down Expand Up @@ -157,21 +157,21 @@ TIME | PROCESS | STATUS | PID | PORT
## Modifying Existing Installations
We always try to esnure new versions of TorQ are backwardly compatible. Try to avoid modifying TorQ itself and instead make application specific modifications. The installation script deploys TorQ in a structure which keeps TorQ and the application separate, and this should be adhered to whenever possible.

To modify the config for a process, do it either in the [config directory](http://aquaqanalytics.github.io/TorQ/gettingstarted/#configuration-loading) or as a command line start up parameter as- almost all config variables that are defined in a config file and exists in a non-root namespace can be overridden from the command line.
To modify the config for a process, do it either in the [config directory](http://dataintellecttech.github.io/TorQ/gettingstarted/#configuration-loading) or as a command line start up parameter as- almost all config variables that are defined in a config file and exists in a non-root namespace can be overridden from the command line.

To make a process load additional files, you can:

- append the additional files to the -load parameter on the start line
- append a set of directories of files using -loaddir on the start line
- place the files in one of the directories that is [loaded by default on start up](http://aquaqanalytics.github.io/TorQ/gettingstarted/#code-loading)
- place the files in one of the directories that is [loaded by default on start up](http://dataintellecttech.github.io/TorQ/gettingstarted/#code-loading)

Start line modifications can be made in process.csv and will be picked up by torq.sh. Of these approaches, the latter is probably preferable.

## Adding Custom Processes

When adding custom processes to TorQ it is important to [understand the significance of proctype](http://aquaqanalytics.github.io/TorQ/gettingstarted/#process-identification). In a nutshell, only processes which do exactly the same thing should have the same proctype. If two processes do roughly the same thing then -parentproctype can be used to share common functionality. Also there isn't any formal association between proctype and code name file name e.g. a process that loads code/processes/rdb.q can have any proctype we like.
When adding custom processes to TorQ it is important to [understand the significance of proctype](http://dataintellecttech.github.io/TorQ/gettingstarted/#process-identification). In a nutshell, only processes which do exactly the same thing should have the same proctype. If two processes do roughly the same thing then -parentproctype can be used to share common functionality. Also there isn't any formal association between proctype and code name file name e.g. a process that loads code/processes/rdb.q can have any proctype we like.

How TorQ [manages connections](http://aquaqanalytics.github.io/TorQ/conn/) is important. Avoid using hopen, use TorQ connection management.
How TorQ [manages connections](http://dataintellecttech.github.io/TorQ/conn/) is important. Avoid using hopen, use TorQ connection management.

TorQ uses the fail fast principle (if you are going to fail, may as well do it as quickly as possible). This helps avoid processes starting up in inconsistent or unexpected states. If running a process with the -debug option, add the -stop or -trap options to stop at, or trap and continue through, start up errors.

Expand Down
Loading

0 comments on commit edad5b0

Please sign in to comment.