2. Data

In the crypto-currency domain, real-time price data is, for most exchanges, freely available. Historical data is oftentimes limited with respect to how far back into past the data is being provided. However, continuously recording real-time data results in a complete historical data set from the time when recording was started and provides the desired data set for this research.

As the focus of this project lies in order execution, the data required is historical limit order book states consisting of bids and asks posted from traders in the past which where willing to buy or sell a certain quantity to a specific price.

Sources

There are various ways in order to build a copy of a historical limit order book. However, the only way to rebuild a complete order book is by processing trade and order events. In case the analysis comes along with either less levels of historical book states, or not every historical state is required, then there are two approaches which require less data processing power.

Trades / Depth updates

The BitcoinExchangeFH project allows to stream

1. price depth updates (change in order book states)
2. and executed trades using web sockets. The received messages can further be stored in a variety of data stores. In our particular case a MySQL database is sufficient to query a set of records.

By default the number of levels reaching away from the spread is 5. This implicitly means that the data represents the order book state at any given point in time, though limited to the specified levels.

As we shall see, 5 levels are not enough in order to explore the matching process in great detail. However, it is a very convenient way to analyze limit orders placed close to the spread.

The data format represents the executed order book state after every trade in the form:

id, trade_px, trade_volume, b1,	b2, b3,	b4, b5,	a1, a2,	a3, a4,	a5, bq1, bq2, bq3, bq4,	bq5, aq1, aq2, aq3, aq4,	aq5, order_date_time, trades_date_time,	update_type

Where “b” stands for bid and “bq” for bid quantity and respectively “a” for ask and “aq” for ask quantity.

Order book states

Bitfinex offers the book endpoint: https://api.bitfinex.com/v1/book/btcusd which returns a snapshot of the order book state up to 50 levels on the bid and ask sides.

Although the order book state is more complete than the native BitcoinExchangeFH approach, the limitation lays in the number of requests (90 per minute). Hence, sub-second book updates cannot be recorded.

The states are represented in the following format:

id, priceBid, priceAsk, volumeBid, volumeAsk, volume, bids, asks, timestamp

Order book events

In order to rebuild a complete order book, exchanges mostly allow to subscribe to web sockets which deliver every market update for a given ticker (trading pair). Thus, this method allows to maintain a copy of the entire order book of an exchange.

Bittrex

The format of an updateExchangeState event received through the Bittrex stocket contains the most recent events in terms of order placement and executed trades. Expressing the response with typescript interfaces makes things more clear:

export interface ExchangeStateUpdate {
    MarketName: string,
    Nounce: number,
    Buys: [Buy],
    Sells: [Sell],
    Fills: [Fill]
}

export type Sell = Buy;

export interface Buy {
    Type: UpdateType,
    Rate: number,
    Quantity: number
}

export interface Fill {
    OrderType: Side,
    Rate: number,
    Quantity: number,
    TimeStamp: string,
}

The information provided by ExchangeStateUpdate a collection of the most recent events at Bittrex.

Update event

Some exchanges combine multiple orders and trades within one message. We segment those events into separate Update events, whereas each event expresses either a buy- or sell order or a filled trade:

export interface Update {
    pair: string,
    seq: number,
    is_trade: boolean,
    is_bid: boolean,
    price: number,
    size: number,
    timestamp: number,
    type: number
}

In case the update was an order placement, the field type (derived from UpdateType) describes the purpose of this order as follows:

• 0: New order entries at matching price
• 1: Cancelled / filled order entries at matching price
• 2: Changed order entries at matching price (partial fills, cancellations)