|
| 1 | +--- |
| 2 | +title: "Databento" |
| 3 | +description: Guide to ingest and analyze live multi-stream market data from Databento using QuestDB and Grafana. |
| 4 | +--- |
| 5 | + |
| 6 | +Databento is a market data aggregator that provides a single, |
| 7 | +normalized feed covering multiple venues, |
| 8 | +simplifying the process of ingesting live market data. |
| 9 | +It interfaces well with QuestDB for real-time data analysis and visualization in Grafana. |
| 10 | + |
| 11 | +This guide will show how to ingest live market data from Databento into QuestDB and visualize it using Grafana. |
| 12 | + |
| 13 | +For a deeper dive, see our [Databento & QuestDB blog](/blog/ingesting-live-market-data-data-bento/). |
| 14 | + |
| 15 | +## Prerequisites |
| 16 | + |
| 17 | +- [QuestDB](/download/) |
| 18 | +- [Databento Python client](https://pypi.org/project/databento/) |
| 19 | +- [QuestDB Python client](/docs/clients/ingest-python/) |
| 20 | +- [Grafana](/docs/third-party-tools/grafana/) (Optional) |
| 21 | + |
| 22 | +Install the required Python libraries: |
| 23 | + |
| 24 | +```python |
| 25 | +pip3 install questdb |
| 26 | +pip3 install databento |
| 27 | +``` |
| 28 | + |
| 29 | +## Ingest Data from Databento into QuestDB |
| 30 | + |
| 31 | +### Create Databento Client |
| 32 | + |
| 33 | +Set up a Databento client with your API key: |
| 34 | + |
| 35 | +```python |
| 36 | +import databento as db |
| 37 | + |
| 38 | +db_client = db.Live(key="YOUR_API_KEY") |
| 39 | +``` |
| 40 | + |
| 41 | +### Subscribe to Market Data |
| 42 | + |
| 43 | +Subscribe to a data feed, such as the CME S&P 500 E-Mini futures: |
| 44 | + |
| 45 | +```python |
| 46 | +db_client.subscribe( |
| 47 | +dataset="GLBX.MDP3", |
| 48 | +schema="mbp-1", |
| 49 | +stype_in="raw_symbol", |
| 50 | +symbols="ESM4" |
| 51 | +) |
| 52 | +``` |
| 53 | + |
| 54 | +### Ingest Data into QuestDB |
| 55 | + |
| 56 | +Ingest the data into QuestDB using the Sender class: |
| 57 | + |
| 58 | +```python |
| 59 | +from questdb.ingress import Sender |
| 60 | +import numpy as np |
| 61 | + |
| 62 | +questdb_conf = "http::addr=localhost:9000;username=admin;password=quest;" |
| 63 | +with Sender.from_conf(questdb_conf) as sender: |
| 64 | +sender.row( |
| 65 | +'top_of_book', |
| 66 | +symbols={'instrument': 'ESM4'}, |
| 67 | +columns={'bid_size': record.levels[0].bid_sz, |
| 68 | +'bid': record.levels[0].bid_px*0.000000001, |
| 69 | +'ask': record.levels[0].ask_px*0.000000001, |
| 70 | +'ask_size': record.levels[0].ask_sz}, |
| 71 | +at=np.datetime64(record.ts_event, 'ns').astype('datetime64[ms]').astype(object)) |
| 72 | +sender.flush() |
| 73 | +``` |
| 74 | + |
| 75 | +## Query QuestDB |
| 76 | + |
| 77 | +Now that data is flowing, you can visit QuestDB at [http://localhost:9000](http://localhost:9000/) to try some queries. |
| 78 | + |
| 79 | +Read our [SQL Overview](/docs/reference/sql/overview/) to learn more about the power and depth of querying. |
| 80 | + |
| 81 | +## Visualize in Grafana |
| 82 | + |
| 83 | +After ingesting the data, you can visualize it in Grafana by creating a dashboard with SQL queries such as: |
| 84 | + |
| 85 | +```sql |
| 86 | +SELECT timestamp, instrument, bid, ask |
| 87 | +FROM top_of_book |
| 88 | +WHERE $\_\_timeFilter(timestamp) AND instrument = $symbol |
| 89 | +``` |
| 90 | + |
| 91 | +For more detailed analysis, create multiple charts using Grafana's variable and repeat options. |
| 92 | + |
| 93 | +To learn the basics of QuestDB and Grafana, see [our blog](/blog/time-series-monitoring-dashboard-grafana-questdb/). |
| 94 | + |
| 95 | +You can substitute the demonstration queries with your own! |
| 96 | + |
| 97 | +## Summary |
| 98 | + |
| 99 | +In this guide, we set up a pipeline to ingest live market data from Databento into QuestDB and optionally created a visualization using Grafana. |
| 100 | +This setup allows you to build powerful dashboards and analyze market data efficiently. |
| 101 | + |
| 102 | +For more information, check out [Databento’s documentation](https://databento.com/docs/). |
0 commit comments