Skip to content

Commit 041aaa6

Browse files
urvisavlaurvisavla
committed
Stellar developer docs for Galexie
1 parent 753be56 commit 041aaa6

File tree

10 files changed

+260
-1
lines changed

10 files changed

+260
-1
lines changed

‎docs/data/README.mdx

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ This section will walk you through the differences between the various platforms
1010

1111
- **[RPC](#rpc)** - live network gateway
1212
- **[Horizon](#horizon)** - API for network state data
13-
- **Galexie** - exports raw ledger metadata files
13+
- **[Galexie](#Galexie)** - exports raw ledger metadata files
1414
- **[Hubble](#hubble)** - analytics database for network data
1515

1616
| Features | RPC | Horizon | Galexie | Hubble |
@@ -70,3 +70,7 @@ Horizon is an API for accessing and interacting with the Stellar network data. I
7070
Horizon stores three types of data (current state, historical state, and derived state) in one database, and the data is available in real-time for transactional use, which makes Horizon more expensive and resource-intensive to operate. If you’re considering using Horizon over the RPC, let us know in the [Stellar Developer Discord](https://discord.gg/stellardev) or file an issue in the [RPC repo](https://github.com/stellar/soroban-rpc) and let us know why!
7171

7272
You can [run your own instance of Horizon](./horizon/admin-guide/README.mdx) or use one of the publicly available Horizon services from [these infrastructure providers](./horizon/horizon-providers.mdx).
73+
74+
## Galexie
75+
76+
Galexie is a tool for exporting Stellar ledger metadata to Google Cloud Storage.

‎docs/data/galexie/REAME.mdx

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
title: Galaxie Introduction
3+
sidebar_position: 0
4+
---
5+
6+
## What is Galexie?
7+
8+
Galexie is a tool for extracting, processing, exporting Stellar ledger metadata to external storage, and creating a data lake of pre-processed ledger metadata. Galaxy is the foundation of the Composable Data Pipeline (CDP) and serves as the first step in extracting raw Stellar ledger metadata and making it accessible. Learn more about CDP’s benefits and applications in this [blog post](https://stellar.org/blog/developers/composable-data-platform).
9+
10+
## What Are the Key Features of Galexie?
11+
12+
Galexie is designed to make streamlined and efficient export of ledger metadata via a simple user-friendly interface. Its key features include:
13+
14+
- Exporting Stellar ledger metadata to cloud storage
15+
- Configurable to export a specified range of ledgers or continuously stream new ledgers as they are created on the Stellar network
16+
- Exporting ledger metadata in XDR which is Stellar Core’s native format.
17+
- Compressing data before export to optimize storage efficiency in the data lake.
18+
19+
**Galexie Architecture**
20+
21+
![](/assets/galexie-architecture.png)
22+
23+
## Why XDR Format?
24+
25+
Exporting data in XDR—the native Stellar Core format—enables Galexie to preserve full transaction metadata, ensuring data integrity while keeping storage efficient. The XDR format maintains compatibility with all Stellar components, providing a solid foundation for applications that require consistent access to historical data. Refer to the [XDR](/docs/learn/encyclopedia/data-format/xdr) documentation for more information on this format.
26+
27+
## Why Run Galexie?
28+
29+
Galexie enables you to make a copy of Stellar ledger metadata over which you have complete control. Galexie can continuously sync your data lake with the latest ledger data freeing you up from tedious data ingestion and allowing you to focus on building customized applications that consume and analyze exported data.
30+
31+
### What Can You Do with the Data Lake Created by Galexie?
32+
33+
Once data is stored in the cloud, it becomes easily accessible for integration with modern data processing and analytics tools, enabling various workflows and insights.
34+
35+
The pre-processed ledger data exported by Galexie can be utilized across various applications, such as:
36+
37+
- Analytics Tools: Analyze trends over time.
38+
- Audit Applications: Retrieve historical transaction data for auditing and compliance.
39+
- Monitoring Systems: Create tools to track network metrics.

‎docs/data/galexie/admin_guide/README.mdx

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
title: Admin Guide
3+
sidebar_position: 15
4+
---
5+
6+
This guide provides step-by-step instructions on installing and running the Galexie.

‎docs/data/galexie/admin_guide/configuring.mdx

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
title: Configuring
3+
sidebar_position: 20
4+
---
5+
6+
# Configuring
7+
8+
## Steps to Configure Galexie
9+
10+
1. **Copy the Sample Configuration**: Use the provided sample configuration file [config.example.toml](https://github.com/stellar/go/blob/master/services/galexie/config.example.toml).
11+
12+
2. **Modify the Configuration File**: Rename the file to `config.toml` and update settings as needed.
13+
14+
### Key settings include:
15+
16+
#### GCS Bucket
17+
18+
Specify the Google Cloud Storage (GCS) bucket where Galexie will export Stellar ledger data. Replace `destination_bucket_path` with the complete path of your GCS bucket, including any subpaths.
19+
20+
Example:
21+
22+
```toml
23+
destination_bucket_path = "stellar-network-data/testnet"
24+
```
25+
26+
#### Stellar Network
27+
28+
Set the Stellar network for which you want to create the data lake.
29+
30+
Example:
31+
32+
```toml
33+
network = "testnet"
34+
```
35+
36+
#### Data Organization (Optional)
37+
38+
Control how the exported data is organized in the GCS bucket. For example, the following configuration adds 64 consecutive ledgers per exported file, and creates a new directory for every 1000 files:
39+
40+
```toml
41+
# Number of ledgers stored in each file
42+
ledgers_per_file = 64
43+
44+
# Number of files per partition/directory
45+
files_per_partition = 1000
46+
```

‎docs/data/galexie/admin_guide/installing.mdx

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
title: Installing
3+
sidebar_position: 30
4+
---
5+
6+
# Installing
7+
8+
To install Galexie, retrieve the Docker image from the [Stellar Docker Hub registry](https://hub.docker.com/r/stellar/stellar-galexie) using the following command:
9+
10+
```shell
11+
docker pull stellar/stellar-galexie
12+
```

‎docs/data/galexie/admin_guide/monitoring.mdx

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
title: Monitoring
3+
sidebar_position: 50
4+
---
5+
6+
# Monitoring

‎docs/data/galexie/admin_guide/prerequisites.mdx

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
title: Prerequisites
3+
sidebar_position: 10
4+
---
5+
6+
# Prerequisites
7+
8+
### 1. Google Cloud Platform (GCP) Account
9+
10+
Galexie exports Stellar ledger metadata to Google Cloud Storage (GCS), so you need a GCP account with:
11+
12+
- Permissions to create a new GCS bucket, or
13+
- Access to an existing bucket with read/write permissions.
14+
15+
### 2. Docker (Recommended)
16+
17+
Galexie is available as a Docker image, which simplifies installation and setup. Ensure you have Docker Engine installed on your system ([Docker installation guide](https://docs.docker.com/engine/install/)).
18+
19+
> **_NOTE:_** While it is possible to natively install Galexie (without Docker), this requires manual dependency management and is recommended only for advanced users.]
20+
21+
### Hardware Requirements
22+
23+
The minimum hardware requirements for running Galexie are:

‎docs/data/galexie/admin_guide/running.mdx

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
title: Running
3+
sidebar_position: 40
4+
---
5+
6+
# Running
7+
8+
With the Docker image available and the configuration file set up, you're now ready to run Galexie and start exporting Stellar ledger data to the GCS bucket.
9+
10+
The primary way of running Galexie is using the `append` command.
11+
12+
## Append Command
13+
14+
Using the `append` command, Galexie can either continuously scan the network for new ledgers and export them, or work on a fixed ledger range and stop when it is exported.
15+
16+
Syntax:
17+
18+
```shell
19+
stellar-galexie append --start <start_ledger> [--end <end_ledger>] [--config-file <config_file>]
20+
```
21+
22+
Arguments:
23+
24+
`--start <start_ledger>` **(required)**
25+
26+
- The starting ledger sequence number of the range being exported.
27+
28+
`--end <end_ledger>` **(optional)**
29+
30+
- The ending ledger sequence number of the range being exported. If unspecified or set to 0, the exporter will continuously export new ledgers as they appear on the network.
31+
32+
`--config-file <config_file_path>` **(optional)**
33+
34+
- The path to the configuration file. If unspecified, the application will look for a file named `config.toml` in the current directory.
35+
36+
Example usage:
37+
38+
```shell
39+
docker run --platform linux/amd64 -d \
40+
-v "$HOME/.config/gcloud/application_default_credentials.json":/.config/gcp/credentials.json:ro \
41+
-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json \
42+
-v ${PWD}/config.toml:/config.toml \
43+
stellar/stellar-galexie \
44+
append --start 350000 --end 450000 --config-file config.toml
45+
```
46+
47+
`--platform linux/amd64`
48+
49+
- Specifies the platform architecture (adjust if needed for your system).
50+
51+
`-v` Mounts volumes to map your local GCP credentials and config.toml file to the container:
52+
53+
- `$HOME/.config/gcloud/application_default_credentials.json`: Your local GCP credentials file.
54+
- `${PWD}/config.toml`: Your local configuration file.
55+
56+
`-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json`
57+
58+
- Sets the environment variable for credentials within the container.
59+
60+
`stellar/stellar-galexie`
61+
62+
- The Docker image name.
63+
64+
### Resumability:
65+
66+
A useful feature of the `append` command is that it supports basic resumability. If Galexie is interrupted while it is exporting ledgers, on restart it scans for the first missing ledger after the specified start ledger. It then resumes exporting from the first missing it detects. Resumability is enabled by default, so when restarting Galexie just use the same start ledger as before, and Galexie will pick up where it left off.
67+
68+
## Scan-and-fill Command
69+
70+
One of the downsides of the `append` command is that it may miss gaps if there are multiple non-sequential gaps in the export range. If you suspect such gaps, use the `scan-and-fill` command. The `scan-and-fill` runs a lot slower than `append` but it comprehensively scans the specified range to detect and fill any missing ledgers, guaranteeing data completeness. Because of its slower speed of execution, this command should be used only when gaps in data are suspected.
71+
72+
Syntax:
73+
74+
```shell
75+
stellar-galexie scan-and-fill --start <start_ledger> --end <end_ledger> [--config-file <config_file>]
76+
```
77+
78+
Arguments:
79+
80+
`--start <start_ledger>` **(required)**
81+
82+
- The starting ledger sequence number of the range being exported.
83+
84+
`--end <end_ledger>` **(required)**
85+
86+
- The ending ledger sequence number of the range being exported.
87+
88+
`--config-file <config_file_path>` **(optional)**:
89+
90+
- The path to the configuration file. If unspecified, the exporter will look for a file named “config.toml” in the current directory.
91+
92+
Example usage:
93+
94+
```shell
95+
docker run --platform linux/amd64 -d \
96+
-v "$HOME/.config/gcloud/application_default_credentials.json":/.config/gcp/credentials.json:ro \
97+
-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json \
98+
-v ${PWD}/config.toml:/config.toml \
99+
stellar/stellar-galexie \
100+
scan-and-fill --start 6400 --end 6800 --config-file config.toml
101+
```

‎docs/data/galexie/admin_guide/setup.mdx

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
title: Setup
3+
sidebar_position: 10
4+
---
5+
6+
# Setup
7+
8+
## Set up your Google Cloud Platform (GCP) credentials
9+
10+
Create application default credentials by using your user account for your GCP project by following these steps:
11+
12+
1. Download the [SDK](https://cloud.google.com/sdk/docs/install).
13+
2. Install and initialize the [gcloud CLI](https://cloud.google.com/sdk/docs/initializing).
14+
3. Create [application default credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc#google-idp) and it should automatically store in this location: `$HOME/.config/gcloud/application_default_credentials.json.`
15+
4. Verify that this file exists before moving on to the next step.
16+
17+
## Create a Google Cloud Storage (GCS) bucket
18+
19+
If you already have a GCS bucket with read and write permissions, you can skip this section. If not, follow these steps:
20+
21+
1. Visit the GCP Console's Storage section (https://console.cloud.google.com/storage) and create a new bucket.
22+
2. Choose a descriptive name for the bucket, such as `stellar-ledger-data`. Refer to [Google Cloud Storage Bucket Naming Guideline](https://cloud.google.com/storage/docs/buckets#naming) for bucket naming conventions. Note down the bucket name, you will need it later during the configuration process.

‎static/assets/galexie-architecture.png

173 KB
Loading

0 commit comments

Comments
 (0)