-
Notifications
You must be signed in to change notification settings - Fork 224
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add integrations section to docs
- Loading branch information
Showing
6 changed files
with
457 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,150 @@ | ||
--- | ||
title: Data Synchronization | ||
--- | ||
|
||
<Note> | ||
This is a <strong>Permify Cloud</strong> feature. To get more information, [schedule a call with an expert](https://calendly.com/d/cj79-kyf-b4z). | ||
</Note> | ||
|
||
Permify can connect to your databases to capture changes in your application’s database and sync authorization-related data in real time. | ||
|
||
By using **Debezium**, **Kafka**, and the **Permify Sync Service**, your system can efficiently handle updates to permissions and roles without manual intervention. | ||
|
||
## Permify Sync Architecture Overview | ||
|
||
![data-sync](https://github.com/user-attachments/assets/a08ee2f2-7a3d-40df-9751-e8de9b9195d7) | ||
|
||
### 1. Your Databases | ||
|
||
The **databases of your applications** serve as the primary source, holding various data like user information, roles, permissions, and entity relationships. These databases contain the **authorization-related data** that will be captured and synchronized. | ||
|
||
### 2. Debezium | ||
|
||
**Debezium** is responsible for **Change Data Capture (CDC)**, monitoring your application's database for any changes (e.g., in tables like `repository` and `organization`). Debezium detects these changes in real-time and **publishes them to Kafka**, ensuring that any modifications are captured without impacting database performance. | ||
|
||
### 3. Kafka | ||
|
||
**Kafka** acts as the **message broker**, receiving change events from Debezium. It queues these events and passes them on for further processing. | ||
|
||
### 4. Permify Sync Service | ||
|
||
The **Permify Sync Service** is responsible for filtering the incoming data from Kafka. It **identifies authorization-related changes** (such as role assignments or permission updates) and **synchronizes** this data with Permify. Only relevant data is extracted and passed on. | ||
|
||
- **Role**: Filters and processes the events from Kafka to focus on authorization data. | ||
- **Integration**: Ensures that only the necessary authorization changes are synced. | ||
|
||
### 5. Permify | ||
|
||
**Permify** is an open-source authorization service. It applies authorization policies and ensures real-time enforcement of roles, permissions, and relationships based on the data synced from your databases. | ||
|
||
- **Role**: Provides a centralized service to manage and enforce authorization policies. | ||
- **Functionality**: Maintains up-to-date access controls, permissions, and entity relationships within your system, based on predefined policies. | ||
|
||
## Permify Sync Configuration | ||
|
||
You can use the configuration file, which maps changes in database tables to Permify entities and their relationships. | ||
|
||
By syncing the database changes with Permify, you ensure that any updates to the authorization data are handled in real time. | ||
|
||
To give an example of how the configuration works, let's examine the following schema. | ||
|
||
### Example Permify Schema | ||
|
||
The **Permify Schema** defines the entities, relationships, and permissions that are relevant to your application. In this example: | ||
|
||
- **Entities** like `user`, `organization`, and `repository` are defined. | ||
- Relationships between entities, such as `admin` or `member` roles in an organization, are established. | ||
- **Permissions** are specified based on relationships or attributes. For example, a repository can be viewed if it is public (`is_public`), and only admins or owners can edit it. | ||
|
||
```yaml | ||
entity user {} | ||
|
||
entity organization { | ||
relation admin @user | ||
relation member @user | ||
} | ||
|
||
entity repository { | ||
relation parent @organization | ||
relation owner @user | ||
|
||
attribute is_public boolean | ||
|
||
permission view = is_public | ||
permission edit = parent.admin or owner | ||
permission delete = owner | ||
} | ||
|
||
``` | ||
|
||
### Postgres Connector Example Config | ||
|
||
In this example, the **Postgres Connector** is configured to track changes in the `repository` and `organization` tables: | ||
|
||
- **Entities**: Each table is mapped to a Permify entity (e.g., `repository`, `organization`). | ||
- **Relationships**: Columns like `owner_id` or `organization_id` in the database are mapped to relationships like `owner` or `parent` in Permify. | ||
- **Attributes**: Columns such as `is_public` are mapped to entity attributes. | ||
- **Logging and Retry Policy**: This configuration includes logging at the `error` level and a retry policy to handle connection issues with an exponential backoff strategy. | ||
- **Recovery**: Backups are scheduled every 12 hours and stored in an S3 bucket. | ||
|
||
```yaml | ||
# my_postgres_instance.yaml | ||
|
||
connector: postgres | ||
metadata: | ||
name: my_postgres_instance | ||
tables: | ||
- name: repository | ||
permify_entity: repository | ||
primary_key: id | ||
relationships: | ||
- column: owner_id | ||
column_type: uint64 | ||
permify_reference: owner | ||
- column: organization_id | ||
column_type: uint64 | ||
permify_reference: parent | ||
attributes: | ||
- column: is_public | ||
column_type: boolean | ||
permify_reference: is_public | ||
- name: organization | ||
permify_entity: organization | ||
primary_key: id | ||
relationships: | ||
- column: user_id | ||
column_type: uint64 | ||
join_table: organization_admins | ||
join_column: organization_id | ||
join_column_type: uint64 | ||
permify_reference: admin | ||
- column: user_id | ||
column_type: uint64 | ||
join_table: organization_members | ||
join_column: organization_id | ||
join_column_type: uint64 | ||
permify_reference: member | ||
logger: | ||
enabled: true | ||
level: error | ||
retry_policy: | ||
max_retries: 5 | ||
backoff_strategy: exponential | ||
backoff_interval: 2s | ||
recovery: | ||
backup: | ||
enabled: true | ||
schedule: "0 */12 * * *" | ||
location: "s3://backup-bucket/my-postgres-instance-backups" | ||
|
||
``` | ||
|
||
This setup ensures a seamless sync between your PostgreSQL database and Permify for managing authorization logic efficiently. | ||
|
||
The **Permify Sync Service** is applied using a resource configuration, like the one above, through a CLI tool. You can apply this service using the following command: | ||
|
||
```bash | ||
permfiy-sync apply -f ./my_postgres_instance.yaml | ||
``` | ||
|
||
This command ensures that the sync service is properly configured with your database. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,73 @@ | ||
--- | ||
title: Jaeger | ||
--- | ||
|
||
Here’s a step-by-step guide to set up [Jaeger](https://www.jaegertracing.io/) for export Permify traces. | ||
|
||
### Step 1: Install Jaeger in Your Environment | ||
|
||
You can either run Jaeger via Docker or install it locally on your machine. Here's how you can do it with Docker: | ||
|
||
```bash | ||
docker run -d --name jaeger \ | ||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ | ||
-p 5775:5775/udp \ | ||
-p 6831:6831/udp \ | ||
-p 6832:6832/udp \ | ||
-p 5778:5778 \ | ||
-p 16686:16686 \ | ||
-p 14268:14268 \ | ||
-p 14250:14250 \ | ||
-p 9411:9411 \ | ||
jaegertracing/all-in-one:1.32 | ||
``` | ||
|
||
This command runs Jaeger’s all-in-one image with all the services (agent, collector, query service, and UI) on default ports. You can access the Jaeger UI at `http://localhost:16686`. | ||
|
||
### Step 2: Configure Permify for Jaeger Tracing | ||
|
||
In your configuration file, you will set up Jaeger as the exporter and provide the endpoint for exporting the traces. | ||
|
||
Here’s an example configuration: | ||
|
||
```yaml | ||
tracer: | ||
exporter: jaeger | ||
endpoint: http://localhost:14268/api/traces | ||
enabled: true | ||
insecure: true | ||
``` | ||
Explanation: | ||
- `exporter: jaeger`: Specifies that Jaeger will be used as the exporter. | ||
- `endpoint: http://localhost:14268/api/traces`: This is the default Jaeger collector endpoint. | ||
- `enabled: true`: Turns on the tracing feature. | ||
- `insecure: true`: Enables HTTP instead of HTTPS for exporting traces. | ||
|
||
<Info> | ||
Refer to the [Configuration](/setting-up/configuration) section for more details on how to use the configuration file. You can also use environment variables (ENVs) for this. | ||
</Info> | ||
|
||
### Step 3: Start Permify with Jaeger Configuration | ||
|
||
Make sure your configuration file is loaded when starting Permify, and ensure the Jaeger instance is running. | ||
|
||
Permify will now start sending tracing data to the configured Jaeger instance. | ||
|
||
### Step 4: View Traces in the Jaeger UI | ||
|
||
Once you’ve set everything up and your application starts sending requests, you can access Jaeger’s UI at `http://localhost:16686` and observe the traces related to your authorization system. | ||
|
||
### Optional: Use Secure Connection | ||
|
||
If you want to use HTTPS for secure communication with Jaeger, you need to set up the `insecure` option to `false` and provide the appropriate endpoint: | ||
|
||
```yaml | ||
tracer: | ||
exporter: jaeger | ||
endpoint: https://your-jaeger-endpoint | ||
enabled: true | ||
insecure: false | ||
``` | ||
|
||
This sets up Jaeger with the correct security protocols. See more configurations on [Configuration](/setting-up/configuration) section |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
--- | ||
title: OpenTelemetry | ||
--- | ||
|
||
Here’s the step-by-step guide to set up [OpenTelemetry](https://opentelemetry.io/) for export Permify traces, metrics and logs. | ||
|
||
### Step 1: Install OpenTelemetry in Your Environment | ||
|
||
To set up OpenTelemetry, you’ll need to install the OpenTelemetry Collector. You can run it via Docker using the following command: | ||
|
||
```bash | ||
docker run --rm -it -p 4317:4317 -p 4318:4318 otel/opentelemetry-collector:latest | ||
``` | ||
|
||
This command runs the OpenTelemetry Collector, exposing the necessary ports for receiving telemetry data. | ||
|
||
### Step 2: Configure Permify for OpenTelemetry Tracing | ||
|
||
In your configuration file, set up OpenTelemetry as the exporter and provide the endpoint for exporting traces. | ||
|
||
Here’s an example configuration: | ||
|
||
```yaml | ||
tracer: | ||
exporter: otlp | ||
endpoint: http://localhost:4317 | ||
enabled: true | ||
insecure: true | ||
``` | ||
Explanation: | ||
- `exporter: otlp`: Specifies that OpenTelemetry (OTLP) will be used as the exporter. | ||
- `endpoint: http://localhost:4317`: This is the default OpenTelemetry Collector endpoint for receiving traces. | ||
- `enabled: true`: Enables the tracing feature. | ||
- `insecure: true`: Uses HTTP instead of HTTPS for exporting traces. | ||
|
||
<Info> | ||
Refer to the [Configuration](/setting-up/configuration) section for more details on how to use the configuration file. You can also use environment variables (ENVs) for this. | ||
</Info> | ||
|
||
### Step 3: Start Permify with OpenTelemetry Configuration | ||
|
||
Ensure that your configuration file is loaded when starting Permify, and confirm that the OpenTelemetry Collector is running. Permify will now start sending trace data to the OpenTelemetry Collector. | ||
|
||
### Step 4: View Traces in Your Chosen UI | ||
|
||
Once the setup is complete and your application begins sending traces, you can view the trace data in a backend such as Jaeger, Zipkin, or any system supported by the OpenTelemetry Collector. | ||
|
||
### Meter Configuration | ||
|
||
We also have a meter configuration section in the Permify configuration file, which allows you to use the OpenTelemetry Collector to export metric data to other sources, such as ClickHouse. | ||
|
||
**Definition:** | ||
|
||
Configuration for observing metrics like check count, cache check count as well as system details such as Permify version, hostname, OS, and architecture. | ||
|
||
**Structure:** | ||
|
||
```yaml | ||
meter: | ||
exporter: otlp | ||
endpoint: http://localhost:4317 | ||
enabled: true | ||
insecure: true | ||
urlpath: /v1/metrics | ||
``` | ||
|
||
| Required | Argument | Default | Description | | ||
|----------|----------|---------|--------------------------------------------------------------| | ||
| [x] | exporter | otlp | [otlp](https://opentelemetry.io/docs/collector/) is the default. | | ||
| [x] | endpoint | - | The export URI for metric observation. | | ||
| [ ] | enabled | true | Switch option for meter tracing. | | ||
| [ ] | urlpath | /v1/metrics | Allows you to override the default URL path for metrics export. | | ||
| [ ] | insecure | false | Whether to use HTTP instead of HTTPS for exporting metrics. | | ||
|
||
### Example: Exporting Metrics to OpenTelemetry Collector | ||
|
||
To export metrics to the **OpenTelemetry Collector**, you can configure your `meter` section like this: | ||
|
||
```yaml | ||
meter: | ||
enabled: true | ||
exporter: otlp | ||
endpoint: collector:4317 | ||
insecure: true | ||
urlpath: /v1/metrics | ||
``` | ||
|
||
In this setup: | ||
- `endpoint: http://collector:4317`: This is the URL of your OpenTelemetry Collector instance for receiving metrics data. | ||
- `urlpath: /v1/metrics`: The URL path used for sending the metrics (you can override it as needed). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
--- | ||
title: SigNoz | ||
--- | ||
|
||
Here’s a step-by-step guide to set up [SigNoz](https://signoz.io/) for export Permify traces. | ||
|
||
### Step 1: Install SigNoz in Your Environment | ||
|
||
You can install SigNoz via Docker. Run the following command to set up SigNoz: | ||
|
||
```bash | ||
git clone https://github.com/SigNoz/signoz.git | ||
cd signoz/deploy/ | ||
./install.sh | ||
``` | ||
|
||
Once the installation is complete, you can access the SigNoz dashboard at `http://localhost:3301`. | ||
|
||
### Step 2: Configure Permify for SigNoz Tracing | ||
|
||
In your configuration file, set up SigNoz as the exporter and provide the appropriate endpoint for exporting the traces. | ||
|
||
Here’s an example configuration: | ||
|
||
```yaml | ||
tracer: | ||
exporter: signoz | ||
endpoint: http://localhost:4317 | ||
enabled: true | ||
insecure: true | ||
``` | ||
Explanation: | ||
- `exporter: signoz`: Specifies that SigNoz will be used as the exporter. | ||
- `endpoint: http://localhost:4317`: This is the default SigNoz OpenTelemetry Collector endpoint. | ||
- `enabled: true`: Turns on the tracing feature. | ||
- `insecure: true`: Enables HTTP instead of HTTPS for exporting traces. | ||
|
||
<Info> | ||
Refer to the [Configuration](/setting-up/configuration) section for more details on how to use the configuration file. You can also use environment variables (ENVs) for this. | ||
</Info> | ||
|
||
### Step 3: Start Permify with SigNoz Configuration | ||
|
||
Ensure your configuration file is correctly loaded when starting Permify, and verify that the SigNoz instance is running. Permify will now start sending tracing data to SigNoz. | ||
|
||
### Step 4: View Traces in the SigNoz Dashboard | ||
|
||
Once the setup is complete and your application starts sending requests, you can access the SigNoz dashboard at `http://localhost:3301` to observe the traces related to your authorization system. | ||
|
||
### Optional: Use Secure Connection | ||
|
||
If you need to use HTTPS for secure communication with SigNoz, adjust the configuration as follows: | ||
|
||
```yaml | ||
tracer: | ||
exporter: signoz | ||
endpoint: https://your-signoz-endpoint | ||
enabled: true | ||
insecure: false | ||
``` | ||
|
||
This sets up a secure connection for exporting traces to SigNoz. | ||
|
||
See more configurations on [Configuration](/setting-up/configuration) section. |
Oops, something went wrong.