Skip to content

Commit

Permalink
Docs: Update backfill info (#1862)
Browse files Browse the repository at this point in the history
* update backfill info

* extra backfill mode info
  • Loading branch information
aeluce authored Jan 14, 2025
1 parent bdfea3c commit 95cfec4
Show file tree
Hide file tree
Showing 12 changed files with 124 additions and 38 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -130,12 +130,12 @@ If you are unable to set the `time_zone` in the database and need to capture tab

## Backfills and performance considerations

When the a MariaDB capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the MariaDB capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,12 @@ a read replica:

## Backfills and performance considerations

When the a MariaDB capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the MariaDB capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
12 changes: 6 additions & 6 deletions site/docs/reference/Connectors/capture-connectors/MySQL/MySQL.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,12 +205,12 @@ If you are unable to set the `time_zone` in the database and need to capture tab

## Backfills and performance considerations

When the a MySQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the MySQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand All @@ -228,11 +228,11 @@ See [connectors](/concepts/connectors.md#using-connectors) to learn more about u
| **`/user`** | Login User | The database user to authenticate as. | string | Required, `"flow_capture"` |
| **`/password`** | Login Password | Password for the specified database user. | string | Required |
| `/timezone` | Timezone | Timezone to use when capturing datetime columns. Should normally be left blank to use the database's `'time_zone'` system variable. Only required if the `'time_zone'` system variable cannot be read and columns with type datetime are being captured. Must be a valid IANA time zone name or +HH:MM offset. Takes precedence over the `'time_zone'` system variable if both are set. | string | |
| `/advanced/dbname` | Database Name | The name of database to connect to. In general this shouldn't matter. The connector can discover and capture from all databases it's authorized to access. | string | `"mysql"` |
| `/advanced/node_id` | Node ID | Node ID for the capture. Each node in a replication cluster must have a unique 32-bit ID. The specific value doesn't matter so long as it is unique. If unset or zero the connector will pick a value. | integer | |
| `/advanced/dbname` | Database Name | The name of the database to connect to. In general this shouldn't matter. The connector can discover and capture from all databases it's authorized to access. | string | `"mysql"` |
| `/advanced/node_id` | Node ID | Node ID for the capture. Each node in a replication cluster must have a unique 32-bit ID. The specific value doesn't matter so long as it is unique. If unset or zero the connector will pick a value. | integer | |
| `/advanced/skip_backfills` | Skip Backfills | A comma-separated list of fully-qualified table names which should not be backfilled. | string | |
| `/advanced/backfill_chunk_size` | Backfill Chunk Size | The number of rows which should be fetched from the database in a single backfill query. | integer | `131072` |
| `/advanced/skip_binlog_retention_check` | Skip Binlog Retention Sanity Check | Bypasses the 'dangerously short binlog retention' sanity check at startup. Only do this if you understand the danger and have a specific need. | boolean | |
| `/advanced/skip_binlog_retention_check` | Skip Binlog Retention Sanity Check | Bypasses the 'dangerously short binlog retention' sanity check at startup. Only do this if you understand the danger and have a specific need. | boolean | |

#### Bindings

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,12 @@ If you are unable to set the `time_zone` in the database and need to capture tab

## Backfills and performance considerations

When the a MySQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the MySQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,12 +94,12 @@ If you are unable to set the `time_zone` in the database and need to capture tab

## Backfills and performance considerations

When the a MySQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the MySQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -223,12 +223,12 @@ ALTER PUBLICATION flow_publication ADD TABLE public.flow_watermarks, <other_tabl

## Backfills and performance considerations

When the a PostgreSQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the PostgreSQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,12 @@ ALTER SYSTEM SET wal_level = logical;

## Backfills and performance considerations

When the a PostgreSQL capture is initiated, by default, the connector first *backfills*, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the PostgreSQL capture is initiated, by default, the connector first *backfills*, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,12 @@ under the name of the root table) but is not required.

## Backfills and performance considerations

When the a PostgreSQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the PostgreSQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,12 @@ under the name of the root table) but is not required.

## Backfills and performance considerations

When the a PostgreSQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the PostgreSQL capture is initiated, by default, the connector first _backfills_, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ For information about configuring allowed IPs in Neon, see [Configure IP Allow](
postgres://cdc_role:[email protected]/dbname?sslmode=require
```
Enter the details for **your connection string** into the source connector fields. Based on the sample connection string above, the values would be specified as shown below. Your values will differ.

- Name: Name of the Capture connector
- Server Address: ep-cool-darkness-123456.us-east-2.aws.neon.tech:5432
- User: cdc_role
Expand All @@ -122,12 +122,12 @@ Optionally, you can change the name of the destination name for each table. You

## Backfills and performance considerations

When the a PostgreSQL capture is initiated, by default, the connector first *backfills*, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the PostgreSQL capture is initiated, by default, the connector first *backfills*, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
8 changes: 4 additions & 4 deletions site/docs/reference/Connectors/capture-connectors/alloydb.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ It's available for use in the Flow web application. For local development or ope

## Prerequisites

You'll need a AlloyDB database setup with the following:
You'll need an AlloyDB database setup with the following:

* Logical decoding enabled
* User role with `REPLICATION` attribute
Expand Down Expand Up @@ -50,12 +50,12 @@ in the same Google Cloud project as your instance.

## Backfills and performance considerations

When the a AlloyDB capture is initiated, by default, the connector first *backfills*, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.
When the AlloyDB capture is initiated, by default, the connector first *backfills*, or captures the targeted tables in their current state. It then transitions to capturing change events on an ongoing basis.

This is desirable in most cases, as in ensures that a complete view of your tables is captured into Flow.
This is desirable in most cases, as it ensures that a complete view of your tables is captured into Flow.
However, you may find it appropriate to skip the backfill, especially for extremely large tables.

In this case, you may turn of backfilling on a per-table basis. See [properties](#properties) for details.
In this case, you may turn off backfilling on a per-table basis. See [properties](#properties) for details.

## Configuration

Expand Down
Loading

0 comments on commit 95cfec4

Please sign in to comment.