Update docs as per default tp_index behaviour change #58
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Preview Available 🚀Commit Author: Karan Popat Preview Link: tailpipe-io-git-docs-tpindex-turbot.vercel.app |
kaidaguerre
requested changes
Jun 18, 2025
kaidaguerre
requested changes
Jun 19, 2025
johnsmyth
requested changes
Jun 20, 2025
docs/collect/configure.md
Outdated
| @@ -53,14 +53,14 @@ Tailpipe uses [hive partitioning](https://duckdb.org/docs/data/partitioning/hive | |||
|
|
|||
| - The data is written to Parquet files in the workspace directory, with a prescribed directory and filename structure. Each partition is written to a separate directory. | |||
|
|
|||
| - For [custom tables](/docs/collect/custom-tables), you can define a `tp_index` column on which to index. For tables implemented by plugins, the index is not *user*-definable. Be aware that defining a `tp_index` does not always increase performance and may, in fact, decrease it as it can result in many small parquet files. | |||
| - The `tp_index` is used to partition the data and defaults to `"default"` if not specified. You can configure the `tp_index` in your partition config to specify a column whose value should be used as tp_index. Be aware that defining a `tp_index` does not always increase performance and may, in fact, decrease it as it can result in many small parquet files. | |||
There was a problem hiding this comment.
Add a link here, eg:
... You can configure the `tp_index` in your [partition config](/docs/reference/config-files/partition)
docs/collect/custom-tables.md
Outdated
| @@ -65,6 +65,8 @@ You may also use one or more [`column` definitions](/docs/reference/config-files | |||
|
|
|||
| In our example, the source format does not define a field named `tp_timestamp`. Since ***`tp_timestamp` is a required column***, we will add a `tp_timestamp` column and map the `timestamp` from the source. Also, the source includes a `plugin_timestamp`, but it is parsed as a number because it is epoch milliseconds. We will transform it to a timestamp data type. | |||
|
|
|||
| > [!NOTE] | |||
| > You cannot set the `tp_index` mapping in the table definition. The `tp_index` can only be configured through the partition config, where it defaults to `"default"` if not specified. | |||
There was a problem hiding this comment.
add link:
... configured through the [partition config](/docs/reference/config-files/partition)
docs/faq/index.md
Outdated
| @@ -93,4 +93,4 @@ partition "aws_cloudtrail_log" "cloudtrail_all" { | |||
|
|
|||
| ## What partition indexes are available for a table? | |||
|
|
|||
| That depends on how the plugin author has defined the common `tp_index` field. For AWS tables, it's the `account_id`. In the dual-partition case above, you could carve the logs by `account_id` using the common `tp_partition` field (but `tp_index` will always be the same). In the single-partition case above, you could carve the logs by `account_id` using `tp_index` (but `tp_partition` will always be the same). | |||
| The `tp_index` value depends on how you have configured it in your partition config. By default, `tp_index` is set to `"default"`, but you can configure it to specify a column whose value should be used as the partition index, as makes sense for the data. For AWS tables, you might set it to `account_id`. | |||
There was a problem hiding this comment.
add link:
... depends on how you have configured it in your [partition config](/docs/reference/config-files/partition)
docs/reference/cli/compact.md
Outdated
| @@ -16,6 +16,7 @@ Compact multiple Parquet files per day to one per day. | |||
| | Flag | Description | |||
| |-|- | |||
| | `--help` | Help for compact | |||
| | `--reindex` | Update the `tp_index` field to the currently configured value | |||
There was a problem hiding this comment.
Description should be something more like :
Reorganize data using the currently configured `tp_index` structure. Any data collected using a different `tp_index` value will be rewritten to new files [partitioned](/docs/reference/config-files/partition) using the current `tp_index`.
docs/reference/config-files/table.md
Outdated
| @@ -111,7 +111,7 @@ Tailpipe supports most of the [DuckDB general-purpose data types](https://duckdb | |||
|
|
|||
| Tailpipe tables include a set of common columns. These mappings enable queries that correlate values across different logs. If you have collected both Cloudtrail and ALB logs, for example, you could query for `tp_ips` to find IP addresses in the `aws_cloudtrail_log` and `aws_alb_access_log` tables using the same syntax. | |||
|
|
|||
| When creating a custom table, `tp_timestamp` is the only required column; ***you must define a `tp_timestamp` column***. This is because Tailpipe uses the timestamp to [organize the data files](/docs/collect/configure#hive-partitioning). The `tp_index` is also used in the hive partitioning scheme. You may set it if you want, but it will default to `default` if not set. | |||
| When creating a custom table, `tp_timestamp` is the only required column; ***you must define a `tp_timestamp` column***. This is because Tailpipe uses the timestamp to [organize the data files](/docs/collect/configure#hive-partitioning). The `tp_index` is also used in the hive partitioning scheme. By default, `tp_index` is set to `"default"`, but you can configure it in your partition config to specify a column whose value should be used as the partition index. | |||
There was a problem hiding this comment.
add link:
...but you can configure it in your [partition config](/docs/reference/config-files/partition)
johnsmyth
approved these changes
Jun 20, 2025
kaidaguerre
approved these changes
Jun 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge only after Tailpipe v0.5.0 is released.