autopartitioning-eng

ydb/docs/en/core/concepts/topic.md

@@ -19,18 +19,63 @@ Messages may contain user-defined attributes in "key-value" format. They are ret
 
 ## Partitioning {#partitioning}
 
-To enable horizontal scaling, a topic is divided into `partitions` that are units of parallelism. Each partition has a limited bandwidth. The recommended write speed is 1 MBps.
+To enable horizontal scaling, a topic is divided into `partitions` that are units of parallelism. Each partition has a limited throughput. The recommended write speed is 1 MBps.
 
 {% note info %}
 
 As for now, you can only reduce the number of partitions in a topic by deleting and recreating a topic with a smaller number of partitions.
 
 {% endnote %}
 
+Partitions can be:
+
+ - **Active.** All partitions by default are active. You can both read from an active partition and write into an active partition.
+ - **Inactive.** You can only read from an inactive partition. Partition become inactive after splitting in case of [autopartitioning](#autopartitioning). Inactive partition is deleted automatically after deletion all messages from such partition due to retention period expired.
+
 ### Offset {#offset}
 
 All messages within a partition have a unique sequence number called an `offset` An offset monotonically increases as new messages are written.
 
+## Autopartitioning {#autopartitioning}
+
+Total topic throughput is defined by count of partitions in this topic and throughput of each partition. Count of partitions and throughput of every partition are defined at the moment of the topic creation. If maximum required write speed to a topic is unknown when it is created, you can use autopartitioning to scale this topic automatically. If you switch on autopartitioning up on any topic then count of partitions in this topic will be increased automatically if write speed increases (see [Autopartitioning modes](#autopartitioning_modes)).
+
+### Guarantees {#autopartitioning_guarantee}
+
+1. SDK and server provide exactly-once guarantee of writing in case of partition split. It means that any message will be written once into the parent partition or into the one of the child partitions. Message can not be written into the parent partition and into the child partition at the same time. Moreover, a message can not be written into the one partition several times.
+2. SDK and server provide reading order. First, the data will be read from the parent partition, and after that the data will be read from the child partitions.
+3. So, exactly-once guarantee of writing and reading order guarantee continue to be fulfilled for the specific [(producer-id)](#producer-id).
+
+### Autopartitioning modes {#autopartitioning_modes}
+
+The following autopartition modes are possible for any topic.
+
+#### DISABLED
+
+Autopartitioning is disabled for this topic. In this case count of partitions is a constant, and there is no automatic scaling.
+
+Initial count of partitions is defined during topic creation. In this mode in case of manual change of partitions count new partitions are added. All previously existed partitions remain active as well as new partitions.
+
+#### UP
+
+Autopartitioning up is switched on on this topic. It means that in case of increasing of writing speed into the topic partitions count will be increased automatically. In case of decreasing of writing speed into the topic partitions count remaines unchanged.
+
+Algorithm of partitions count increasing is following. If during defined period of time write speed into the some partitions increases defined threshold (in % of maximum write speed into the partition), this partition is splitted into the two child partitions. Original partition becomes inactive. Only reading is possible from such partition. When retention period expires, and all messages from this partition will be deleted, this partition also will be deleted. Two new child partitions become active, and reading and writing are both possible for these partitions.
+
+#### PAUSED
+
+Autopartitioning is paused for this topic. There is no automatic increasing of partitions count. If necessary, you can switch on autopartitioning up again for this topic.
+
+Examples of YQL-queries for switching between different modes of autopartitioning you can see [here](../yql/reference/syntax/alter-topic.md#autopartitioning).
+
+### Autopartitioning constraints {#autopartitioning_constraints}
+
+There are following constraints during using autopartitioning:
+
+1. If you switch on autopartitioning up on some topic, you can not stop autopartitioning, just pause it.
+2. If you switch on autopartitioning up on some topic, it's impossible to read and write into this topic through [Kafka API](../reference/kafka-api/index.md).
+3. Autopartitioning can be switched on only on topics with reserved capacity mode.
+
 ## Message sources and groups {#producer-id}
 
 Messages are ordered using the `producer_id` and `message_group_id`. The order of written messages is maintained within pairs: `<producer ID, message group ID>`.

ydb/docs/en/core/reference/kafka-api/constraints.md

@@ -8,4 +8,5 @@
 4. Transactions are not supported.
 5. DDL operations are not supported. Use the [{{ ydb-short-name }} SDK](../ydb-sdk/index.md) or [{{ ydb-short-name }} CLI](../ydb-cli/index.md) to perform them.
 6. Data schema validation not supported.
-7. Kafka Connect is only supported in standalone mode.
+7. Kafka Connect is only supported in standalone mode.
+8. If autopartitioning is switched on on some topic you can not read from this topic by Kafka API or write into this topic by Kafka API.

ydb/docs/en/core/yql/reference/yql-core/syntax/alter-topic.md

@@ -141,3 +141,31 @@ ALTER TOPIC `my_topic` RESET (
 * `supported_codecs`: List of [codecs](../../../../concepts/topic#message-codec) supported by the topic. Value type: `String`.
 
 {% endif %}
+
+### Change autopartitioning modes for the topic {#autopartitioning}
+
+Next command set [autopartitioning](../../../../concepts/topic#autopartitioning) into the mode UP:
+
+```yql
+ALTER TOPIC `my_topic` SET (
+    min_active_partitions = 1,
+    max_active_partitions = 5,
+    auto_partitioning_strategy = 'scale_up'
+);
+```
+
+Next command set [autopartitioning](../../../../concepts/topic#autopartitioning) into the mode PAUSED:
+
+```yql
+ALTER TOPIC `my_topic` SET (
+    auto_partitioning_strategy = 'paused'
+);
+```
+
+Next command set [autopartitioning](../../../../concepts/topic#autopartitioning) into the mode UP from mode PAUSED:
+
+```yql
+ALTER TOPIC `my_topic` SET (
+    auto_partitioning_strategy = 'scale_up'
+);
+```