scylladb · karol-kokoszka · Jun 28, 2024 · Jun 26, 2024 · Jun 26, 2024 · Jun 26, 2024
@@ -44,9 +44,14 @@ Process
 The backup procedure consists of multiple steps executed sequentially.
 
 #. **Snapshot** - Take a snapshot of data on each node (according to backup configuration settings).
-#. (Optional) **Schema** - Upload the schema in CQL text format to the backup storage destination,
+
+   Note that ScyllaDB Manager halts `tablets <https://opensource.docs.scylladb.com/stable/architecture/tablets.html>`_  migration for the duration of this step.
+#. **Schema** - Upload the schema in CQL text format to the backup storage destination,
    this requires that you added the cluster with CQL username and password.
    If you didn't you can :ref:`update the cluster using sctool <cluster-update>` at any point in time.
+
+   Starting from ScyllaDB 6.0 and 2024.2 (and compatible ScyllaDB Manager 3.3),
+   these CQL files are necessary for schema restoration (:ref:`sctool restore --restore-schema <sctool-restore>`).
 #. **Upload** - Upload the snapshot to the backup storage destination.
 #. **Manifest** - Upload the manifest file containing metadata about the backup.
 #. **Purge** - If the retention threshold has been reached, remove the oldest backup from the storage location.

@@ -61,27 +61,37 @@ You may find that the directory contains manifest files with ``.tmp`` suffix.
 Those are manifests files of backups that are being uploaded to the backup location.
 They are are marked as temporary until all the backup files are fully uploaded.
 
+.. _backup-schema-spec:
+
 schema
 ......
 
 The schema directory contains text dumps of database schema.
-They are mainly for information purposes, for restore the backup ``system_schema`` keyspace shall be used.
+
+Starting from ScyllaDB 6.0 and 2024.2 (and compatible ScyllaDB Manager 3.3), they are required for the schema restoration.
+They have ``schema_with_internals.json.gz`` suffix and represent the output of ``DESCRIBE SCHEMA WITH INTERNALS`` CQL query,
+which returns rows in the format ``keyspace|type|name|CQL create statement`` in the correct order.
+
+If you are using an earlier ScyllaDB version, those files are mainly for information purposes.
+To restore the backup, use the ``system_schema`` keyspace.
+They have ``schema.tar.gz`` suffix and represent schema archive divided among keyspaces.
+
 To enable upload of the files make sure that the cluster is added with username and password flags.
 
 .. code-block:: none
 
    schema
    └── cluster
        └── 3e99d4a8-67d2-45fe-87fb-87b1b90ea2dc
-           ├── task_f70117d8-c10e-4e90-9606-2587936b3757_tag_sm_20210809095541UTC_schema.tar.gz
-           ├── task_f70117d8-c10e-4e90-9606-2587936b3757_tag_sm_20210809095542UTC_schema.tar.gz
-           └── task_f70117d8-c10e-4e90-9606-2587936b3757_tag_sm_20210809095748UTC_schema.tar.gz
+           ├── task_f70117d8-c10e-4e90-9606-2587936b3757_tag_sm_20210809095541UTC_schema_with_internals.json.gz
+           ├── task_f70117d8-c10e-4e90-9606-2587936b3757_tag_sm_20210809095542UTC_schema_with_internals.json.gz
+           └── task_f70117d8-c10e-4e90-9606-2587936b3757_tag_sm_20210809095748UTC_schema_with_internals.json.gz
 
 The schema file path is structured as follows.
 
 .. code-block:: none
 
-   meta/cluster/<cluster ID>/task_<task ID>_tag_<snapshot tag>_schema.tar.gz
+   meta/cluster/<cluster ID>/task_<task ID>_tag_<snapshot tag>_schema_with_internals.json.gz
 
 sst
 ...

@@ -19,12 +19,14 @@ To learn more about repairs please consult `this Scylla University lesson <https
 
 .. note:: If you are using ScyllaDB Manager deployed by ScyllaDB Operator, see dedicated `ScyllaDB Operator documentation <https://operator.docs.scylladb.com/stable/manager.html>`_.
 
-Scylla Manager automates the repair process and allows you to configure how and when repair occurs.
-*Scylla Manager repair task* revolves around scheduling many *Scylla repair jobs* with selected ``--intensity`` in ``--parallel``.
+ScyllaDB Manager automates the repair process and allows you to configure how and when repair occurs.
+*ScyllaDB Manager repair task* revolves around scheduling many *ScyllaDB repair jobs* with selected ``--intensity`` in ``--parallel``.
 *Repair task* is responsible for fully repairing all tables selected with ``--keyspace`` parameter, while a single *repair job* repairs
-chosen (by Scylla Manager) token ranges of a given table owned by a specific replica set. All nodes from this replica set take part in
+chosen (by ScyllaDB Manager) token ranges of a given table owned by a specific replica set. All nodes from this replica set take part in
 the repair job and any node can take part only in a single repair job at any given time.
 
+Note that ScyllaDB Manager stops `tablets <https://opensource.docs.scylladb.com/stable/architecture/tablets.html>`_  migration for the duration of repair.
+
 When you create a cluster a repair task is automatically scheduled.
 This task is set to occur each week by default, but you can change it to another time, change its parameters or add additional repair tasks if needed.
 
@@ -34,6 +36,7 @@ Features
 * Glob patterns to select keyspaces or tables to repair
 * Parallel repairs
 * Control over repair intensity and parallelism even for ongoing repairs
+* Ranges batching
 * Repair order improving performance and stability
 * Resilience to schema changes
 * Retries
@@ -71,10 +74,9 @@ so max effective parallelism might change depending on which keyspace is being r
 Repair intensity
 ================
 
-Intensity specifies how many token ranges can be repaired in a Scylla node at every given time.
+Intensity specifies how many token ranges can be repaired in a ScyllaDB node at every given time.
 The default intensity is one, you can change that using :ref:`sctool repair --intensity flag <sctool-repair>`.
 
-Scylla Manager 2.2 adds support for intensity value zero.
 In that case, the number of token ranges is calculated based on node memory and adjusted to ScyllaDB's maximum number of ranges that can be repaired in parallel.
 If you want to repair faster, try using intensity zero.
 
@@ -93,6 +95,16 @@ ScyllaDB Manager avoids repairing more than ``max_repair_ranges_in_parallel`` on
 the max effective intensity for a given repair job is equal to the **minimum** ``max_repair_ranges_in_parallel``
 value of nodes taking part in the job.
 
+Ranges batching
+===============
+
+In order to improve cluster utilization, ScyllaDB Manager sends all ranges owned by given replica set in a single repair job.
+The ``--intensity`` constraint is ensured by the ``ranges_parallelism`` repair job parameter.
+
+Even though this improves repair performance (especially for tablet keyspaces), it reduces task granularity.
+In order to ensure task progress, batching is disabled (ScyllaDB Manager sends ``--intensity`` amount of ranges per repair job),
+when task execution is resumed after finishing with error or when it ran out of the maintenance window (``--window`` flag).
+
 Changing repair speed
 =====================
 

@@ -12,6 +12,9 @@ The following table shows which version of Scylla Manager restore task supports
    * - ScyllaDB Manager Version
      - ScyllaDB Open Source Version
      - ScyllaDB Enterprise Version
+   * - 3.3
+     - 5.4, 6.0
+     - 2023.1, 2024.1
    * - 3.2
      - 5.0, 5.1, 5.2, 5.4
      - 2022.1, 2022.2, 2023.1, 2024.1

@@ -8,6 +8,7 @@ Restore
 
    restore-tables
    restore-schema
+   old-restore-schema
    examples
    compatibility-matrix
 
@@ -34,9 +35,6 @@ Restore task has to be one of two types:
 
   * :doc:`restore schema <restore-schema>` - restores the ScyllaDB cluster schema
 
-Each of those types has required prerequisites and follow-up actions.
-For more information, please read given restore type documentation.
-
 If both the schema and the content of the tables need to be restored, you must start with restoring the schema. Only after the schema is successfully restored can you proceed with restoring the content of the tables.
 
 Features
@@ -58,6 +56,9 @@ Restore speed and granularity
 Restore speed is controlled by two parameters: ``--parallel`` and ``--batch-size``.
 Parallel specifies how many nodes can be used in restore procedure at the same time.
 Batch size specifies how many SSTable bundles can be restored from backup location in a single job.
+Note that increasing the default batch size might significantly increase restore performance,
+as only one shard can work on restoring a single SSTable bundle.
+
 Those parameters can be set when you:
 
 * Schedule a restore with :ref:`sctool restore <sctool-restore>`

@@ -0,0 +1,69 @@
+===============================================
+Restore schema for ScyllaDB 5.4/2024.1 or older
+===============================================
+
+.. note:: Currently, Scylla Manager supports only entire schema restoration, so ``--keyspace`` flag is not allowed.
+
+.. note:: Because of small size of schema files, resuming schema restoration always starts from scratch.
+
+.. include:: _common/restore-raft-schema-warn.rst
+
+| In order to restore ScyllaDB cluster schema use :ref:`sctool restore <sctool-restore>` with ``--restore-schema`` flag.
+| Please note that the term *schema* specifically refers to the data residing in the ``system_schema keyspace``, such as keyspace and table definitions. All other data stored in keyspaces managed by ScyllaDB, such as authentication data in the ``system_auth`` keyspace, is restored as part of the :doc:`restore tables procedure <restore-tables>`.
+| The restore schema procedure works with any cluster size, so the backed-up cluster can have a different number of nodes per data center than the restore destination cluster. However, it is important that the restore destination cluster consists of at least all of the data centers present in the backed-up cluster.
+
+Prerequisites
+=============
+
+* ScyllaDB Manager with CQL credentials to restore destination cluster.
+
+* It is strongly advised to restore schema only into an empty cluster with no schema change history of the keyspace that is going to be restored.
+   Otherwise, the restored schema might be overwritten by the already existing one and cause unexpected errors.
+
+* All nodes in restore destination cluster should be in the ``UN`` state (See `nodetool status <https://docs.scylladb.com/stable/operating-scylla/nodetool-commands/status.html>`_ for details).
+
+Procedure
+=========
+
+This section contains a description of the restore-schema procedure performed by ScyllaDB Manager.
+
+Because of being unable to alter schema tables ``tombstone_gc`` option, restore procedure "simulates ad-hoc repair"
+by duplicating data from **each backed-up node into each node** in restore destination cluster.
+However, the small size of schema files makes this overhead negligible.
+
+    * Validate that all nodes are in the ``UN`` state
+    * For each backup location:
+
+      * Find all ScyllaDB *nodes* with location access and use them for restoring schema from this location
+      * List backup manifests for specified snapshot tag
+    * For each manifest:
+
+        * Filter relevant tables from the manifest
+        * For each table:
+
+          * For each *node* (in ``--parallel``):
+
+            * Download all SSTables
+    * For all nodes in restore destination cluster:
+
+        * `nodetool refresh <https://docs.scylladb.com/stable/operating-scylla/nodetool-commands/refresh.html#nodetool-refresh>`_ on all downloaded schema tables (full parallel)
+
+Follow-up action
+================
+
+After successful restore it is important to perform necessary follow-up action. In case of restoring schema,
+you should make a `rolling restart <https://docs.scylladb.com/stable/operating-scylla/procedures/config-change/rolling-restart.html>`_ of an entire cluster.
+Without the restart, the restored schema might not be visible, and querying it can return various errors.
+
+.. _restore-schema-workaround:
+
+Restoring schema into a cluster with ScyllaDB **5.4.X** or **2024.1.X** with **consistent_cluster_management**
+==============================================================================================================
+
+Restoring schema when using ScyllaDB **5.4.X** or **2024.1.X** with ``consistent_cluster_management: true`` in ``scylla.yaml``
+is not supported. In such case, you should perform the following workaround:
+
+    * Create a fresh cluster with ``consistent_cluster_management: false`` configured in ``scylla.yaml`` and a desired ScyllaDB version.
+    * Restore schema via :ref:`sctool restore <sctool-restore>` with ``--restore-schema`` flag.
+    * Perform `rolling restart <https://docs.scylladb.com/stable/operating-scylla/procedures/config-change/rolling-restart.html>`_ of an entire cluster.
+    * Follow the steps of the `Enable Raft procedure <https://opensource.docs.scylladb.com/stable/architecture/raft.html#enabling-raft>`_.
@@ -1,69 +1,37 @@
-==============
-Restore schema
-==============
+===============================================
+Restore schema for ScyllaDB 6.0/2024.2 or newer
+===============================================
 
-.. note:: Currently, Scylla Manager supports only entire schema restoration, so ``--keyspace`` flag is not allowed.
+.. note:: Currently, ScyllaDB Manager supports only entire schema restoration, so ``--keyspace`` flag is not allowed.
 
-.. note:: Because of small size of schema files, resuming schema restoration always starts from scratch.
+.. note:: Currently, restoring schema containing `alternator tables <https://opensource.docs.scylladb.com/stable/using-scylla/alternator/>`_ is not supported.
 
-.. include:: _common/restore-raft-schema-warn.rst
-
-| In order to restore Scylla cluster schema use :ref:`sctool restore <sctool-restore>` with ``--restore-schema`` flag.
-| Please note that the term *schema* specifically refers to the data residing in the ``system_schema keyspace``, such as keyspace and table definitions. All other data stored in keyspaces managed by ScyllaDB, such as authentication data in the ``system_auth`` keyspace, is restored as part of the :doc:`restore tables procedure <restore-tables>`.
-| The restore schema procedure works with any cluster size, so the backed-up cluster can have a different number of nodes per data center than the restore destination cluster. However, it is important that the restore destination cluster consists of at least all of the data centers present in the backed-up cluster.
+| In order to restore ScyllaDB cluster schema use :ref:`sctool restore <sctool-restore>` with ``--restore-schema`` flag.
+| Please note that the term *schema* specifically refers to the data residing in the ``system_schema keyspace``, such as keyspace and table definitions. All other data stored in keyspaces managed by ScyllaDB is restored as part of the :doc:`restore tables procedure <restore-tables>`.
+| The restore schema procedure works with any cluster size, so the backed-up cluster can have a different number of nodes than the restore destination cluster.
 
 Prerequisites
 =============
 
-* Scylla Manager with CQL credentials to restore destination cluster.
-
-* It is strongly advised to restore schema only into an empty cluster with no schema change history of the keyspace that is going to be restored.
-   Otherwise, the restored schema might be overwritten by the already existing one and cause unexpected errors.
+* ScyllaDB Manager requires CQL credentials with
 
-* All nodes in restore destination cluster should be in the ``UN`` state (See `nodetool status <https://docs.scylladb.com/stable/operating-scylla/nodetool-commands/status.html>`_ for details).
+    * `permission to create <https://opensource.docs.scylladb.com/stable/operating-scylla/security/authorization.html#permissions>`_ restored keyspaces.
 
-Follow-up action
-================
+* No overlapping schema in restore destination cluster (see the procedure below for more details)
 
-After successful restore it is important to perform necessary follow-up action. In case of restoring schema,
-you should make a `rolling restart <https://docs.scylladb.com/stable/operating-scylla/procedures/config-change/rolling-restart.html>`_ of an entire cluster.
-Without the restart, the restored schema might not be visible, and querying it can return various errors.
+* Restore destination cluster must consist of the same DCs as the backed up cluster (see the procedure below for more details)
 
 Procedure
 =========
 
-This section contains a description of the restore-schema procedure performed by ScyllaDB Manager.
-
-Because of being unable to alter schema tables ``tombstone_gc`` option, restore procedure "simulates ad-hoc repair"
-by duplicating data from **each backed-up node into each node** in restore destination cluster.
-Fortunately, the small size of schema files makes this overhead negligible.
-
-    * Validate that all nodes are in the ``UN`` state
-    * For each backup location:
-
-      * Find all Scylla *nodes* with location access and use them for restoring schema from this location
-      * List backup manifests for specified snapshot tag
-    * For each manifest:
-
-        * Filter relevant tables from the manifest
-        * For each table:
-
-          * For each *node* (in ``--parallel``):
-
-            * Download all SSTables
-    * For all nodes in restore destination cluster:
-
-        * `nodetool refresh <https://docs.scylladb.com/stable/operating-scylla/nodetool-commands/refresh.html#nodetool-refresh>`_ on all downloaded schema tables (full parallel)
-
-.. _restore-schema-workaround:
+ScyllaDB Manager simply applies the backed up output of ``DESCRIBE SCHEMA WITH INTERNALS`` via CQL.
 
-Restoring schema into a cluster with ScyllaDB **5.4.X** or **2024.1.X** with **consistent_cluster_management**
-==============================================================================================================
+For this reason, restoring schema will fail when any restored CQL object (keyspace/table/type/...) is already present in the cluster.
+In such case, you should first drop the overlapping schema and then proceed with restore.
 
-Restoring schema when using ScyllaDB **5.4.X** or **2024.1.X** with ``consistent_cluster_management: true`` in ``scylla.yaml``
-is not supported. In such case, you should perform the following workaround:
+Another problem could be that restored keyspace was defined with ``NetworkTopologyStrategy`` containing DCs that are not present in the restore destination cluster.
+This would result in CQL error when trying to create such keyspace.
+In such case, you should manually fetch the backed-up schema file (see :ref:`backup schema specification <backup-schema-spec>`),
+change problematic DC names, and apply all CQL statements.
 
-    * Create a fresh cluster with ``consistent_cluster_management: false`` configured in ``scylla.yaml`` and a desired ScyllaDB version.
-    * Restore schema via :ref:`sctool restore <sctool-restore>` with ``--restore-schema`` flag.
-    * Perform `rolling restart <https://docs.scylladb.com/stable/operating-scylla/procedures/config-change/rolling-restart.html>`_ of an entire cluster.
-    * Follow the steps of the `Enable Raft procedure <https://opensource.docs.scylladb.com/stable/architecture/raft.html#enabling-raft>`_.
+In case of an error, Manager will try to rollback all applied schema changes.
@@ -4,6 +4,8 @@ Restore tables
 
 .. note:: Currently, Scylla Manager does not support restoring content of `CDC log tables <https://docs.scylladb.com/stable/using-scylla/cdc/cdc-log-table.html>`_.
 
+.. warning:: Restoring data related to *authentication* (``system_auth``) and *service levels* (``system_distributed.service_levels``) is not supported in ScyllaDB 6.0.
+
 | To restore the content of the tables (rows), use the :ref:`sctool restore <sctool-restore>` command with the ``--restore-tables`` flag.
 | The restore tables procedure works with any cluster topologies, so the backed-up cluster can have a different number of nodes or data centers than the restore destination cluster.