add optimal parameter configurations (pingcap#4728)

* add optimal parameters configuration

* Apply suggestions from code review

Co-authored-by: TomShawn <[email protected]>

Co-authored-by: TomShawn <[email protected]>

check-before-deployment.md

@@ -241,13 +241,17 @@ sudo systemctl start ntpd.service && \
 sudo systemctl enable ntpd.service
 ```
 
-## Check and disable THP
+## Check and configure the optimal parameters of the operating system
 
-It is **NOT** recommended to use Transparent Huge Pages (THP) for database applications, because databases often have sparse rather than continuous memory access patterns. If high-level memory fragmentation is serious, a higher latency will occur when THP pages are allocated. If the direct compaction is enabled for THP, the CPU usage will surge. Therefore, it is recommended to disable THP.
+For TiDB in the production environment, it is recommended to optimize the operating system configuration in the following ways:
 
-Take the following steps to check whether THP is disabled. If not, follow the steps to disable it:
+1. Disable THP (Transparent Huge Pages). The memory access pattern of databases tends to be sparse rather than consecutive. If the high-level memory fragmentation is serious, higher latency will occur when THP pages are allocated. 
+2. Set the I/O Scheduler of the storage media to `noop`. For the high-speed SSD storage media, the kernel's I/O scheduling operations can cause performance loss. After the Scheduler is set to `noop`, the performance is better because the kernel directly sends I/O requests to the hardware without other operations. Also, the noop Scheduler is better applicable.
+3. Choose the `performance` mode for the cpufrequ module which controls the CPU frequency. The performance is maximized when the CPU frequency is fixed at its highest supported operating frequency without dynamic adjustment.
 
-1. Execute the following command to see whether THP is enabled or disabled. If `[always] madvise never` is output, THP is enabled.
+Take the following steps to check the current operating system configuration and configure optimal parameters:
+
+1. Execute the following command to see whether THP is enabled or disabled:
 
     {{< copyable "shell-regular" >}}
 
@@ -259,126 +263,272 @@ Take the following steps to check whether THP is disabled. If not, follow the st
     [always] madvise never
     ```
 
-2. Execute the `grubby` command to see the default kernel version:
-
     > **Note:**
     >
-    > Before you execute `grubby`, you should first install the `grubby` package.
+    > If `[always] madvise never` is output, THP is enabled. You need to disable it.
+
+2. Execute the following command to see the I/O Scheduler of the disk where the data directory is located. Assume that you create data directories on both sdb and sdc disks:
 
     {{< copyable "shell-regular" >}}
 
     ```bash
-    grubby --default-kernel
+    cat /sys/block/sd[bc]/queue/scheduler
     ```
 
-    ```bash
-    /boot/vmlinuz-3.10.0-957.el7.x86_64
+    ```
+    noop [deadline] cfq
+    noop [deadline] cfq
     ```
 
-3. Execute `grubby --update-kernel` to modify the kernel configuration:
+    > **Note:**
+    >
+    > If `noop [deadline] cfq` is output, the I/O Scheduler for the disk is in the `deadline` mode. You need to change it to `noop`.
+
+3. Execute the following command to see the `ID_SERIAL` of the disk:
 
     {{< copyable "shell-regular" >}}
 
     ```bash
-    grubby --args="transparent_hugepage=never" --update-kernel /boot/vmlinuz-3.10.0-957.el7.x86_64
+    udevadm info --name=/dev/sdb | grep ID_SERIAL
+    ```
+
+    ```
+    E: ID_SERIAL=36d0946606d79f90025f3e09a0c1f9e81
+    E: ID_SERIAL_SHORT=6d0946606d79f90025f3e09a0c1f9e81
     ```
 
     > **Note:**
     >
-    > `--update-kernel` is followed by the actual default kernel version.
+    > If multiple disks are allocated with data directories, you need to execute the above command several times to record the `ID_SERIAL` of each disk.
 
-4. Execute `grubby --info` to see the modified default kernel configuration:
+4. Execute the following command to see the power policy of the cpufreq module:
 
     {{< copyable "shell-regular" >}}
 
     ```bash
-    grubby --info /boot/vmlinuz-3.10.0-957.el7.x86_64
+    cpupower frequency-info --policy
+    ```
+
+    ```
+    analyzing CPU 0:
+    current policy: frequency should be within 1.20 GHz and 3.10 GHz.
+                  The governor "powersave" may decide which speed to use within this range.
     ```
 
     > **Note:**
     >
-    > `--info` is followed by the actual default kernel version.
+    > If `The governor "powersave"` is output, the power policy of the cpufreq module is `powersave`. You need to modify it to `performance`. If you use a virtual machine or a cloud host, the output is usually `Unable to determine current policy`, and you do not need to change anything.
 
-    ```bash
-    index=0
-    kernel=/boot/vmlinuz-3.10.0-957.el7.x86_64
-    args="ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8 transparent_hugepage=never"
-    root=/dev/mapper/centos-root
-    initrd=/boot/initramfs-3.10.0-957.el7.x86_64.img
-    title=CentOS Linux (3.10.0-957.el7.x86_64) 7 (Core)
-    ```
+5. Configure optimal parameters of the operating system:
 
-5. Execute `reboot` to reboot the machine or modify the current kernel configuration:
+    + Method one: Use tuned (Recommended)
 
-    - To reboot the machine for the configuration change above to take effect, execute `reboot`:
+        1. Execute the `tuned-adm list` command to see the tuned profile of the current operating system:
 
-        {{< copyable "shell-regular" >}}
+            {{< copyable "shell-regular" >}}
 
-        ```bash
-        reboot
-        ```
+            ```bash
+            tuned-adm list
+            ```
 
-    - If you do not want to reboot the machine, modify the current kernel configuration to take effect immediately:
+            ```
+            Available profiles:
+            - balanced                    - General non-specialized tuned profile
+            - desktop                     - Optimize for the desktop use-case
+            - hpc-compute                 - Optimize for HPC compute workloads
+            - latency-performance         - Optimize for deterministic performance at the cost of increased power consumption
+            - network-latency             - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance
+            - network-throughput          - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks
+            - powersave                   - Optimize for low power consumption
+            - throughput-performance      - Broadly applicable tuning that provides excellent performance across a variety of common server workloads
+            - virtual-guest               - Optimize for running inside a virtual guest
+            - virtual-host                - Optimize for running KVM guests
+            Current active profile: balanced
+            ```
 
-        {{< copyable "shell-regular" >}}
+            The output `Current active profile: balanced` means that the tuned profile of the current operating system is `balanced`. It is recommended to optimize the configuration of the operating system based on the current profile.
 
-        ```bash
-        echo never > /sys/kernel/mm/transparent_hugepage/enabled
-        echo never > /sys/kernel/mm/transparent_hugepage/defrag
-        ```
+        2. Create a new tuned profile:
 
-6. Check the default kernel configuration that has taken effect after reboot or modification. If `always madvise [never]` is output, THP is disabled.
+            {{< copyable "shell-regular" >}}
 
-    {{< copyable "shell-regular" >}}
+            ```bash
+            mkdir /etc/tuned/balanced-tidb-optimal/
+            vi /etc/tuned/balanced-tidb-optimal/tuned.conf
+            ```
 
-    ```bash
-    cat /sys/kernel/mm/transparent_hugepage/enabled
-    ```
+            ```
+            [main]
+            include=balanced
 
-    ```bash
-    always madvise [never]
-    ```
+            [cpu]
+            governor=performance
 
-7. If THP is still enabled, use tuned or ktune (two tools used for dynamic kernel debugging) to modify the THP configuration. Take the following steps:
+            [vm]
+            transparent_hugepages=never
 
-    1. Activate tuned:
+            [disk]
+            devices_udev_regex=(ID_SERIAL=36d0946606d79f90025f3e09a0c1fc035)|(ID_SERIAL=36d0946606d79f90025f3e09a0c1f9e81)
+            elevator=noop
+            ```
 
-        {{< copyable "shell-regular" >}}
+            The output `include=balanced` means to add the optimization configuration of the operating system to the current `balanced` profile.
 
-        ```bash
-        tuned-adm active
-        ```
+        3. Apply the new tuned profile:
 
-        ```bash
-        Current active profile: virtual-guest
-        ```
+            {{< copyable "shell-regular" >}}
 
-    2. Create a new custom profile:
+            ```bash
+            tuned-adm profile balanced-tidb-optimal
+            ```
 
-        {{< copyable "shell-regular" >}}
+    + Method two: Configure using scripts. Skip this method if you already use method one.
 
-        ```bash
-        mkdir /etc/tuned/virtual-guest-no-thp
-        vi /etc/tuned/virtual-guest-no-thp/tuned.conf
-        ```
+        1. Execute the `grubby` command to see the default kernel version:
 
-        ```bash
-        [main]
-        include=virtual-guest
+            > **Note:**
+            >
+            > Install the `grubby` package first before you execute `grubby`.
 
-        [vm]
-        transparent_hugepages=never
-        ```
+            {{< copyable "shell-regular" >}}
 
-    3. Apply the new custom profile:
+            ```bash
+            grubby --default-kernel
+            ```
 
-        {{< copyable "shell-regular" >}}
+            ```bash
+            /boot/vmlinuz-3.10.0-957.el7.x86_64
+            ```
 
-        ```bash
-        tuned-adm profile virtual-guest-no-thp
-        ```
+        2. Execute `grubby --update-kernel` to modify the kernel configuration:
+
+            {{< copyable "shell-regular" >}}
+
+            ```bash
+            grubby --args="transparent_hugepage=never" --update-kernel /boot/vmlinuz-3.10.0-957.el7.x86_64
+            ```
+
+            > **Note:**
+            >
+            > `--update-kernel` is followed by the actual default kernel version.
+
+        3. Execute `grubby --info` to see the modified default kernel configuration:
+
+            {{< copyable "shell-regular" >}}
+
+            ```bash
+            grubby --info /boot/vmlinuz-3.10.0-957.el7.x86_64
+            ```
+
+            > **Note:**
+            >
+            > `--info` is followed by the actual default kernel version.
+
+            ```
+            index=0
+            kernel=/boot/vmlinuz-3.10.0-957.el7.x86_64
+            args="ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8 transparent_hugepage=never"
+            root=/dev/mapper/centos-root
+            initrd=/boot/initramfs-3.10.0-957.el7.x86_64.img
+            title=CentOS Linux (3.10.0-957.el7.x86_64) 7 (Core)
+            ```
+
+        4. Modify the current kernel configuration to immediately disable THP:
+
+            {{< copyable "shell-regular" >}}
+
+            ```bash
+            echo never > /sys/kernel/mm/transparent_hugepage/enabled
+            echo never > /sys/kernel/mm/transparent_hugepage/defrag
+            ```
+
+        5. Configure the I/O Scheduler in the udev script:
 
-    4. Check again whether THP is disabled.
+            {{< copyable "shell-regular" >}}
+
+            ```bash
+            vi /etc/udev/rules.d/60-tidb-schedulers.rules
+            ```
+
+            ```
+            ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36d0946606d79f90025f3e09a0c1fc035", ATTR{queue/scheduler}="noop"
+            ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_SERIAL}=="36d0946606d79f90025f3e09a0c1f9e81", ATTR{queue/scheduler}="noop"
+
+            ```
+
+        6. Apply the udev script:
+
+            {{< copyable "shell-regular" >}}
+
+            ```bash
+            udevadm control --reload-rules
+            udevadm trigger --type=devices --action=change
+            ```
+
+        7. Create a service to configure the CPU power policy:
+
+            {{< copyable "shell-regular" >}}
+
+            ```bash
+            cat  >> /etc/systemd/system/cpupower.service << EOF
+            [Unit]
+            Description=CPU performance
+            [Service]
+            Type=oneshot
+            ExecStart=/usr/bin/cpupower frequency-set --governor performance
+            [Install]
+            WantedBy=multi-user.target
+            EOF
+            ```
+
+        8. Apply the CPU power policy configuration service:
+
+            {{< copyable "shell-regular" >}}
+
+            ```bash
+            systemctl daemon-reload
+            systemctl enable cpupower.service
+            systemctl start cpupower.service
+            ```
+
+6. Execute the following command to verify the THP status:
+
+    {{< copyable "shell-regular" >}}
+
+    ```bash
+    cat /sys/kernel/mm/transparent_hugepage/enabled
+    ```
+
+    ```
+    always madvise [never]
+    ```
+
+7. Execute the following command to verify the I/O Scheduler of the disk where the data directory is located:
+
+    {{< copyable "shell-regular" >}}
+
+    ```bash
+    cat /sys/block/sd[bc]/queue/scheduler
+    ```
+
+    ```
+    [noop] deadline cfq
+    [noop] deadline cfq
+    ```
+
+8. Execute the following command to see the power policy of the cpufreq module:
+
+    {{< copyable "shell-regular" >}}
+
+    ```bash
+    cpupower frequency-info --policy
+      ```
+
+    ```
+    analyzing CPU 0:
+    current policy: frequency should be within 1.20 GHz and 3.10 GHz.
+                  The governor "performance" may decide which speed to use within this range.
+    ```
 
 ## Manually configure the SSH mutual trust and sudo without password
 
@@ -442,7 +592,7 @@ This section describes how to manually configure the SSH mutual trust and sudo w
 This section describes how to install the NUMA tool. In online environments, because the hardware configuration is usually higher than required, to better plan the hardware resources, multiple instances of TiDB or TiKV can be deployed on a single machine. In such scenarios, you can use NUMA tools to prevent the competition for CPU resources which might cause reduced performance.
 
 > **Note:**
-> 
+>
 > - Binding cores using NUMA is a method to isolate CPU resources and is suitable for deploying multiple instances on highly configured physical machines.
 > - After completing deployment using `tiup cluster deploy`, you can use the `exec` command to perform cluster level management operations.