KubeBlocks v0.6.4

What's Changed

• fix: merge dags lost the dependencies of sub-dags by @leon-inf in #5403

Full Changelog: v0.6.3...v0.6.4

KubeBlocks v0.6.3

What's Changed

• fix: probe crash when create cm failed by @xuriwuyun in #5260
• feat: support to specify volume mode when creating clusters by @leon-inf in #5333
• fix: reduce mongodb log size by @xuriwuyun in #5334

Full Changelog: v0.6.2...v0.6.3

KubeBlocks v0.6.2

What's Changed

• feat: mongodb support ft by @xuriwuyun in #4960
• fix: mongodb connect by @xuriwuyun in #5032
• chore: add options --set logCollector.enabled for apecloud-mysql addon by @sophon-zt in #5042

Full Changelog: v0.6.1...v0.6.2

KubeBlocks v0.6.1

Full Changelog: v0.6.0...v0.6.1

KubeBlocks v0.6.0

KubeBlocks 0.6.0 (2023-08-18)

We are happy to announce the official release of KubeBlocks 0.6.0! 🚀 🎉 🎈

This version supports stream computing engines Kafka and Pulsar and supports MySQL read-write splitting, introduces an interactive parameter configuration method.

We would like to extend our appreciation to all contributors who helped make this release happen.

Highlights

• KubeBlocks supports Kafka v3.3
  • Apache Kafka is an open-source distributed event storage and streaming computing system that provides high reliability, throughput, and low latency for data pipelines, streaming analysis, and data integration. It is widely used in log collection and metric monitoring scenarios. KubeBlocks supports Kafka v3.3, which announced that KRaft meets production requirements, providing better partition scalability and resilience, and saving the additional costs of ZooKeeper. In addition, KubeBlocks also supports pushing data changes from MySQL and PostgreSQL to Kafka, making it easier for users to further process and handle the data.
• KubeBlocks supports Pulsar v2.11
  • Apache Pulsar is an open-source distributed messaging and stream processing platform. It aims to provide scalability, high performance, and reliability to meet the needs of modern data processing and real-time messaging. KubeBlocks supports Apache Pulsar v2.11, and compared to traditional deployment methods, KubeBlocks can automate fault tolerance, scaling, and other day2 operations.
• KubeBlocks supports MySQL read-write splitting
  • Read-write splitting is designed to improve the read-only processing capability of a MySQL database cluster. All write queries are sent to the master node, while read-only queries that do not modify data are distributed to multiple slave nodes. Read-write splitting is used together with the MySQL Raft Group cluster, which automatically detects changes in the master node and uses the current master node of the cluster to achieve failover. Set read_write_splitting_policy as read-write splitting at the global or session level, and the default policy is LEAST_CURRENT_OPERATIONS, which routes read-only queries to the slave node with the least active operations. The MySQL Raft Group cluster supports up to 5 nodes.

Breaking changes

• For KubeBlocks v0.6, newly created Redis and PG clusters need password to connect, while v0.5 used to allow password-free login
• For KubeBlocks v0.6, the backup and restore function is updated greatly. You need to manually upgrade the 0.5 version cluster to 0.6 and update backupPolicy to use the new function.
• For KubeBlocks v0.6, since password-free login function is not supported for Postgres cluster, when restoring cluster upgraded from v0.5, if the new restored cluster is in creating status for a long time, you can check the pod logs to see whether there is password authentication failed error, you can update password to fix this.

Acknowledgements

Thanks to everyone who made this release possible!

What's New

Pulsar

• Cluster lifecycle management and operation management, supporting the creation of Pulsar clusters, cluster deletion, cluster restarting, horizontal scaling, vertical scaling, volume expanding, and parameter modifying.
• For monitoring, support performance monitoring of CPU, memory, network read/write traffic, and more for ZooKeeper, BookKeeper, and Broker.

Kafka

• Cluster lifecycle management and operation management, supporting cluster creation, deletion, horizontal scaling, vertical scaling, volume expanding, and modifying parameters.
  • Horizontal scaling: In combined mode, broker replicas support 1, 3, or 5 copies. In separated mode, brokers support 1 to 100 copies and Kafka controllers support 1, 3, or 5 copies.
• For monitoring, supports performance monitoring of Broker's CPU, memory, network read/write traffic, etc.

MySQL

• MySQL RaftGroup Cluster supports Switchover
• MySQL RaftGroup Cluster supports MySQL read-write splitting.
• Data Recovery, Non-Destructive Recovery by Point-in-time（PITR）. It's a beta feature.
• Support proxy enable when creating cluster. (Beta)
  • The default specification of Vitess proxy can meet the vast majority of user scenarios. Vitess proxy automatically triggers resource increase or decrease based on changes in database nodes and node specifications, without manual operation. The CPU of Vitess proxy is 1/6 of the total number of CPU cores in the cluster nodes (three nodes), rounded up to 0.5c granularity, with a minimum of 0.5c and a maximum of 64c. The default number of replicas is 1, and currently, modifying the number of replicas is not supported
  • Connection Address: The proxy has a default connection address that supports read-write splitting. The expose command can generate VPC addresses and public network addresses for the proxy connection address.
  • Supports setting Vitess proxy read-write splitting policies.

PostgreSQL

• Support switchover for PG Replication Cluster
• Built-in pgBouncer

MongoDB

• MongoDB replicaset supports switchover
• Data recovery, non-destructive PITR (Beta)

Data migration

• Add kbcli migration command, including creating migration tasks, viewing migration task list, viewing migration task details, terminating migration tasks, viewing logs, viewing migration templates, and other functions. Supports full migration and incremental synchronization.
• Support MySQL data migration from MySQL 8.0 to MySQL 8.0.
• Support PostgreSQL data migration from PostgreSQL 14 to PostgreSQL 14.
• Support MongoDB data migration from MongoDB5.X/6.0 to MongoDB 5.X/6.0.

Compatibility

• Support remote write to Prometheus server through Prometheus v2.41 - 2.45 compatibility testing
• Adapt kbcli to package managers of Ubuntu and Amazon Linux 2
• Adapt kbcli to Windows PowerShell and package managers
• Support local environment running on Ubuntu and Amazon Linux 2 for kbcli playground
• Support local environment running on Windows for kbcli playground

Usability

• kbcli supports users to modify KubeBlocks parameters with local editing tools on the operating system
• kbcli supports the "fault-inject" extension for fault injection
• kbcli supports the "report" command to package cluster context information into a compressed file for problem troubleshooting assistance
• kbcli supports interactive editing of configuration information for DB clusters. For cluster create, support --edit to interactively edit the yaml, and show the cluster summary before create.
• Support canceling Hscale/Vscale OpsRequest that are running (Beta)
• Add kbcli playground Grafana overview page
• Kbcli alert can set email server
• Support initializing the creation of databases and users
• Specify a configuration file for initialization settings when installing KubeBlocks
• Disk full lock for MySQL, PostgreSQL, MongoDB: When the database storage space is about to be full (disk usage is more than 90%), set the database to a read-only state.

Backup and Restore

• The backup storage repository (backupRepo) refers to the target path where backup files are stored. KubeBlocks supports public cloud object storage and S3-compatible object storage.
• Support modifying cluster configuration when restoring a new cluster
• Add "describe-backup" command to view backup details

Observability

• Support integration with external monitoring systems include Prometheus, VictoriaMetrics, and AMP. Cluster performance monitoring metrics are output to the target monitoring system through remote write.
• Real-time dumping of operation logs for K8s/KubeBlocks clusters to object storage, divided by time and size, and providing read-only addresses
• Real-time dumping of error logs for K8s/KubeBlocks clusters to object storage, divided by time and size, and providing read-only addresses

Bug Fixes

• There is a quota problem, no default resource limits are set for containers such as metrics. (#2738, @nashtsai)
• cli ops command output lost ops name (#2904, @ldming)
• probe event lost (#3172, @xuriwuyun)
• update probe internal module (#3414, @xuriwuyun)
• cli playground init cluster without sa, role and rolebinding (#3428, @ldming)
• cli created cluster with wrong sa name (#3500, @ldming)
• The cluster status and pod status are inconsistent. clusters can terminate faster than pods. (#3558, @lynnleelhl)
• enable consensus_set controller by feature gate (#3616, @free6om)
• licensing description (#3664, @free6om)
• change preflight checks rule for low spec machine in test case (#3722, @yangmeilly)
• redis prohibits passwordless access by default (#3726,@Y-Rookie)
• mongodb post start script (#3956, @xuriwuyun)
• mongodb pod terminating takes too long time (#3971, @xuriwuyun)
• mongodb restore problem, reset password (#3987, @xuriwuyun)
• MongoDB does not receive signals when terminating pods. (#3990, @xuriwuyun)
• add more judgements for mongodb replset initialization (#4034, @xuriwuyun)
• parallel_transformer concurrent map writes ([#4079](https:...

KubeBlocks v0.5.3

What's Changed

• chore: pick connection-auth-error related fixes to release by @shanshanying in #3714
• fix: merge service annotations has logic error by @yimeisun in #3830
• chore: have Addon's chartLocationURL attribute setup as '.Values.addonChartLocationBase' template value by @nashtsai in #3899
• fix: set failed when volumesnapshot config error (#3816) by @wangyelei in #3918
• chore: addon support to set private helm registry by @ldming in #3920
• chore: logfile backup fail with the same name cluster by @wangyelei in #3927
• chore: disable performance schema for smaller class by @ldming in #3934
• fix: pitr recoverable time is incorrect by @wangyelei in #3943
• fix: mongodb post start script by @xuriwuyun in #3956
• fix: mongodb pod terminating take too long time by @xuriwuyun in #3972

Full Changelog: v0.5.2...v0.5.3

KubeBlocks v0.5.2

What's Changed

• chore: remove volume from class by @iziang in #3462
• docs: adjust docs 5.0 (#3597) by @ldming in #3600
• docs: cherry pick from main by @michelle-0808 in #3618
• chore: remove mysql classes by @iziang in #3634
• chore: add missing delphic chart by @iziang in #3635
• chore: update postgresql wal relative params and hba.conf by @Y-Rookie in #3638
• fix: do not create fsnotify.Watcher when using sync mode (#3617) by @sophon-zt in #3645
• fix: cli playground error when repeatedly init by @ldming in #3651
• chore: release failed test case by @ldming in #3656
• fix: only call leader when scale by @lynnleelhl in #3677
• chore: cp embedded wesql images by @demian0110 in #3687
• fix: a typo in doc by @xtcyclist in #3704

New Contributors

• @xtcyclist made their first contribution in #3704

Full Changelog: v0.5.1...v0.5.2

KubeBlocks v0.5.1

KubeBlocks 0.5.1

We're happy to announce the release of KubeBlocks 0.5.1! 🚀 🎉 🎈

This release introduces Redis, a key-value database, and MongoDB, a document-based database. It also supports the primary-secondary topology of PostgreSQL, adapts to more public cloud vendors' hosted Kubernetes versions, improves data backup and recovery experiences, and builds basic data migration capability. We noticed that some users may think that K8s reduces database performance. So in this release we include a comparison test result to explain the throughput and RT differences of various MySQL 8.0 deployment forms on AWS.

We would like to extend our appreciation to all contributors who helped make this release happen.

Highlights

• KubeBlocks supports the primary-secondary topology of PostgreSQL
  Users can actively switch the primary-secondary role of the database cluster with kbcli, or passively trigger failover by deleting a specified Kubernetes pod with kubectl. Failover generally completes within 30 seconds when there are no long transactions and large table DDLs.
• KubeBlocks supports Redis v7.0
  Redis is currently the most popular open-source key-value database, supporting data types such as key-value, string, list, set, hash table, and ordered set. It provides extremely fast data read and write operations and is suitable for cache scenarios in e-commerce, social communication, game, and other internet applications. To provide stable, secure, and efficient Redis services to users, KubeBlocks has adopted Redis 7.0 version, which is currently recommended officially, supporting standalone and primary-secondary topologies. Thus, users can perform operations such as creating, deleting, scaling, backing up, restoring, monitoring, alerting, and changing parameters of Redis clusters in development, testing, and production environments.
• KubeBlocks supports MongoDB v5.0
  MongoDB is currently the most popular document-based database, using JSON data types and dynamic schema designs to maintain high flexibility and scalability. KubeBlocks supports the replica set topology of MongoDB v5.0, providing data redundancy and automatic failover capabilities, ensuring data availability and consistency in the event of a node failure. The replica set topology cluster has one primary node (Primary) and several secondary nodes (Secondary), with the primary node handling all write requests and the secondary nodes handling some read requests. If the primary node fails, one of the secondary nodes is elected as the new primary node.
• KubeBlocks supports the private deployment of ChatGPT retrieval plugin
  For users who do not want to expose sensitive information (such as company documents, meeting minutes, emails), OpenAI has open-sourced the ChatGPT retrieval plugin to enhance the ChatGPT experience. As long as users meet OpenAI's requirements, they can run the ChatGPT retrieval plugin through KubeBlocks addon, store the vectorized data of sensitive information in a private database, and enable ChatGPT to have longer memory of the context while ensuring information security.
• KubeBlocks supports one-command launching of playgrounds on Alibaba Cloud, Tencent Cloud, and GCP
  Public cloud vendors' hosted Kubernetes services have significant differences in version, functionality, and integration, so even if the deployment of stateful services is not difficult, but Kubernetes administrators have to do a lot of extra heavy lifting to run stateful services normally. After supporting AWS, KubeBlocks provides the ability to one-command launch playgrounds on Alibaba Cloud, Tencent Cloud, and GCP. Users only need to set up public cloud AK locally, and then execute the kbcli playground init command, and KubeBlocks will automatically apply for resources and configure permissions in the specified region, making it easy for users to experience complete functionality. After trying KubeBlocks out, you can clean up the playground environment with one command to avoid incurring costs.

Breaking changes

• Breaking changes between v0.5 and v0.4. Uninstall v0.4 (including any older version) before installing v0.5.
  • Move the backupPolicyTemplate API from dataprotection group to apps group.
    Before installing v0.5, please ensure that the resources have been cleaned up:

      kubectl delete backuppolicytemplates.dataprotection.kubeblocks.io --all
      kubectl delete backuppolicies.dataprotection.kubeblocks.io --all
    

  • redefines the phase of cluster and component.
    Before installing v0.5, please ensure that the resources have been cleaned up:

      kubectl delete clusters.apps.kubeblocks.io --all
      kubectl delete opsrequets.apps.kubeblocks.io --all
    

• addons.extensions.kubeblocks.io API deleted spec.helm.valuesMapping.jsonMap.additionalProperties, spec.helm.valuesMapping.valueMap.additionalProperties, spec.helm.valuesMapping.extras.jsonMap.additionalProperties and spec.helm.valuesMapping.extras.valueMap.additionalProperties attributes that was introduced by CRD generator, all existing Addons API YAML shouldn't have referenced these attributes.

Known issues and limitations

• Limitations of cluster's horizontal scale operation:
  • Only support VolumeSnapshot API to make a clone of Cluster's PV for syncing data when horizontal scaling.
  • Only 1st pod container and 1st volume mount associated PV will be processed for VolumeSnapshot, do assure that data volume is placed in 1st pod container's 1st volume mount.
  • Unused PVCs will be deleted in 30 minutes after scale in.

If you're new to KubeBlocks, visit the getting started page and get a quick start with KubeBlocks.

Note: This release contains a few breaking changes.

See this section to upgrade KubeBlocks to version 0.5.0.

Acknowledgements

Thanks to everyone who made this release possible!

@1aal, @free6om, @heng4fun, @iziang, @kizuna-lek, @ldming, @nashtsai, @nayutah, @sophon-zt, @TalktoCrystal, @xuriwuyun, @Y-Rookie, @yangmeilly, @ZhaoDiankui

What's Changed

New Features

MySQL

• Support ZEngine storage engine
• Account management supports creating, modifying, and deleting database accounts with different permissions
  PostgreSQL
• Support migration from AWS RDS to KubeBlocks, supporting pre-checks, full migration, and incremental synchronization, verifying the data migration capabilities of CadenceWorkflow and OpenStreetMap
• Support for pgvector extension
• Support for the primary-secondary topology of PostgreSQL
• Automated failover and self-healing
• Support point-in-time recovery
• Account management supports creating, modifying, and deleting database accounts with different permissions

Redis

• Full lifecycle management, including creation, deletion, restart, horizontal/vertical scaling
• Support Redis primary-secondary topology
• Automated failover and self-healing
• Support snapshot backup and recovery
• Metric monitoring, including cluster's basic operation status, connection, OS resources, performance, primary-secondary replication status and other metrics
• Alerts including cluster downtime, OS resource, abnormal connection number, primary-secondary replication abnormality, etc.
• Parameter reconfigure
• Account management

MongoDB

• Full lifecycle management, including creation, deletion, restart, vertical scaling, and disk expansion
• Endpoint exposes the access addresses of all nodes
• File-based full backup and recovery
• Automated failover and self-healing
• Monitoring, alerting and logs
• Parameter reconfigure

Improvements

• add support of Redis cluster (#1886, @heng4fun)
• add support of ApeCloud MySQL-Scale (Vitess) cluster (#2116, @ZhaoDiankui)
• add support of MongoDB cluster (#2182, #2682, @xuriwuyun)
• kbcli playground add supports of more cloud providers (#2241, @ldming)
• add support of standalone Milvus DB (VectorDB) (#2310, @nayutah)
• PostgreSQL add support of pgvector extension (#2406, @ldming)
• complete cleanup of KubeBlocks for playground destroy (#2457, @ldming)
• Addon resources add failed pod logs to status condition errors and events (#3058, @nashtsai)

Bug Fixes

• cli playground use default kubeconfig file (#2150, @ldming)
• update running check (#2174, @xuriwuyun)
• set cluster default storage size to 20Gi (#2254, @ldming)
• cli kubeblocks upgrade command output dashboard info (#2290, @ldming)
• set default storage size to 10Gi for TKE (#2317, @ldming)
• cli playground pull latest cloud provider repo (#2373, @ldming)
• cli playground does not output error message when kubernetes cluster is not ready (#2391, @ldming)
• github action uploads kbcli asset for windows and add powershell script to install on windows (#2449, @1aal)
• trim single quotes for the parameters value in the pg config file ...