Author: vivo Internet server team – Zhang Rong

As an open source cloud-native multi-cloud container orchestration project, Karmada has attracted many enterprises to participate in the project development and run in the production environment. At the same time, multi-cloud has gradually become the infrastructure for data center construction, and scenarios such as multi-region disaster recovery and multi-activity, large-scale multi-cluster management, and cross-cloud elasticity and migration have promoted the rapid development of cloud-native multi-cloud-related technologies.

First, the background

As vivo services continue to migrate to k8s, the scale of clusters and the number of clusters have grown rapidly, and the difficulty of O&M has also increased dramatically. In order to build a multi-cluster technology, we also developed our own multi-cluster management, but it could not solve more problems we encountered. After doing meticulous research and testing on community-related projects, we finally chose Karmada.

The main reasons are as follows:

With the unified management capability of multiple sets of K8s clusters, the service manages resources through service dimensions, reducing the management difficulty of the container platform.

Cross-cluster elastic scaling and scheduling capabilities enable rational utilization of resources across clusters, thereby improving resource utilization and saving costs.

Karmada uses entirely K8s-native APIs and has a low cost of retrofitting.

Disaster recovery, Karmada control plane and member cluster decoupling, cluster abnormality support resource reallocation.

Extensibility, such as adding self-developed scheduling plugins and adding self-developed Openkruise interpreter plugins.

As we explored how to use Karmada, we also had problems with Karmada’s own operations.

There are many community deployment tools, which require users to choose themselves. The current user deployment method is as follows:


Karmada charts

Binary deployment

hack the script in the directory

For several of the tools above, a questionnaire was conducted in the community of Karmada and statistical reports were generated.

The main summary is as follows:

There are many community deployment tools, which require users to choose themselves.

The deployment script also has flaws that need to be solved by the user himself, and there are many questions on github about this.

Black screen operation, no k8s api operation, users are difficult to productize, we mainly expect to dock our container platform to achieve visual installation.

Lack of release plans for CI testing and deployment tools.

etcd clusters lack key feature points for production environments, such as high availability, regular backups, and recoveries of etcd.

There are many dependent plugins that need to be installed, involving the Karmada control plane, Karmada’s host cluster, and member cluster.

Lack of one-click deployment and cumbersome configuration and other pain points.

In view of the above problems, this article will share the vivo practice of Karmada-Operator, including Operator’s solution selection, API, architecture design and CI construction.

Second, the implementation of Karmada-Operator

2.1 Introduction to the Operator SDK

The Operator Framework is an open source toolkit for managing Kubernetes-native applications, known as Operators, in an efficient, automated, and extensible way. Operators leverage the scalability of Kubernetes to demonstrate the automation benefits of cloud services, such as provisioning, scaling, and backup and recovery, while running anywhere Kubernetes can run.

Operators help simplify the management of complex, stateful applications on Kubernetes. However, writing an Operator today is not easy and presents challenges such as using low-level APIs, writing boilerplates, and lack of modular functionality (which can lead to duplication of effort).

The Operator SDK is a framework that reduces the difficulty of writing an Operator by providing the following:

Advanced APIs and abstractions for writing operational logic more intuitively

Scaffolding and code generation tools for quickly booting new projects

extensions that cover common Operator use cases

As shown in the figure above, operator SDKs can build operators based on helm, ansilbe, and go, and we need to choose our appropriate operator framework based on the current situation.

2.2 Scheme selection

Scenario one: golang develops the Operator

Solution two: ansible develops Operator

Scenario Three: Golang and Ansible Hybrid Development Operator

According to the actual production deployment of Karmada and vivo’s own practice, it can be summarized as follows:

To support binary deployments in K8s clusters and without relying on K8s clusters.

It supports external independent etcd cluster deployment or docking with existing etcd clusters.

Karmada clusters have migration capabilities, such as room layoffs and machine failures, so etcd cluster management has backup and recovery capabilities, such as quickly restoring clusters in other data centers based on etcd backup data.

Need to support third-party VIP to provide load balancing to Karmada-apiserver, at present vivo is based on external vip, and provides domain names. The service that does not use K8s provides load balancing for Karmada-apiserver.

Karmada control plane one-click deployment and automatic registration and deregistration of member clusters. You also need to get the kubeconfig of the member cluster, and the pull mode also needs to deploy the Karmada-agent in the member cluster.

The installation of the addons plugin of the Karmada cluster, such as istio, anp, third-party crd, etc., needs to be configured on the control plane of the Karmada, the host cluster, and even the member cluster.

Provides APIs for visual deployment.

Individual and full upgrades for individual components of Karmada.

Support in offline and offline modes.

In the face of such complex conditions and limitations of Karmada, let’s analyze who may be more suitable for the above 3 solutions.

Scheme one, based on go-developed Operator, is more suitable for stateful service management based on K8s clusters, such as etcd, database, etc., and more mature etcd-Operator. Karmada involves binary deployment that does not rely on K8s clusters, external etcd, member cluster registration, deregistration, and plugin installation, and cannot be well supported or needs to increase the amount of development.

The second solution is that the Operator developed based on ansible can be based on the state service management of K8s clusters, or it can be separated from K8s clusters for binary deployment, external etcd, member cluster registration, deregistration, and plug-in installation without relying on K8s clusters. This is mainly managed through the ssh login capability and K8s module of ansible, and through the survey we also found that more than 90% of users can provide ssh login.

Scheme three, based on go+ansible’s hybrid Operator, readers can read the Kubernetes-Operator developed by Vivo, which is based on this scheme. Scenario three has all the capabilities of scenario two, because the underlying layer is implemented through ansible.

First of all, we excluded the first plan, for the second and third programs, I also struggled for a long time, and finally we chose the second plan. The main reasons are as follows:

The Operator SDK ansible already has the same capabilities as the Operator SDK go, and provides K8s, K8s_status modules, similar webhook functions to manage K8s’ resources, and reconciliation capabilities.

Meet the needs of the actual production deployment of Karmada today.

It’s easy to learn, just know the jinja template of ansbile and the same yaml file as K8s. You just need to write an ansible task, out of the box, and reconciliation is solved by the Operator SDK.

It is friendly to people who commonly use ansible and does not need to write golang code.

Strong scalability, user-definable plug-ins. The management side also supports local, ssh, and zeromq connections. Local mode can be directly connected to the K8s interface, ssh mode can log in to execute scripts. It can be well mixed to solve our current needs.

Karmada operation is relatively simple compared to K8s, does not require complex crd definitions, ansible needs to parse a small amount of vars to execute the playbook. The golang+ansible model is more suitable for complex CRD definitions and complex business logic systems.

2.3 API design

As shown in the image above, we only need to execute the Operator-SDK create api command to create the CRD of KarmadaDeployment, and then we can define the API of KarmadaDeployment. Implement Reconcile’s business logic in watches.yaml.

Here are mainly defined KarmadaDeployment, EtcdBackup and EtcdRestore resources, respectively, for the deployment of Karmada, and etcd data backup and recovery. The ansible Operator resolves to ansible’s vars according to the definition in the spec. Status is output as a user-defined state via ansible runner. The status of KarmadaDeployment can also be updated through ansible’s k8s_status. The main consideration at present is to run Karmada in K8s, and the binary deployment mode will be added later, which is not covered by the current CR.

2.4 Architecture design

As shown in the figure, Karmada Operator provides containerized and binary cluster deployment design, in which Karmada’s containerized deployment does not require ssh login, and only needs to complete the control of karmada control surface through K8s and k8s_status. Karmada’s binary deployment is mainly through the ssh login to complete the control of the Karmada control plane. The join and unjoin of the member cluster need to provide the kubeconfig file of the member cluster in advance, and you can also set the member’s login permission operation, and you need to define the user and key of the member cluster in the CR.

The execution process is as follows.

Users define Karmada actions through KarmadaDeployment

The Karmada Operator senses the CR changes in KarmadaDeployment and begins to enter the controller logic

According to the user’s definition, select a containerized deployment or a binary deployment to start operations such as installation, capacity expansion, and backup

Perform the join/unjoin operation, register the member cluster with the Karmada cluster, or unregister the member cluster

2.5 Karmada control plane management

As shown in the figure above, mainly karmada control plane lifecycle management, compared with the current community deployment tools we optimize as follows:

Standardized certificate management, mainly generating certificates with openssl. Among them, the etcd and Karmada certificates are maintained separately, and the k8s cluster certificate is named the same, which is convenient for accessing our monitoring.

Karmada-apiserver supports external load balancing, not limited to load balancing provided by the current k8s service.

More flexible upgrade strategy, support for individual component upgrades and full upgrades.

Richer definition of global variables, scheduling support for component configuration changes, etc.

2.6 etcd cluster management

The etcd cluster is Karmada’s metadata cluster, and it is necessary to ensure high availability and failure recovery of the etcd cluster in production. The preceding figure shows the necessary production factors of the etcd cluster, such as automatic scaling, upgrade, backup, and fault recovery of the etcd cluster. We have developed plugins and libraries based on ansible to implement the following ability of etcd cluster management:

Add member to the existing etcd cluster.

The etcd cluster deletes the member.

Backup of etcd clusters, such as data backups that support cephfs.

etcd cluster failure recovery.

etcd cluster health status query.

The CR of etcdBackup and etcdRestore is defined here and is not merged into KarmadaDeployment. Mainly considering the safety of the etcd cluster itself and simplifying KarmadaDeployment’s ansible task. The etcdRestore function can backup data according to the etcd cluster and import it into the new etcd cluster, thereby restoring all the business states of the Karmada cluster. The main scenarios are as follows:

The computer room where the Karmada cluster is located is dismissed, and the etcd data needs to be backed up and migrated to the new Karmada cluster.

It is expected to manage Karmada clusters through Karmada-Operator, simply back up etcd data and implement the etcdRestore function.

Karmada cluster failure, you can back up data through etcd, combined with etcdRestroe to achieve failure recovery.

2.7 Member cluster management

The lifecycle management of the member cluster mainly includes registration and deregistration, and the above figure is the process of execution. To handle the registration and deregistration of the member cluster, an inventory is dynamically generated here. Ansible Inventory is a two-part collection of static Inventory and dynamic Inventory, which refers to the host and group specified in the file, and Dynamic Inventory refers to the list of hosts obtained by an external script and returned to the ansilbe command in the format required by ansi.

Here Karmada-Operator implements dynamic inventory plugins based on k8s’s CR, mainly by parsing the member definition of KarmadaDeployment to dynamically generate inventory. Here add the add-member and del-member 2 roles, the cluster in the add-member will be registered to the Karmada control plane, the del-member cluster will be deregistered from the Karmada control plane, so that multiple member clusters can be registered and deregistered concurrently. At the same time, it can also provide ssh login mode to facilitate later expansion.

III. Introduction to the CI of Karmada-Operator

In order to better improve the developer experience, it is planned to provide CI building capabilities for Karmada-Operator. Here are github’s self-hosted runners and kubevirt deployed in K8s clusters.

Users submit PRs on github

Trigger github Actions, the process we defined in self-hosted

Perform syntax and unit tests

Create VMs through kubevirt

Deploy 1 host and 2 member clusters across multiple VMs

Deploy Karmada and add a member cluster

Perform Karmada e2e and bookfinfo case tests

The CI matrix tests planned to be added are as follows:

Grammar test:






Cluster deployment test:

Karmadactl, charts, yaml, and binary deployments and all configuration item installation tests

join/ unjoin member cluster

Upgrade Karmada

Backup and recovery of etcd clusters

Functional tests:

Karmada e2e test

Create a bookinfo case

Performance test:

We mainly simulated multiple 2000-node member clusters through the kubemark component for performance testing, one of the test cases was cluster failover, and the conclusion was that 4w stateless pods were able to complete the failover in 15 minutes, and there was an opportunity to share our performance tests.

IV. Summary

Through community research and vivo’s practice, the design of Karmada-Operator program was finalized. Karmada-Operator is highly scalable, reliable, more intuitive to write operational logic, and out-of-the-box use, and we believe that managing Karmada through this highly scalable, declarative, self-healing cloud-native system provides a strong and reliable guarantee for us to fully switch to Karmada to manage our business.

An ansible-based operator also has the following drawbacks. The first point does not provide the ability to webhook, you need to add it yourself or add the relevant checks in the ansible task; The second point is that a generic CRD template is automatically generated, and there are no detailed scaffolding tools to automatically generate a CRD.

At present, Karmada-operator is still in the initial stage, providing solutions and some practices, and the specific functions need to be continuously improved and improved. You can check Vivo’s Karmada-Operator repository specifically, welcome to try and make suggestions. The current code provides a capability matrix that allows you to view the project plan.


Guess you like it

Performance tuning for high-performance Java computing services

Vivo is based on JaCoCo’s test coverage design and practice

JNI in action in intensive computing scenarios