Our company’s cluster is always on the verge of collapse, through nearly three months of mastery, it was found that the reasons for the instability of our company’s cluster are as follows:

1. The

release process is unstable

2. The lack of monitoring platform [the most important reason]

3. Lack of log system

4. Extreme lack of

relevant operation documents

5, the

request route is unclear

In general, the main reason for the problem is the lack of a predictable monitoring platform, and problems are always known. The secondary causes are the unclear role of the server and the instability of the release process.

Refactoring the release process. The business is fully K8S, and a CI/CD process with Kubernetes as the core is built.

Build a reliable federated monitoring platform that conforms to our cluster environment to realize simultaneous monitoring and pre-fault alarm for several cluster environments, and intervene in advance.

1) Monitoring and early warning logic chart

The monitoring and early warning logic diagram is as follows:

In general, the monitoring scheme I use here is prometheus+shell script or go script + sentry. The alarm method used is WeChat or enterprise email. The three colored lines in the figure above represent three monitoring methods to pay attention to. Scripts are mainly used for backup alarms, certificate alarms, and thieves. Prometheus here uses a list of Prometheus resources modified according to Prometheus-opertor, and the data is stored on the NAS. Sentry strictly speaking, it belongs to the log collection platform, here I classify it as the monitoring class, because I fancy its ability to collect the crash information of the underlying code of the application, belongs to the business logic monitoring, aims to collect and summarize the error logs generated during the operation of the business system and monitor the alarm.

Note that a federal monitoring platform is used here, and a normal monitoring platform is deployed.

2) Logic diagram of the federal

monitoring and early

warning platform multi-cluster federal monitoring and early warning platform is as follows:

Because our company has several K8S clusters, if a set of monitoring and early warning platforms are deployed on each cluster, it is too inconvenient to manage, so the strategy I adopt here is to use the strategy of implementing a federation of each monitoring and early warning platform and using a unified visual interface management. Here I will implement three levels of hunger monitoring: operating system level, application level, and business level. Traffic monitoring can be done directly for Kong, template 7424.

demand for log systems will become more desirable, and the characteristic of K8S is that service fault logs are difficult to obtain. Establishing an observable and filterable log system can make it easier to analyze failures.

The logic diagram for the logging system is as follows:

class=”rich_pages wxw-img js_insertlocalimg” src=”https://mmbiz.qpic.cn/mmbiz_png/ufWcjcomw8bX4WAVb7V7yOz099AicmhibqWCvvKFlI6u9QlS8Kdzpxrqtm2LqBJOk0AibbvAmLEicue9xjBLbCNTcQ/640?wx_fmt=png”>

Analysis: After the K8S business is comprehensively implemented, it is convenient to manage and maintain, but the difficulty of log management is appropriately increased. We know that pod restarts are multi-factorial and uncontrollable, and each pod reboot will be re-logged, that is, the logs before the new pod are not visible. Of course, there are multiple ways to store logs for a long time: remote storage logs, local mount logs, etc. For visualization, analysis, and more, you chose to build a log collection system using Elasticsearch.

Establish a document center centered on the relevant materials of Yuque-> O&M, and record relevant operations, problems, scripts, etc. in detail for viewing at any time.

For security reasons, it is not convenient for too many colleagues to consult. The work of operation and maintenance is relatively special, and security and documentation must be guaranteed. I think whether it is operation and maintenance or operation and maintenance development, writing documents must be mastered, for yourself or for him. The documentation can be abbreviated, but it must include the steps of the bud core. I still think every step of the operation and maintenance should be documented.

new idea of cluster reconstruction, the cluster-level traffic request route is reorganized to build a traffic management that integrates authentication, authentication, proxy, connection, protection, control, and observation, and effectively controls the scope of fault explosion.

The request route logic diagram is as follows:

class=”rich_pages wxw-img js_insertlocalimg” src=”https://mmbiz.qpic.cn/mmbiz_png/ufWcjcomw8bX4WAVb7V7yOz099Aicmhibq9K904SjCHvWWtVia3r0Ku8JuSoYoibVBB9XKH6Ljlib52u9loicA4JqicWw/640?wx_fmt=png”>

Analysis: Customer access https://www.cnblogs.com/zisefeizhu After the Kong gateway authentication, enter a specific namespace (distinguish items by namespace), because the service has been split into microservices, the communication between services has been authenticated and authorized by istio, and those who need to interact with the database to find the database, those who need to write or read the storage to find PV, and those who need to convert the service to find the conversion service… Then return a response.

To sum up, the construction of CI/CD distribution process with Kubernetes as the core, the federal monitoring and early warning platform with Prometheus as the core, the log collection system with Elasticsearch as the core, the document management center with Yuque as the core, and the north-south east-west traffic integration service with Kong and Istio as the core can be found in Gaopingfa. High reliability is well guaranteed.

Attached: Overall architecture logic diagram

class=”rich_pages wxw-img js_insertlocalimg” src=”https://mmbiz.qpic.cn/mmbiz_png/ufWcjcomw8bX4WAVb7V7yOz099AicmhibqSLDJBeL5T7yiaM8v2xk0iarhhcjkmKqOCC4tkA7Zp9lwzY78dSVvHkyw/640?wx_fmt=png”>

Analysis: The above figure seems to be too chaotic, calm down, according to the above split module layer by layer analysis can still be seen clearly. Here I use different colored lines to represent the system of different modules, and it is quite clear to walk according to the arrows.

According to our current business traffic, the above functional modules can theoretically achieve the stability of the cluster. I personally believe that this solution can ensure the stable operation of the business on the k8s cluster for a period of time, and any more problems belong to the code level. Middleware is not used here, but the cache redis is used but not drawn. I plan to add middleware kafka or rq to the logging system and conversion service after the above figure is completed.