For infrastructure, many services have similar architectures, database interactions, and UIs, and this article describes the practice of PayPal automating infrastructure microservices scaffolding code through templates, maximizing the time saved in developing services and ensuring service consistency. The Art of Automating Automation[1]

As part of the PayPal Site Reliability & Cloud Engineering team, we’ve been working to improve efficiency through automation. However, writing software that allows people to intuitively automate the job is not a simple task. In PayPal infrastructure, the automation components of any operation must provide the highest levels of reliability, security, efficiency, and self-service capabilities. In this article we will discuss how to automate in Global Network Services (GNS).

GNS teams have a large number of operations that need to be automated, ranging from ordinary tasks that require a few hours of human input to complex tasks that require many days of effort by the operations team. The biggest challenge for us is how to automate these operations, large and small, without consuming a lot of development cycles, and it’s especially important that we try to justify ROI (return on investment).

To turn any network function into a service, you need to complete the following set of tasks:

User interface development

Database interaction and table management

Core business logic

Integration with the automation tool Terraform

Authentication and authorization

Throttling and speed limiting for service accounts, API consumers

As you can see from the list above, building a viable network service requires a lot of time to develop important peripheral components. Ideally, we would be able to spend our time just writing the core business logic and not doing anything else. In two ways, we have reached the ideal state of writing less code.

The first way to solve this problem is to migrate to a real microservices component and build a pluggable library architecture, and the second is template-based automatic code generation, which generates UI, backend, and database layer code based on common templates.

Data modeling has long been the backbone of well-designed software products. Once the basic data building blocks are identified, it is easier to build software components that work around the data. If the data is obtained in a standard format, there is a lot of room for automation for the various software components surrounding the data.

That’s exactly the approach we’re taking. All of our team’s projects start with identifying the data model. The data model is defined in YAML format so that our dynamic code generation module can generate code for various parts of the system. The following diagram shows the process:

The React framework has been widely adopted by the industry, including PayPal. PayPal application team has a very modular React library for building customer-facing applications. We base this library and use it wherever it suits.

Most system programmers are not familiar with UI programming. To facilitate rapid UI component development, we rely on model-driven design. With model-driven design, a typical GNS developer only needs to specify the UI layout without writing any UI code.

The GNS code generation engine takes a user-defined model and uses the React framework to generate a separate React code container with a fully functional UI. This approach allows developers to focus on the business logic of the application.

In the past, developing microservices that provided REST API functionality was cumbersome, and developers needed to write software for each endpoint and configure a web services gateway. However, the Flask framework has been widely recognized by Python developers as a standard framework. Similarly, Golang has a gorilla/mux framework in addition to built-in support.

Most GNS services are written in Golang or Python. With the help of the above framework, we designed the GNS code generation engine based on the Jinja template to generate common code.

Based on the data model defined in YAML format, the code generation engine uses predefined Jinja templates to generate the underlying code for the service. GNS developers can implement precise business logic on the basis of this code, reducing programming errors and eliminating the tedious work of writing everything from scratch.

Most GNS services require some form of database to store the information described in the modeling phase. Writing code to add, remove, and modify entries in the database is tedious, and prone to bugs, stylistic inconsistencies, and so on. Python and Golang have open source libraries that handle object relational mapping (ORM, Object Relationship Mapper), and we chose sqlalchemy to normalize database operations.

Since the model is defined in the YAML format, our code generation engine generates database processing methods in addition to the back-end code.

Terraform has become the most common tool for automation infrastructure. As a GNS service provider, we have decided that all automation services will have interfaces to Terraform, allowing them to adapt to the continuous deployment pipelines that are being adopted.

The Terraform architecture is flexible and easily scalable, enabling us to add providers to any automation we like. See the terrform documentation for more details.

Since Terraform Provider is a well-constructed library, our code generation engine can build golang-based Provider programs based on Jinja templates and then build that code into a pluggable module that can be installed into a running terraform instance. Once this is done, users can perform standard terraform operations on the GNS service, just like any other service.

Most of the tasks required to automate any network service are already generated by our code generation engine, allowing us to focus only on the core logic of the service. The shift to a microservices architecture and a template-based code generation and model-driven architecture allow us to produce services at a faster rate. At the time of writing, we can complete the working framework for any service in a day, helping the team to focus on the core logic and generate more services in less time.

References: [1] The Art of Automating Automation:

Hello, I am Yu Fan, did research and development in Motorola, and now does technical work in Mavenir, and has always maintained a strong interest in communications, networks, back-end architectures, cloud native, DevOps, CICD, blockchain, AI and other technologies, usually like to read, think, believe in continuous learning, lifelong growth, welcome to exchange and learn together. WeChat public account: DeepNoMind