This article is an introduction to change data capture (CDC) practices, not a deep dive into specific tools.

Let’s say we’re building a simple web application. In most cases, such projects start with a minimal data architecture. For example, a relational database like MySQL or PostgreSQL is sufficient to process and store data that many users can use. They enter queries, update them, close them, correct them, and usually do many things. It can be CRM, ERP, automated banking systems, billing systems, or even POS terminals.

However, the information stored in a database can be of interest to many third-party systems, often analytical systems. Businesses need to know the status of applications or other entities stored in the system – accounts, deposits, manufacturing, HR, etc. Data plays an important role in almost every business operation. As a result, businesses generate regular reports that reflect all the main metrics of interest to the business and are necessary to make further management decisions.

Reporting and analytical calculations are often very resource intensive. Queries can take hours to complete, which often severely affects the performance of the system from which the data is retrieved. Another downside is that sending all this data puts a lot of strain on the network. Finally, business decisions based on that data are delayed due to the frequency of queries. So, if you update the data every night, that means you won’t know what happened yesterday until the next day.

If the system

has a definite period of load reduction (e.g. at night) and this time is sufficient to offload all the necessary data without affecting the main activity of the system – then using a direct query to the RDBMS may be an acceptable option. But what if there is no period of reduced load, or the allocated load window is not enough to completely unload the entire changed data?

Here the CDC process comes to the rescue. As the name implies, change data capture only captures changes in the data, which is one of the ETL patterns for replicated data. It is a mechanism to determine the data that interests us, that is, to track changes in the source database and apply them to the target database or data warehouse. And in the target database or data warehouse, we can carry out all types of analysis, report generation, etc., without even affecting the performance of the source database.

As a result, users can use the original system without degrading performance, and management can get the reports they need to make management decisions at any time.

CDC

,

therefore, the essence of CDC – to provide historical change information for user tables by capturing the fact of data manipulation language (DML) changes (inserts/updates/deletes) and the changed data itself. CDC extracts them in a form that can be replicated in upstream data systems. In jargon, such data is also called “delta”.

You can think of CDC as a mechanism that constantly monitors changes to raw data systems, extracts them, and distributes them to upstream systems. Change Data Capture eliminates the process of bulk data loading by enabling incremental data loading in near real time.

So, how can using CDC solve the problems we mentioned?

Well, you

don’t run very large requests on a regular basis because your load rate is actually peak load behavior is not high, therefore, you have to do the network to make sure that all the data you want is sent in a timely manner, and not all this data ends up because the data is released continuously, and for small data, You can get more networks that can be breached by multiple data breaches to function properly and significantly showcase the results of your work, allowing you to have different lines of business. Send data to your data warehouse for updates, so the data in the warehouse is up-to-date, to provide real-time information for business decisions based on the data.

The Change Data Operations Data Center is your best way to store data.

Extracting increments

in analysis is also the “delta” of warehouses, CRM, MDM hubs, disaster times, and extraction transactions is the architectural task which creates the system, and when there is a period of parallel operations, people who migrate data items from one system often have tasks handed over to another.

In the early days, Increment knew that our entire list of issues was updated now. will become a potential outcome, and you may discard some data.

To ensure any loss of this data, but the engineers also tried to control the rows and came up with about the same result – it works, but it is very resourceful. Not to.

All problems are solved with the simple appearance.

is a special process in the database. The specificity of the sample procedure is that after each event in the database, a simple example in the SQL example in the sample is executed.

So, we need a simple table to track all changes so that a table is created for each created object that will be used for changes.

But there is also a class – although there are any changes to the data in the table now, simple and simple. But some CDC products are still based on simplicity.

CDC’s modern approach

We can recall that databases are actually transactional and have similar database logs (also known as database transaction logs or transaction log). Almost all management systems have transaction log files that record each transaction in all databases we need to do to access the transaction log and select the changes we want to track.

Therefore, in CDC, changes are made using reading changes from transactions and are passed through the administrator’s process of setting up administrators in the corresponding change table.

The changes part of the log is that we set these transactions correctly and track the actual past changes. Apply them to the target.

Modern read transactions are systematically processed in the memory of a stand-alone server, highlighting the need to make remote changes to these changes This type of notification shows that the architecture of the system is not and provides impressive source code that tracks changes in the data as a predictable target.

To make a CDC system that needs to be produced before, in addition to extraction, we consider the problem in one system:

1. The changes that occur must be followed, otherwise the system may appear in different states;

In terms of delivery,

2. delivery guarantees Yes, therefore, CDC must deliver a timely notification message at least once, which could result in the state of the entire system if a subsequent system delivers a change event;

3. Finally, simple message conversion is possible because data formats for different systems must be supported.

All messages of the system

are reported – all message sources make changes to the messages of the change feed/subscription, continuously listen to the system target objects, and then when changing these objects, can be within the scope of the change message. when using them.

The solution offers many benefits and scalability. The subscription method allows the primary source/can send more updates to the target system, and the number of this user can be scaled accordingly to process data when needed.

The second desired benefit is that the two systems are now connected. If the source system changes its database or moves a particular dataset to a different location, the target does not need to use the same changes as the pull system does. As long as the source system continues to send messages in the same format, it will not continue to receive system update messages, showing that the source has changed anything.

Source: https://luminousmen.com/post/change-data-capture

End 


 
 


 
 
 
 public number (zhisheng ) reply to  Face, ClickHouse, ES, Flink, Spring, Java, Kafka, Monitor keywords such as  to view more articles corresponding to keywords. 

 
 
like + Looking, less bugs 👇