CN118467637B

CN118467637B - Database synchronization system, method, device, electronic device and medium

Info

Publication number: CN118467637B
Application number: CN202410662455.3A
Authority: CN
Inventors: 吴程程; 刘宝菊; 张烨; 王泽宇
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2024-05-27
Filing date: 2024-05-27
Publication date: 2025-01-24
Anticipated expiration: 2044-05-27
Also published as: CN118467637A

Abstract

The present disclosure provides a database synchronization system, method, device, electronic device and medium, which relate to the field of computer technology, and in particular to the field of big data technology. The specific implementation scheme is: a producer microservice is used to obtain the operation event executed when the source system updates the data in the source database, and store the operation event in a message queue, wherein the operation event includes a business identifier. A consumer microservice is used to obtain the operation event from the message queue, determine the consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database. It is achieved that the execution order of the operation event is guaranteed on the basis of improving the real-time performance of database synchronization.

Description

Database synchronization system, method, device, electronic equipment and medium

Technical Field

The present disclosure relates to the field of computer technology, and in particular, to the field of big data technology.

Background

With the development of big data technology, there is a need for data synchronization between databases in many scenarios. For example, when a user opens an album cloud storage service, whenever a user terminal performs operations such as adding, deleting or modifying an image in a local album, the cloud storage system needs to synchronously modify the album of the user terminal stored in the cloud database. For another example, in order to eliminate the data islanding phenomenon, the utilization rate of the data is improved, different data producers can share the data produced by themselves, and at this time, each time the data in the database of one data producer is updated, other data producers can synchronously update the data stored in their own databases.

Disclosure of Invention

The disclosure provides a database synchronization system, a database synchronization method, a database synchronization device, an electronic device and a database synchronization medium.

In a first aspect of an embodiment of the present disclosure, a database synchronization system is provided, comprising a producer microservice, a message queue, and a consumer microservice;

The producer microservice is used for acquiring operation events executed when the source system updates data in the source database, and storing the operation events into the message queue, wherein the operation events comprise service identifiers;

The consumer micro service is used for acquiring the operation event from the message queue, determining a consumption thread corresponding to the service identifier of the operation event, and executing the operation event on a target database by utilizing the consumption thread.

In a second aspect of the embodiments of the present disclosure, a database synchronization method is provided, applied to a producer microservice, including:

Acquiring an operation event executed when a source system updates data in a source database, wherein the operation event comprises a service identifier;

and storing the operation event into the message queue so that the consumer micro service obtains the operation event from the message queue, determines a consumption thread corresponding to the service identification of the operation event, and executes the operation event on a target database by using the consumption thread.

In a third aspect of embodiments of the present disclosure, a database synchronization method is provided, applied to a consumer micro service, including:

Acquiring an operation event stored by a producer micro-service from the message queue, wherein the operation event is an event executed when a source system updates data in a source database, and the operation event comprises a service identifier;

And determining a consumption thread corresponding to the service identification of the operation event, and executing the operation event on a target database by using the consumption thread. A database synchronization method, applied to a message queue, comprising:

Receiving an operation event stored by a producer micro-service, wherein the operation event is an event executed when a source system updates data in a source database, and the operation event comprises a service identifier;

And sending the operation event to the consumer micro-service so that the consumer micro-service determines a consumption thread corresponding to the service identification of the operation event, and executing the operation event on a target database by utilizing the consumption thread.

In a fourth aspect of the embodiments of the present disclosure, a database synchronization method is provided, applied to a message queue, including:

In a fifth aspect of embodiments of the present disclosure, there is provided a database synchronization apparatus applied to a producer microservice, including:

the acquisition module is used for acquiring operation events executed when the source system updates the data in the source database, wherein the operation events comprise service identifiers;

And the storage module is used for storing the operation event into the message queue so that the consumer micro service obtains the operation event from the message queue, determines a consumption thread corresponding to the service identification of the operation event, and executes the operation event on a target database by utilizing the consumption thread.

A sixth aspect of embodiments of the present disclosure provides a database synchronization apparatus for use with a consumer micro service, comprising:

The acquisition module is used for acquiring operation events stored in the micro-services of the producer from the message queue, wherein the operation events are events executed when the source system updates data in the source database, and the operation events comprise service identifiers;

And the consumption module is used for determining a consumption thread corresponding to the service identifier of the operation event and executing the operation event on a target database by utilizing the consumption thread.

A seventh aspect of the embodiments of the present disclosure provides a database synchronization apparatus, applied to a message queue, including:

The receiving module is used for receiving operation events stored in the micro-services of the producer, wherein the operation events are events executed when the source system updates data in the source database, and the operation events comprise service identifiers;

And the sending module is used for sending the operation event to the consumer micro-service so that the consumer micro-service determines a consumption thread corresponding to the service identifier of the operation event and executes the operation event on a target database by utilizing the consumption thread.

An eighth aspect of the embodiments of the present disclosure provides an electronic device, including:

at least one processor, and

A memory communicatively coupled to the at least one processor, wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the second to fourth aspects.

A ninth aspect of the disclosed embodiments provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method according to any one of the second to fourth aspects.

A tenth aspect of the disclosed embodiments provides a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the second to fourth aspects.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram of a database synchronization system according to an embodiment of the present disclosure;

FIG. 2 is an exemplary schematic diagram of a first database synchronization process provided by an embodiment of the present disclosure;

FIG. 3 is an exemplary schematic diagram of a second database synchronization process provided by an embodiment of the present disclosure;

FIG. 4 is an exemplary schematic diagram of a third database synchronization process provided by an embodiment of the present disclosure;

FIG. 5 is a flow chart of a first database synchronization method provided by an embodiment of the present disclosure;

FIG. 6 is a flow chart of a second database synchronization method provided by an embodiment of the present disclosure;

FIG. 7 is a flow chart of a third database synchronization method provided by an embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of a first database synchronization apparatus according to an embodiment of the present disclosure;

FIG. 9 is a schematic diagram of a second database synchronization apparatus according to an embodiment of the disclosure;

FIG. 10 is a schematic diagram of a third database synchronization apparatus according to an embodiment of the present disclosure;

fig. 11 is a block diagram of an electronic device for implementing a database synchronization method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

At present, when data synchronization is performed on databases of two systems, a log synchronization or real-time interface call synchronization mode is generally adopted. For convenience of description, the following description will be given by taking a case where there is a change in data in the database 1 of the system 1, and the data in the database 2 of the system 2 needs to be updated synchronously.

The log synchronization mode is that the system 2 acquires an operation log from the system 1, wherein various operations performed by the system 1 on the database 1 within a period of time are recorded in the operation log. And then the system 2 analyzes the operation log to obtain operation events which are included in the operation log and used for updating the data in the database, and then each operation event is executed on the database 2. However, since some system operation events of the database are not recorded in the operation log in real time, that is, the update of the database is not immediately reflected in the operation log, and a great amount of computing resources and transmission resources are required for acquiring and analyzing the operation log, the log synchronization mode is generally executed at a longer time interval, so that the real-time performance of the mode is lacking.

The real-time interface call synchronization mode is that each time the system 1 executes an operation event on the database 1 to update the data in the database 1, the corresponding interface of the system 2 is called, so that the system 2 executes the same operation event on the database 2. However, in a high concurrency scenario, in order to improve the data synchronization efficiency, the system 2 uses multiple threads to execute operation events on the database 2 in parallel, where it is difficult to ensure that the sequence of each operation event executed by the system 2 on the database 2 is the same as the sequence of each operation event executed by the system 1 on the database 1. However, the order of the operational events performed on the databases is important, directly affecting the consistency of the data stored in the two databases. For example, operation event 1 is adding data a in the database, and operation event 2 is modifying data a to data B in the database. If the operation event 1 is executed first and then the operation event 2 is executed, the data stored in the database after the execution is completed is data B. If the operation event 2 is executed first and then the operation event 1 is executed, the operation event 2 may not be executed, so that the data stored in the database after the execution of the operation event 1 is completed is data a.

In order to improve the real-time performance of a database synchronization process and ensure the execution sequence of operation events, the embodiment of the disclosure provides a database synchronization system.

Before introducing the database synchronization system provided by the embodiments of the present disclosure, each noun related to the embodiments of the present disclosure is explained first:

Service bus a message queue product for transmitting data as messages between applications and services. Wherein the message contains data and metadata, which is data describing the properties of the data, e.g. metadata comprises the size and type of the data, etc. The data in the message can be various types of data, including structured data encoded in a common format, for example, the types of data in the message are a JS object numbered musical notation (JavaScript Object Notation, JSON), extensible markup language (Extensible Markup Language, XML), apaz serialization data (Apache Avro), plain text, and the like.

The session function of the message can process a series of related messages in an exclusive and orderly manner, namely, the series of related messages are used as the messages in the same session, the messages in the same session are ensured to be executed serially according to the sequence, and the first-in first-out principle in the message processing process is ensured. For application in high availability systems of scalable capacity, session functionality also supports storing session states.

Session identification (session ID) of a message for marking the session to which the message belongs, a plurality of messages may have the same session ID, i.e. the same session may comprise a plurality of messages. The session ID is often used in a web browsing scenario, in which, before a web page is opened by a user terminal and closed by the user terminal, each operation performed on the web page corresponds to a message sent by the user terminal to a server, and the messages include the same session ID, where the session ID is used to indicate that the message originates from the same access procedure to the web page. The embodiments of the present disclosure use session IDs to distinguish between different services to which an operation event to a database belongs, and the specific manner may be described below.

Dead letter queue-a queue for storing messages that temporarily cannot be handled normally. Messages in the message queue that are temporarily not normally processed may be transferred to the dead message queue. When the messages in the dead letter queue can be processed normally, the messages can be transferred to the message queue.

Duplicate detection-the message queue can check if duplicate messages exist in its own stored messages and delete the duplicate messages. The message sender may have an error after storing the message in the message queue, in which case the message sender may incorrectly consider that the last message was not successfully stored in the message queue after restarting, and thus store the message in the message queue again, such that there are duplicate messages in the message queue. Or the message sender obtains two identical messages due to the network and the like, and sends the two identical messages to the message queue based on the two identical messages. Or storing duplicate messages in a message queue, may also be caused by other reasons, as embodiments of the present disclosure are not specifically limited in this regard.

Case identification (case ID) for indicating the service to which the operation event belongs.

Event identification (event ID), the unique identification of the operation event, i.e. the event IDs of different operation events are different. The case ID and the event ID may be in a one-to-many relationship, that is, the case IDs are the same although the event IDs are different from each other for different operation events generated by the same service.

The source system is used for managing the source database, and can perform operations of adding data, deleting data, modifying data, inquiring data and the like on the source database. In the embodiment of the disclosure, a database for which an operation event is originally aimed is referred to as a source database, and a database that needs to be consistent with data in the source database is referred to as a target database.

The producer of the event, i.e., the source system, is able to generate operational events to the source database, with a sequence of generation between the operational events.

The consumer of the event, i.e., the database synchronization system, is able to obtain the operational event from the source system and execute the operational event on the target database. When the consumer of the event executes the operation event on the target database, the order of the operation event is required to be produced according to the producer of the event.

As shown in fig. 1, the database synchronization system provided by the embodiment of the present disclosure includes a producer micro service 101, a message queue 102, and a consumer micro service 103;

The producer microservice 101 is configured to obtain operation events that are executed when the source system updates data in the source database, and store the operation events in the message queue 102. Wherein the operational event includes a service identification.

Consumer micro service 103 is configured to obtain an operation event from message queue 102, determine a consuming thread corresponding to a service identifier of the operation event, and execute the operation event on the target database by using the consuming thread. Wherein consumer microservice 103 has multiple consuming threads, each of which can execute an operational event.

In embodiments of the present disclosure, the operational events performed when the source system updates the data in the source database may include at least one of an add event for adding data in the database, a delete event for deleting data in the database, and a modify event for modifying data in the database.

The source system is a software system and can be operated on the first electronic equipment. The database synchronization system is a software system and can be operated on a second electronic device, and the second electronic device where each of the producer micro-service, the message queue and the consumer micro-service is located can be physically the same electronic device, or the second electronic device where each of the producer micro-service, the message queue and the consumer micro-service is located can be physically different electronic devices. The first electronic device may be an electronic device with database operation capability such as a server, a desktop computer or a notebook computer, and the second electronic device may be an electronic device with database operation capability such as a server, a desktop computer or a notebook computer.

The database synchronization system of the embodiment of the disclosure can execute the same operation event on the target database according to the operation event executed by the source database, thereby realizing the data synchronization between the source database and the target database. In addition, the data synchronization process of the embodiment of the disclosure does not depend on the log of the source system, so that the problem of synchronization delay caused by a log synchronization method is avoided, and the real-time performance of database synchronization is improved. In addition, the consumer micro-service in the embodiment of the disclosure can execute the operation event by using the consumption thread corresponding to the service identifier of the operation event, so that the operation event of the same service can be executed by the same consumption thread in series, thereby guaranteeing the execution sequence of the operation event, namely guaranteeing the data consistency in the source database and the target database.

The following specifically describes a process of implementing data synchronization by the database synchronization system in the embodiment of the present disclosure:

in some embodiments of the present disclosure, the manner in which the producer microservice retrieves operational events includes the following two.

In a first implementation of acquiring an operation event, the producer microservice is specifically configured to perform the following steps:

step one, periodically calling an event list application programming interface (Application Programming Interface, API) of the source system to obtain event identifications (event ids) of operation events executed when the source system updates data in a source database in the current period.

The event list API is a software interface provided by the source system for acquiring event identification of operation events executed by the source system. The event list API may be preconfigured in the producer microservice.

The period of the producer microservice call event list may be preset according to the frequency of the source system executing the operation event, the event processing capability of the source system, and/or the event processing capability of the database synchronization system. For example, the period is 1 minute, i.e., the producer microservice calls the event list API every 1 minute interval, resulting in an event identification for each operational event that the source system executed in the previous minute.

Optionally, after calling the event list API, the producer microservice may obtain, in addition to the event identifier of the operation event, other information such as the service identifier of the operation event, which is not limited in particular by the embodiment of the present application.

And step two, sending an operation event acquisition request to the source system. Wherein the operational event acquisition request includes each event identification.

Step three, receiving operation events corresponding to the event identifications sent by the source system.

The operational events may include, among other things, a service identification, an event type (EVENT TYPE), and an event identification. The operational events may also include other parameters, as embodiments of the present disclosure are not specifically limited in this regard. The event types comprise adding data, deleting data, modifying data and the like.

For example, the following is a data structure of an operation event in XML format:

1<item>

2<event type>123</event type>

3<case id>321</case id>

4<occurred at>2024-01-01 01:01:01</occurred at>

5<additional>aaa</additional>

6<event id>456789</event id>

7</item>

The data between "< item >" and "</item >" in the data structure represents an operation event, "123" between "< event_type >" and "</event_type >" represents an event type of the operation event, "321" between "< case id >" and "</case id >" represents a service identification of the operation event, "2024-01-01:01:01" between "< occurred at >" and "</occurred at >" represents an execution time of the operation event, "aaa" between "< additional >" and "</additional >" represents an additional parameter of the operation event, which may be set according to service requirements, for example, the additional parameter is an approval artifact number of an approval service, "456789" between "< event >" and "</event id >" represents an event identification of the operation event.

In an actual application scenario, the parameters included in each operation event are not limited to the above examples, and the parameters included in each operation event need not cover all the parameters included in the above examples.

In the embodiment of the disclosure, the source system and the database synchronization system may operate independently, that is, the source system does not need to consider the database synchronization system when executing the operation event, for example, the source system does not need to control the database synchronization system in real time to execute the operation event, and does not need to execute the next operation event after the database synchronization system is successfully executed. Moreover, the database synchronization system can actively acquire and execute operation events based on the event list API without being controlled by the source system. Therefore, in the embodiment of the disclosure, the source system can realize decoupling between the source system and the database synchronization system by only providing the event list API, so that the two systems can operate independently, the influence of errors in one system on other systems is reduced, and the stability and reliability of the respective operation of the source system and the database synchronization system are improved.

That is, as shown in FIG. 2, the producer of the event is the source system and the consumer of the event is the database synchronization system. The producer micro-service of the database synchronization system periodically calls an event list API of the source system to obtain event identifications of all operation events in an operation event list executed when the source system updates data in the source database in the current period. And then the producer microservice obtains an operation event list formed by each operation event according to the event identification, and stores each operation event in the operation event list into a message queue. The consumer micro service obtains each operation event from the message queue in sequence, determines a consumption thread corresponding to the service identification of the operation event, and executes the operation event to the target database by using the consumption thread.

Because the producer micro-service can acquire the operation events executed by the endogenous system on the source database for a period of time after calling the event list API, that is, acquire a plurality of operation events, in order to ensure the execution sequence of the plurality of operation events, the producer micro-service can store the plurality of operation events into the message queue according to the sequence.

That is, the producer microservice is specifically configured to store received operation events in a message queue in order of execution time from first to last.

Wherein the operational event includes an execution time of the operational event by the source system. For example, the execution time represents the start execution time of the operation event by the source system, the execution completion time of the operation event by the source system, or the like.

The producer micro-service can store each operation event into the message queue according to the execution time sequence of each received operation event, and the message queue has the characteristic of first-in first-out, so that each operation event can be executed by the consumer micro-service according to the sequence to update the data in the target database.

In a second implementation of acquiring an operation event, the producer microservice is specifically configured to perform the following steps:

Step 1, receiving an event identifier of an operation event sent by a source system.

Wherein the event identification is sent after the source system performs the operational event to update the data in the source database. That is, each time the source system performs an operational event for updating data in the source database, it sends an event identification of the operational event to the producer microservice.

And step 2, sending an operation event acquisition request to the source system. Wherein the operational event acquisition request includes an event identification.

And step 3, receiving an operation event corresponding to the event identifier sent by the source system.

By the method, the source system can enable the producer microservice to acquire the operation event after executing the operation event on the source database each time, so that the operation event is synchronously executed on the target database, and the real-time performance of data synchronization between the source database and the target database is improved.

In order that the operational events of the same service may be executed serially in sequence, the producer microservice may store the operational events of the same service in the same session of the message queue, i.e. the producer microservice is specifically configured to perform the following steps:

and step one, carrying out hash operation on the service identifier included in the operation event by using a preset hash algorithm to obtain a hash value.

And carrying out hash operation on the service identifier by using a preset hash algorithm, wherein the obtained hash value is an integer in a preset range, and the number of integers included in the preset range is the same as the number of sessions in the message queue. For example, the preset hash algorithm is a consistent hash algorithm. The number of sessions in the message queue may be preset according to the actual service requirements. For example, because the operation events in different sessions of the message queue can be processed by the consumer micro-service concurrency, i.e. the session number can be regarded as the number of operation events which can be processed concurrency, the session number in the message queue can be set according to the requirement of the service on the real-time synchronization of the data.

In one implementation, each business may be provided with a unique business identification (case id), e.g., a business identification "01" for an operational event generated by a financial business, a business identification "02" for an operational event generated by an educational business, etc. In this case, the producer micro-service may directly perform hash operation on the service identifier included in the operation event to obtain a hash value.

In another implementation, there is a case where the service identifier of the operation event is composed of a plurality of identifiers. For example, the operational event generated by the approval service of the financial transaction includes a financial transaction identifier and an approval service identifier, and the financial transaction identifier and the approval service identifier together form a service identifier of the operational event. In this case, the producer microservice is further configured to obtain, from the identifiers included in the operation event, other identifiers except the event identifier, and if the number of other identifiers is plural, splice the other identifiers as service identifiers before executing the step (one).

For each operation event, the producer microservice can screen data with the attribute including 'id' from various data included in the operation event, then exclude the data with the attribute of 'event id', splice the rest data, and serve as a service identifier of the operation event. For example, the operation event includes { "SERVICE ID": "11", "case id": "1", "event id": "01", "execution time": "00:00:01" }, the operation event includes 4 items of data, each item of data is "separated" in which the parameter preceding ": represents an attribute of the data, and the parameter following": represents one item of data of the operation event. In determining the service identifier, the producer micro-service may obtain data including "id" in the attribute, that is, obtain 11, 1, and 01, then exclude 01 having the attribute "event id", and then splice 11 and 1 to obtain 111 as the service identifier of the operation event.

By the method, the embodiment of the disclosure can support the service identification of the operation event represented by only one item of data and also support the service identification of the operation event represented by the combination of multiple items of data, so that the application range is wider, and the database synchronization can be realized under various application scenes.

And step two, taking the hash value as a session identifier (session id) of the operation event. The session identity may also be referred to as group identity (group id).

And step three, storing the operation event into the session corresponding to the session identifier of the message queue.

The producer microservice may set a session identifier for each operation event, such that after the operation event is stored in the buffer of the message queue, the message queue may route the operation event to a session corresponding to the session identifier according to the session identifier of the operation event.

Because the conversation mechanism in the message queue can ensure that a series of operation events can be executed exclusively and orderly, in the embodiment of the disclosure, the producer micro-service can set the conversation mark of the operation event according to the service of the operation event, so that the message queue is indicated to store the operation event of the same service into the same conversation according to the conversation mark, and the operation event of the same service can be executed orderly.

In the embodiment of the disclosure, different service identifiers are subjected to hash operation to obtain the same hash value. That is, there is a case where the same session includes operation events of a plurality of services. For example, 4 sessions are created in the message queue, namely session1, session2, session3 and session4, and session1 is identified as 1, session2 is identified as 2, session3 is identified as 3, and session4 is identified as 4. The preset hash operation is assumed to be that a remainder of dividing the service identifier by 4 is calculated, 4 is used as a hash value when the remainder is 0, and the remainder is used as the hash value when the remainder is not 0. At this time, when the service identifier of the operation event is 1, the hash value is 1, that is, the session identifier is 1, and session1 needs to be added. When the service identifier of the operation event is 2, the hash value is 2, that is, the session identifier is 2, and session2 needs to be added. By the pushing, when the service identifier of the operation event is 5, the hash value is 1, that is, the session identifier is 1, and session1 needs to be added. It can be seen that the session1 may include an operation event with a service identifier of 1 and an operation event with a service identifier of 5.

Although the same session may include operation events of a plurality of services, all of the operation events of the plurality of services are executed in series, and thus the execution order of the operation events of a single service can be ensured in this case as well.

In the disclosed embodiments, the consumer microservice is also able to periodically poll itself for disconnection from subscribed sessions. For example, the consumer micro-service may check whether session exception information sent by the message queue is received, where the session exception information is used to indicate that the consumer micro-service is disconnected from the subscribed session, and if so, request again to create a connection to the message queue for the session. And if the message queue detects that the session is subscribed by the other consumer micro-services, sending a connection rejection message to the consumer micro-services.

In some embodiments of the present disclosure, there are multiple consumer micro services in the database synchronization system, each of which handles the same process of operating events. Each consumer microservice may also be referred to as a consumer instance of an operational event.

For example, as shown in fig. 3, the database synchronization system includes N consumer micro services, where the operation event provided by the source system may be executed by the N consumer micro services, and each consumer micro service may determine the business process performed on the target database by the event type included by the operation event. For example, event 1 has an event type of 1, which indicates that business process 1 needs to be performed, i.e., data in the database is deleted, event 2 has an event type of 2, which indicates that business process 2 needs to be performed, i.e., data in the database is added, and event 3 has an event type of 3, which indicates that business process 3 needs to be performed, i.e., data in the database is modified.

The consumer micro service is specifically configured to, when monitoring that an operation event exists in a session subscribed in the message queue, obtain the operation event, determine a consumption thread for executing the subscribed session, and execute the operation event on the target database by using the consumption thread. Wherein each customer micro-service subscription session is different.

Each consumer micro-service in the embodiments of the present disclosure may configure a preset number of consuming threads, where the preset number is a number of sessions included in a message queue, and each consuming thread corresponds to one session in the message queue. Because each consumer micro-service may subscribe to at least one session, and there may be variations in the session to which the consumer micro-service subscribes, the consumer micro-service may create multiple consuming threads and set one session in each consuming thread's corresponding message queue.

Each consumer micro-service may receive a manually assigned session identification of at least one session and create a session connection with the message queue to subscribe to the session based on the session identification. After the session connection is established successfully, the consumer micro-service can monitor and acquire the operation event from the session.

Or the message queue can allocate at least one session in the message queue for each consumer micro-service according to the load balancing principle, and establish session connection with the consumer micro-service according to the session identification of the session allocated for the consumer micro-service, so that the consumer micro-service can monitor and acquire operation events from the session.

Or may otherwise allocate sessions for each consumer micro service subscription, as embodiments of the present disclosure are not specifically limited.

In the embodiment of the disclosure, because the session of each consumer micro service subscription is different, and only one operation event is executed by a single consumption thread at a time, the operation event of the same service can be executed by the same thread of the same consumer micro service, thereby guaranteeing the execution sequence of the operation event, namely guaranteeing the sequence of the operation event executed by a source system on a source database, keeping the sequence of the operation event executed by a database synchronization system on a target database consistent, and improving the accuracy of data synchronization between databases.

Moreover, because the session subscribed by each consumer micro-service is different, and each consumer micro-service can process the operation events concurrently, that is, the embodiment of the disclosure can process the operation events of different services in parallel, and the execution sequence of the operation events of different services does not affect the accuracy of the data stored in the target database. Compared with a mode of executing each operation event in the message queue in series, the embodiment of the invention can process the operation events of different services concurrently, improves the processing efficiency of the operation events, ensures that the database synchronization system has high availability and can be applied to high concurrency scenes.

In the embodiment of the disclosure, in order to ensure idempotency of data in the target database, the consumer micro-service may further obtain a service identifier of an operation event before processing the operation event, and check whether the target database stores data corresponding to the service identifier. If not, the service identifier is created in the target database, and the operation event is executed on the data corresponding to the service identifier.

In the disclosed embodiments, the message queue may be a service bus.

Under the condition that the message queue is a service bus, the message queue can also repeatedly detect the operation event stored by the message queue, namely, check whether the repeated operation event is stored or not, delete the repeated operation event, and prevent the situation that the data stored in the target database is inconsistent with the source database due to repeated consumption of the operation event.

According to the embodiment of the disclosure, the service bus is used as the message queue, and because the service bus is a first-in first-out queue, the first-out of the operation events stored in the message queue can be guaranteed, namely, the sequence of the operation events stored in the message queue is guaranteed, and the sequence of the operation events which are removed from the message queue and processed is the same. And the service bus can also ensure that all stored operation events are processed, and ensure the consistency of data stored in the target database and the source database.

In order to guarantee the execution sequence of the operation events, the message queue is used for executing the following steps that when the target consumer micro-service is detected to acquire the operation events from the target session subscribed by the target consumer micro-service for the first time, the target session is set to be in a locking state, the session can also be called locking, and the lock is occupied by the consumer micro-service subscribed to the session.

The target consumer micro-service is any consumer micro-service in the database synchronization system, and the locking state is used for indicating that the operation event in the target session is only acquired by the consumer micro-service subscribed to the target session.

And the message queue is also used for receiving an execution completion message sent by the target consumer micro-service after the target consumer micro-service acquires any operation event in the target session and deleting the operation event from the target session.

Although the message queue can store operation events on a first-in first-out basis, in a case where there are a plurality of consumer micro services, since each consumer micro service can process operation events in parallel, operation events of the same service may be executed in parallel, and even a case where a processing order is wrong occurs. In the embodiment of the disclosure, the message queue may lock the session, so that the operation event in one session is only processed by one consumer micro-service at the same time, thereby avoiding the situation that the consumer micro-service processes the operation event of the same service concurrently, and avoiding the influence that the execution sequence of the operation event is wrong in the data synchronization process to the accuracy of the data stored in the target database. And the message queue can be deleted from the session after the execution of the operation event is completed, so that the execution omission of the operation event is avoided.

The message queue can also be used for setting the target session to an unlocked state if the execution completion message sent by the target consumer micro-service is not received within a timeout period after the target consumer micro-service acquires one of the operation events in the target session in order to ensure that the operation event is not lost because the consumer micro-service cannot successfully execute the operation event. And selecting one consumer micro-service from other consumer micro-services except the target consumer micro-service to subscribe to the target session so that the selected consumer micro-service acquires the operation event from the target session, and setting the target session to be in a locking state when detecting that the selected consumer micro-service acquires the operation event from the target session for the first time.

The message queue does not receive the execution completion message sent by the target consumer micro-service within the timeout period after the target consumer micro-service acquires the operation event, which indicates that the target consumer micro-service cannot currently execute the operation event or is difficult to ensure the execution efficiency of the operation event, so that the message queue can reselect one consumer micro-service from other consumer micro-services, for example, select one consumer micro-service with the least number of subscribed sessions according to the load balancing principle. And then the message queue is connected with the customer micro-service creation session so that the customer micro-service subscribes to the target session, the customer micro-service obtains the operation event from the target session, and the message queue sets the target session to be in a locking state.

By the method, the message queue can distribute the operation event of the target session to another consumer micro-service for processing when the target consumer micro-service cannot timely process the operation event of the target session, for example, the target consumer micro-service is manually deleted in a volume reduction scene or the target consumer micro-service is down, so that the session is moved among the consumer micro-services, and the execution efficiency of the operation event in the session is ensured.

The embodiment of the disclosure can support capacity expansion of the database synchronization system, namely, increase the number of consumer micro services included in the database synchronization system. At this time, the message queue is further configured to allocate a session subscribed by each customer micro-service according to a preset load balancing policy when detecting that there is a newly added customer micro-service in the database synchronization system.

The load balancing policy may include that each consumer micro-service is allocated at least one session, and that a difference between a maximum value and a minimum value in a number of sessions to which each consumer micro-service is allocated does not exceed a preset threshold. The load balancing policy may also include other policies, as embodiments of the present disclosure are not specifically limited in this regard.

In the embodiment of the disclosure, a customer micro-service may be manually added in the database synchronization system, the message queue may receive manually input related information of the newly added customer micro-service, such as a service identifier, and at this time, the message queue may determine that the newly added customer micro-service exists in the database synchronization system and execute a session allocation process.

By the method, the embodiment of the disclosure can support the real-time capacity expansion of the database synchronization system, thereby meeting the execution requirement of large-scale operation events in a large-scale service scene and flexibly coping with the change of the service requirement.

In some embodiments of the present disclosure, there may be situations where an operational event cannot be handled temporarily, e.g., the target database is in a read-only state, and data in the target database cannot be deleted. For another example, the target database storage space is already fully occupied and no data can be added to the target database.

In order to ensure the processing sequence of the operation events, the message queue is further used for determining a dead message session to which the operation event meeting the dead message condition belongs if any operation event in the message queue meets the preset dead message condition is detected, and then storing each operation event in the dead message session into the dead message queue according to the sequence.

Optionally, the preset dead letter condition comprises that the retry number of the operation event exceeds the preset number and/or the storage time of the operation event in the message queue exceeds the preset storage time. Of course, the dead letter condition may also include other conditions, which are not particularly limited by the embodiments of the present disclosure.

For example, after an operation event of a session is sent to a consumer micro-service by a message queue, if a processing failure message sent by the consumer micro-service is received, 1 is added to the retry number of the operation event, and the operation event is sent to the consumer micro-service again until the retry number exceeds a preset number.

By the method, after any operation event of the message queue meets the dead letter condition, the message queue can transfer all the messages in the session to which the operation event belongs in the message queue to the dead letter queue, instead of transferring the operation event meeting the dead letter condition to the dead letter queue, so that the situation that the execution sequence of each event operation is wrong due to omission of part of operation events in the same session is avoided, and the execution sequence of the operation events in the same session is ensured.

The operation event in the dead letter queue can be transferred to the message queue by the message queue when the operation event in the dead letter queue can be processed after the error cause of the operation event in the dead letter queue is checked manually and the fault is removed manually.

Referring to fig. 4, the following describes a database synchronization process provided by the embodiment of the present disclosure in conjunction with an actual application scenario:

The database synchronization system is provided with a timing task for notifying the producer micro-service to call the event list API of the source system at fixed time, so that the producer micro-service can call the event list API of the source system periodically to obtain event identifications of operation events executed when the source system updates data in the source database in the current period, and the operation events are acquired from the source system based on the event identifications. Thereafter, the operational events obtained from the source system are stored in a message queue. The consumer micro-service may obtain the operation event from the message queue, determine a consumption thread corresponding to the service identifier of the operation event, and execute the operation event on the target database by using the consumption thread. The message queue can also determine a dead message session to which the operation event meeting the dead message condition belongs when any operation event in the message queue meets the preset dead message condition, and store each operation event in the dead message session into the dead message queue in sequence. When the operation event in the dead message queue can be processed, the message queue can transfer the operation event in the dead message queue to the message queue again.

The database synchronization system provided by the embodiment of the disclosure can meet the execution requirement of large-scale operation events in a large-scale service scene, and has the high performances of processing order, high reliability, capacity expansion and the like of the operation events.

In the technical scheme of the disclosure, the related processing such as collection, storage, use, processing, transmission, provision and disclosure of the operation events all conform to the regulations of related laws and regulations and do not violate the public order harmony.

Based on the same inventive concept, the embodiments of the present disclosure further provide a database synchronization method applied to a producer micro-service, as shown in fig. 5, the method including:

s501, acquiring an operation event executed when the source system updates data in the source database, wherein the operation event comprises a service identifier.

S502, storing the operation event into a message queue, so that the consumer micro service obtains the operation event from the message queue, determines a consumption thread corresponding to the service identifier of the operation event, and executes the operation event to the target database by using the consumption thread.

The database synchronization method disclosed by the embodiment of the invention can execute the same operation event on the target database according to the operation event executed by the source database, thereby realizing the data synchronization between the source database and the target database. In addition, the data synchronization process of the embodiment of the disclosure does not depend on the log of the source system, so that the problem of synchronization delay caused by a log synchronization method is avoided, and the real-time performance of database synchronization is improved. In addition, the consumer micro-service in the embodiment of the disclosure can execute the operation event by using the consumption thread corresponding to the service identifier of the operation event, so that the operation event of the same service can be executed by the same consumption thread in series, thereby guaranteeing the execution sequence of the operation event, namely guaranteeing the data consistency in the source database and the target database.

In some embodiments of the present disclosure, obtaining operational events performed when a source system updates data in a source database includes:

Periodically calling an event list Application Programming Interface (API) of the source system to obtain event identifiers of operation events executed when the source system updates data in a source database in the current period;

Sending an operation event acquisition request to a source system, wherein the operation event acquisition request comprises each event identifier;

and receiving operation events corresponding to the event identifications sent by the source system.

In some embodiments of the present disclosure, the operation event includes an execution time of the operation event by the source system, and storing the operation event in the message queue includes:

and storing the received operation events into a message queue according to the order of the execution time from the beginning to the end.

Receiving an event identifier of an operation event sent by a source system, wherein the event identifier is sent after the source system executes the operation event to update data in a source database;

sending an operation event acquisition request to a source system, wherein the operation event acquisition request comprises an event identifier;

and receiving an operation event corresponding to the event identifier sent by the source system.

In some embodiments of the present disclosure, storing the operational event in the message queue includes:

carrying out hash operation on service identifiers included in the operation event by using a preset hash algorithm to obtain a hash value;

Taking the hash value as a session identifier of the operation event;

And storing the operation event into a session corresponding to the session identifier of the message queue.

In some embodiments of the present disclosure, before performing a hash operation on a service identifier included in an operation event by using a preset hash algorithm, the method further includes:

Acquiring other identifiers except the event identifier from the identifiers included in the operation event;

if the number of the other identifiers is a plurality of, splicing the other identifiers to serve as service identifiers.

Based on the same inventive concept, the embodiments of the present disclosure also provide a database synchronization method applied to a consumer micro service, as shown in fig. 6, the method including:

s601, acquiring an operation event stored by a producer micro-service from a message queue, wherein the operation event is an event executed when a source system updates data in a source database, and the operation event comprises a service identifier.

S602, determining a consumption thread corresponding to the service identification of the operation event, and executing the operation event on the target database by using the consumption thread.

In some embodiments of the present disclosure, there are multiple consumer micro services;

Obtaining the operation event of the producer micro-service storage from the message queue comprises the following steps:

Acquiring an operation event when the operation event exists in the session subscribed in the message queue, wherein the session subscribed by each consumer micro-service is different;

Determining a consumption thread corresponding to the service identifier of the operation event comprises the following steps:

A consumption thread for executing the subscribed session is determined.

Based on the same inventive concept, the embodiments of the present disclosure further provide a database synchronization method applied to a message queue, as shown in fig. 7, the method includes:

S701, receiving an operation event stored by a producer micro-service, wherein the operation event is an event executed when a source system updates data in a source database, the operation event comprises a service identifier, so that a consumer micro-service acquires the operation event from a message queue, determines a consumption thread corresponding to the service identifier of the operation event, and executes the operation event to a target database by using the consumption thread.

In some embodiments of the present disclosure, there are a plurality of consumer micro services, the method further comprising:

Setting the target session to be in a locking state when detecting that the target consumer micro-service acquires an operation event from the subscribed target session for the first time, wherein the target consumer micro-service is any consumer micro-service in the database synchronous system, and the locking state is used for indicating that the operation event in the target session is acquired only by the consumer micro-service subscribed to the target session;

The method further comprises the steps of:

After the target consumer micro-service acquires any operation event in the target session, receiving an execution completion message sent by the target consumer micro-service;

The operational event is deleted from the target session.

In some embodiments of the present disclosure, the method further comprises:

If the execution completion message sent by the target consumer micro-service is not received within the timeout period after the target consumer micro-service acquires one operation event in the target session, setting the target session to be in an unlocked state;

Selecting one consumer micro-service from other consumer micro-services than the target consumer micro-service to subscribe to the target session, such that the selected consumer micro-service obtains the operational event from the target session;

when the selected consumer micro-service is detected to acquire the operation event from the target session for the first time, the target session is set to be in a locked state.

In some embodiments of the present disclosure, the method further comprises:

When detecting that the newly added consumer micro-service exists in the database synchronization system, distributing the session subscribed by each consumer micro-service according to a preset load balancing strategy.

In some embodiments of the present disclosure, the method further comprises:

If any operation event in the message queue is detected to meet the preset dead message condition, determining a dead message session to which the operation event meeting the dead message condition belongs;

And storing the operation events in the dead message session into a dead message queue in sequence.

In some embodiments of the present disclosure, the message queue is a service bus.

Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization apparatus applied to a producer micro service, as shown in fig. 8, the apparatus includes an acquisition module 801 and a storage module 802;

an obtaining module 801, configured to obtain an operation event executed when the source system updates data in the source database, where the operation event includes a service identifier;

The storage module 802 is configured to store the operation event in the message queue, so that the consumer micro service obtains the operation event from the message queue, determines a consumption thread corresponding to a service identifier of the operation event, and executes the operation event on the target database by using the consumption thread.

In some embodiments of the present disclosure, the obtaining module 801 is specifically configured to:

In some embodiments of the present disclosure, the operation event includes an execution time of the operation event by the source system, a storage module 802, specifically configured to:

In some embodiments of the present disclosure, the storage module 802 is specifically configured to:

Taking the hash value as a session identifier of the operation event;

In some embodiments of the present disclosure, the apparatus further comprises:

the obtaining module 801 is further configured to obtain, before performing hash operation on a service identifier included in the operation event by using a preset hash algorithm to obtain a hash value, other identifiers except the event identifier from each identifier included in the operation event;

and the splicing module is used for splicing the other identifiers to serve as service identifiers if the number of the other identifiers is multiple.

Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization apparatus applied to a consumer micro service, as shown in fig. 9, the apparatus includes an acquisition module 901 and a consumption module 902;

The acquiring module 901 is configured to acquire an operation event stored in the micro-service of the producer from the message queue, where the operation event is an event executed when the source system updates data in the source database, and the operation event includes a service identifier;

and the consumption module 902 is configured to determine a consumption thread corresponding to the service identifier of the operation event, and execute the operation event on the target database by using the consumption thread.

In some embodiments of the present disclosure, there are multiple consumer micro services, an acquisition module 901, specifically for:

the consumption module 902 is specifically configured to:

A consumption thread for executing the subscribed session is determined.

Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization apparatus applied to a message queue, as shown in fig. 10, the apparatus includes a receiving module 1001;

the receiving module 1001 is configured to receive an operation event stored in the micro-service of the producer, where the operation event is an event executed when the source system updates data in the source database, and the operation event includes a service identifier, so that the micro-service of the consumer obtains the operation event from the message queue, determines a consumption thread corresponding to the service identifier of the operation event, and executes the operation event on the target database by using the consumption thread.

In some embodiments of the present disclosure, there are a plurality of consumer micro services, the apparatus further comprising:

The system comprises a setting module, a target consumer micro-service acquisition module and a target user micro-service acquisition module, wherein the setting module is used for setting a target session to be in a locking state when detecting that the target consumer micro-service acquires an operation event from a target session subscribed by the target consumer micro-service for the first time, wherein the target consumer micro-service is any consumer micro-service in a database synchronization system, and the locking state is used for indicating that the operation event in the target session is acquired only by the consumer micro-service subscribed to the target session;

the apparatus further comprises:

The receiving module 1001 is further configured to receive an execution completion message sent by the target consumer micro-service after the target consumer micro-service obtains any operation event in the target session;

and the deleting module is used for deleting the operation event from the target session.

In some embodiments of the present disclosure, the apparatus further comprises:

the setting module is further configured to set the target session to an unlocked state if the execution completion message sent by the target consumer micro-service is not received within a timeout period after the target consumer micro-service acquires an operation event in the target session;

A selection module for selecting one consumer micro-service from other consumer micro-services except for the target consumer micro-service to subscribe to the target session, so that the selected consumer micro-service obtains the operation event from the target session;

the setting module is further configured to set the target session to a locked state when detecting that the selected consumer micro-service acquires the operation event from the target session for the first time.

In some embodiments of the present disclosure, the apparatus further comprises:

And the distribution module is used for distributing the session subscribed by each consumer micro-service according to a preset load balancing strategy when detecting that the newly added consumer micro-service exists in the database synchronization system.

In some embodiments of the present disclosure, the apparatus further comprises:

The determining module is used for determining a dead letter session to which the operation event meeting the dead letter condition belongs if any operation event in the message queue is detected to meet the preset dead letter condition;

And the transfer module is used for storing all operation events in the dead message session into the dead message queue according to the sequence.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 11 illustrates a schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 11, the electronic device 1100 includes a computing unit 1101 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, ROM 1102, and RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.

Various components in the electronic device 1100 are connected to the I/O interface 1105, including an input unit 1106, e.g., keyboard, mouse, etc., an output unit 1107, e.g., various types of displays, speakers, etc., a storage unit 1108, e.g., magnetic disk, optical disk, etc., and a communication unit 1109, e.g., network card, modem, wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunications networks.

The computing unit 1101 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 1101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as database synchronization methods. For example, in some embodiments, the database synchronization method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1108. In some embodiments, some or all of the computer programs may be loaded and/or installed onto electronic device 1100 via ROM 1102 and/or communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the database synchronization method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the database synchronization method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be a special or general purpose programmable processor, operable to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A database synchronization system, comprising: a producer microservice, a message queue and a consumer microservice;

The producer microservice is used to obtain an operation event performed when the source system updates the data in the source database, and store the operation event in the message queue, wherein the operation event includes a business identifier;

The consumer microservice is used to obtain an operation event from the message queue, determine a consumption thread corresponding to a business identifier of the operation event, and use the consumption thread to execute the operation event on a target database. There are multiple consumer microservices.

The message queue is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

2. The system according to claim 1, wherein the producer microservice is specifically used for:

Periodically calling the event list application programming interface API of the source system to obtain event identifiers of various operation events executed when the source system updates the data in the source database in the current period;

Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes each event identifier;

Receive operation events corresponding to the event identifiers sent by the source system.

3. The system according to claim 2, wherein the operation event includes the execution time of the source system on the operation event; the producer microservice is specifically used to:

The received operation events are stored in the message queue in the order of execution time from earliest to latest.

4. The system according to claim 1, wherein the producer microservice is specifically used for:

receiving an event identifier of an operation event sent by the source system, wherein the event identifier is sent after the source system executes the operation event to update data in the source database;

Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes the event identifier;

An operation event corresponding to the event identifier sent by the source system is received.

5. The system according to claim 1, wherein the producer microservice is specifically used for:

Using a preset hash algorithm, a hash operation is performed on the service identifier included in the operation event to obtain a hash value;

Using the hash value as a session identifier of the operation event;

The operation event is stored in the session corresponding to the session identifier of the message queue.

6. The system according to claim 5, wherein the producer microservice is further used for:

Before using a preset hash algorithm to perform a hash operation on the service identifier included in the operation event to obtain a hash value, obtaining other identifiers except the event identifier from each identifier included in the operation event;

If there are multiple other identifiers, the other identifiers are concatenated to serve as the service identifier.

7. The system according to claim 5 or 6, wherein there are multiple consumer microservices; the consumer microservice is specifically used for:

When monitoring that an operation event exists in the session subscribed by itself in the message queue, the operation event is obtained, wherein each consumer microservice subscribes to a different session;

A consumption thread for executing the subscribed session is determined, and the operation event is executed on the target database using the consumption thread.

8. The system according to claim 5 or 6, wherein the message queue is further used for:

After the target consumer microservice obtains any operation event in the target session, receiving an execution completion message sent by the target consumer microservice;

Delete the operation event from the target session.

9. The system according to claim 8, wherein the message queue is further used for:

If the target consumer microservice does not receive an execution completion message sent by the target consumer microservice within the timeout period after the target consumer microservice obtains an operation event in the target session, setting the target session to an unlocked state;

Selecting a consumer microservice from other consumer microservices except the target consumer microservice to subscribe to the target session, so that the selected consumer microservice obtains operation events from the target session;

When it is detected that the selected consumer microservice obtains an operation event from the target session for the first time, the target session is set to a locked state.

10. The system according to claim 8, wherein the message queue is further used for:

When it is detected that there is a newly added consumer microservice in the database synchronization system, the session subscribed by each consumer microservice is allocated according to a preset load balancing strategy.

11. The system according to claim 5 or 6, wherein the message queue is further used for:

If it is detected that any operation event in the message queue meets a preset dead letter condition, determining the dead letter session to which the operation event meeting the dead letter condition belongs;

Each operation event in the dead letter session is stored in the dead letter queue in sequence.

12. The system according to any one of claims 1 to 11, wherein the message queue is a service bus.

13. A database synchronization method, applied to a producer microservice, comprising:

Acquire an operation event performed when the source system updates data in the source database, wherein the operation event includes a business identifier;

The operation event is stored in a message queue, so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database;

There are multiple consumer microservices; the message queue is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

14. The method according to claim 13, wherein the obtaining of the operation event performed when the source system updates the data in the source database comprises:

15. The method according to claim 14, wherein the operation event includes the execution time of the operation event by the source system; and storing the operation event in the message queue comprises:

16. The method according to claim 13, wherein the obtaining of the operation event performed when the source system updates the data in the source database comprises:

17. The method according to claim 13, wherein storing the operation event into the message queue comprises:

Using the hash value as a session identifier of the operation event;

18. The method according to claim 17, before using a preset hash algorithm to perform a hash operation on the service identifier included in the operation event to obtain a hash value, further comprising:

From the various identifiers included in the operation event, obtain other identifiers except the event identifier;

19. A database synchronization method, applied to a consumer microservice, comprising:

Obtaining an operation event stored by the producer microservice from the message queue, wherein the operation event is an event executed when the source system updates data in the source database, and the operation event includes a business identifier;

Determine a consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database;

20. The method according to claim 19, wherein there are multiple consumer microservices;

The obtaining the operation event stored by the producer microservice from the message queue includes:

The determining of the consumption thread corresponding to the service identifier of the operation event includes:

A consumer thread for the session that is to execute the subscription is determined.

21. A database synchronization method, applied to a message queue, comprising:

Receive an operation event stored by a producer microservice, where the operation event is an event executed when a source system updates data in a source database, and the operation event includes a business identifier, so that a consumer microservice obtains the operation event from the message queue, determines a consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on a target database;

There are multiple consumer microservices, and the method further includes:

When it is detected that the target consumer microservice obtains an operation event from a target session to which it has subscribed for the first time, the target session is set to a locked state; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

22. The method according to claim 21, further comprising:

Delete the operation event from the target session.

23. The method according to claim 22, further comprising:

24. The method of claim 22, further comprising:

25. The method of claim 21, further comprising:

26. The method according to any one of claims 21-25, wherein the message queue is a service bus.

27. A database synchronization device, applied to a producer microservice, comprising:

An acquisition module, used to acquire an operation event performed when the source system updates data in the source database, wherein the operation event includes a business identifier;

A storage module, used to store the operation event in a message queue, so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database;

28. The device according to claim 27, wherein the acquisition module is specifically used to:

29. The apparatus according to claim 28, wherein the operation event comprises the execution time of the operation event by the source system; and the storage module is specifically configured to:

30. The device according to claim 27, wherein the acquisition module is specifically used to:

31. The device according to claim 27, wherein the storage module is specifically used for:

Using the hash value as a session identifier of the operation event;

32. The apparatus of claim 31 , further comprising:

The acquisition module is further configured to acquire other identifiers except the event identifier from the identifiers included in the operation event before performing a hash operation on the service identifier included in the operation event using a preset hash algorithm to obtain a hash value;

The splicing module is used to splice the other identifiers as the service identifier if there are multiple other identifiers.

33. A database synchronization device, applied to a consumer microservice, comprising:

An acquisition module is used to acquire operation events stored by the producer microservice from the message queue, wherein the operation event is an event executed when the source system updates data in the source database, and the operation event includes a business identifier;

A consumption module, used to determine a consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database;

34. The apparatus according to claim 33, wherein there are multiple consumer microservices; and the acquisition module is specifically used to:

The consumption module is specifically used for:

35. A database synchronization device, applied to a message queue, comprising:

A receiving module, used to receive an operation event stored by a producer microservice, wherein the operation event is an event executed when a source system updates data in a source database, and the operation event includes a business identifier, so that a consumer microservice obtains the operation event from the message queue, determines a consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on a target database;

There are multiple consumer microservices; the device also includes:

A setting module is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

36. The apparatus of claim 35, further comprising:

The receiving module is further configured to receive an execution completion message sent by the target consumer microservice after the target consumer microservice obtains any operation event in the target session;

The deleting module is used to delete the operation event from the target session.

37. The apparatus of claim 36, further comprising:

The setting module is further configured to set the target session to an unlocked state if no execution completion message sent by the target consumer microservice is received within a timeout period after the target consumer microservice obtains an operation event in the target session;

A selection module, configured to select a consumer microservice from other consumer microservices except the target consumer microservice to subscribe to the target session, so that the selected consumer microservice obtains an operation event from the target session;

The setting module is further configured to set the target session to a locked state when it is detected that the selected consumer microservice obtains an operation event from the target session for the first time.

38. The apparatus of claim 36, further comprising:

The allocation module is used to allocate the sessions subscribed by each consumer microservice according to a preset load balancing strategy when a new consumer microservice is detected in the database synchronization system.

39. The apparatus of claim 35, further comprising:

A determination module, configured to determine the dead letter session to which the operation event that meets the dead letter condition belongs if it is detected that any operation event in the message queue meets a preset dead letter condition;

The transfer module is used to store each operation event in the dead letter session in the dead letter queue in sequence.

40. The device according to any one of claims 35 to 39, wherein the message queue is a service bus.

41. An electronic device comprising:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 13 to 26.

42. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to any one of claims 13-26.

43. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 13 to 26.