CN111897877B - High-performance high-reliability data sharing system and method based on distributed ideas - Google Patents

Abstract

The invention discloses a high-performance and high-reliability data sharing system and method based on distributed ideas, which belongs to the field of information technology. The technical problems to be solved are that large-scale data sharing is inefficient and unsafe, and data sharing is difficult to manage and the sharing method is inconsistent. Intuitive and shared content is difficult to monitor. The technical solution is as follows: the system includes a database connection management unit, a scheduled task management unit, a task log management unit, a task statistics unit, a task monitoring unit and a failure reporting unit. The method is as follows: S1. Configure and start a data sharing task. The task regularly extracts data from the source database through defined rules to form a data text file, compresses and encrypts the file, then uploads it to the document center and records the access link; S2. Provides data to the outside world. Sharing interface, the user obtains the download link through the data sharing interface; S3, calls the download link to download the encrypted file in the document center, and decrypts it through the pre-assigned key to obtain the data.

Description

High-performance and high-reliability data sharing system and method based on distributed thinking

Technical field

The invention relates to the field of information technology, and mainly relates to efficient and safe data extraction and data sharing technology between multiple information systems. Specifically, it is a high-performance and high-reliability data sharing system and method based on distributed thinking.

Background technique

At present, the informatization construction of various enterprises has reached a certain scale, and a large amount of highly valuable data has been accumulated. Data is an important asset of an enterprise and the soul for its survival and sustainable development. Usually various system developers ( There are various reasons why these data resources are exclusive to each developer. However, the analysis of data within the enterprise requires different developers to open some data to cooperate. In order to enable various developers to share data conveniently and quickly, companies currently generally use three methods to share data:

The first method: Enterprises usually need to coordinate various developers to share data in the form of interfaces. This method can solve the problem of small amounts of data sharing, but it is helpless for large amounts of data. For example, it is necessary to synchronize the sales order data of retail terminals in the national tobacco industry (approximately tens of millions of orders per day) to the tobacco marketing analysis system every day, and ordinary methods are stretched. Moreover, interfaces are generally developed and held by each developer themselves, and enterprises generally cannot achieve real-time management and monitoring of shared data.

The second method: Enterprises need to coordinate developers and open read-only permissions on the database to achieve the purpose of data sharing. This method requires the detailed design of the database table structure to be made public so that other manufacturers can extract the data they need. However, this method can easily leak the underlying design of the developer's database, which is not only unsafe but also unfair to developers participating in data sharing.

The third way: Enterprises need to coordinate with developers and use existing data extraction tools such as ETL to extract the data that needs to be shared into a separate shared database. This method requires a lot of work, consumes a lot of human resources, and is unsafe and unintuitive.

To sum up, the existing technology has the following problems: it has long been difficult to share data within enterprises, large-scale data is difficult to share, traditional sharing methods are inefficient and unsafe, data sharing is difficult to manage, the sharing method is not intuitive, and the content is difficult to share. monitor.

Contents of the invention

The technical task of the present invention is to provide a high-performance and high-reliability data sharing system and method based on distributed thinking to solve the problem of low efficiency and insecurity of large-scale data sharing. At the same time, data sharing is difficult to manage, the sharing method is not intuitive, and the sharing content is difficult to manage. Difficult to monitor issues.

The technical task of the present invention is achieved in the following manner: a high-performance and high-reliability data sharing system based on distributed thinking. The data sharing system includes:

Database connection management unit, used to add source databases to be shared and modify or delete database connections;

Scheduled task management unit, used to add, modify, start, stop, delete and manually compensate scheduled tasks;

The task log management unit is used to view the detailed execution process of each historical task, the execution process of each historical task, and the storage location of the generated encrypted data files in the document center. It also provides viewing SQL, deleting logs, and advanced operations;

The task statistics unit is used to collect statistics on the execution process of each task;

The task monitoring unit is used to monitor any task in real time, view and analyze the task running status in the form of visual charts, and provide administrators with comprehensive data sharing task monitoring information;

The failure reporting unit is used to generate reports on failed tasks. Administrators can intuitively see the problem areas of failed tasks.

Preferably, the database connection management unit includes:

The new database connection module is used to add source databases to be shared through the database connection management function. It also provides a test connection button to test database connectivity and manage multiple database connections at any time;

The database connection modification and deletion module is used to modify or delete existing database connections at any time.

More preferably, the new content of the source database to be shared includes connection name, connection address, connection port number, database name, database user name, database password and database type.

Preferably, the scheduled task management unit includes:

A new module for scheduled tasks is added, which is used to add scheduled tasks through the new functions of scheduled task management;

The scheduled task modification module is used to modify existing data sharing scheduled tasks;

The scheduled task start, stop or delete module is used to start, stop or delete existing data sharing scheduled tasks accordingly;

The manual compensation task module is used to quickly and freely perform manual compensation on historical data.

More preferably, the information that needs to be maintained by the new scheduled task includes task name, agreed table name, connection name, cron expression, task step size, extraction format, extraction start time, time offset, and key type. , SQL type, task SQL and the full name of the timestamp field;

Among them, the agreed table name is used to protect the developer's underlying design strategy, and the developer can fill in this field at will;

The cron expression refers to the time command expression commonly used under the Linux operating system;

Task step size refers to how long the data needs to be fetched from the source database when executing this task, measured in minutes;

Time offset refers to how far forward it needs to be when selecting a task step. It is used to solve the problem of data entry delay when using database transactions to process complex businesses;

Key type refers to the ability to choose different public and private key pairs for encryption when encrypting files;

The SQL type allows the system administrator to tell the system the parameter filling strategy when executing SQL tasks, including automatic splicing parameters and manually defined parameters;

Task SQL refers to the structured query language for data sharing tasks;

The full name of the timestamp field is used to solve the problem of diverse naming of timestamp fields by various developers, and is also used to indicate which field in the task sql represents the timestamp.

Preferably, the task log management unit includes:

The task log clearing module is used to clear all the history of the selected task and clear unnecessary history records;

The SQL viewing module is used to view the specific execution SQL of the selected task and assist in analyzing problems that arise during the data sharing process;

Log deletion module, used to delete task logs one by one;

Advanced operation module, used to provide more advanced operations for log management.

Preferably, the content counted by the task statistics unit includes the number of shared data rows, the size of the shared data text file, the size of the shared data encrypted file, and the duration of the connection to the database.

As an option, the data sharing system provides two working modes, as follows:

①. Single Master mode: The data sharing system works on one server and will automatically simulate multiple Slave sub-nodes through multi-threading;

②. Master-Slave mode: Manually split the data sharing system into Master and Slave. The Master node works on one server and the Slave node works on multiple other servers. The MQ message intermediate is used between the Master node and Slave node. to communicate.

A high-performance and high-reliability data sharing method based on distributed thinking. The method is as follows:

S1. The system administrator configures and starts the data sharing task. The task regularly extracts data from the source database through defined rules to form a data text file, compresses and encrypts the file, and then uploads it to the document center (internal unstructured database management system of the enterprise). and record the access link;

S2. The data sharing system provides a data sharing interface to the outside world, and the user obtains the download link through the data sharing interface;

S3. Go to the document center to download the encrypted file by calling the download link, and decrypt it with the pre-assigned key to obtain the data.

As an option, the data sharing system can automatically split a sharing task with a huge amount of data according to the time dimension according to the task step set by the administrator, and perform distributed execution to reduce the pressure on the source database;

The data sharing system forms a text file from the source database and encrypts it through the RSA asymmetric encryption algorithm to form an encrypted file. The encrypted file is sent to the document center for storage. Users who need to share data directly download the data from the document center through the link. pressure to shift data-sharing systems;

Users who need to share data need to obtain the target data in two steps, as follows:

① First call the interface provided by the data sharing system;

②. Based on the result obtained in the first step, call the interface of the document center to obtain the encrypted file.

The high-performance and high-reliability data sharing system and method based on distributed thinking of the present invention have the following advantages:

(1) The present invention provides a brand new solution for internal data sharing within the enterprise, breaking the barriers of traditional data sharing difficulties. The sharing method is very intuitive, the sharing process is efficient, safe and reliable, and it can realize convenient and efficient management sharing tasks and be able to Real-time monitoring of data sharing tasks can save a lot of human resource costs and provide strong support for internal data sharing within the enterprise, especially the rapid sharing of ultra-large-scale data;

(2) The present invention can effectively solve the problem of data sharing within the enterprise, especially the problem of large-scale data sharing, and can manage data sharing tasks conveniently and quickly, and achieve large-scale data sharing efficiently and safely. Provide safe and efficient solutions for intra-enterprise data sharing, especially the sharing of large amounts of data;

(3) In order to reduce the access pressure on the source database during the data aggregation process and avoid affecting the normal operation of other business systems, the present invention divides a large number of data sharing tasks into N small data sharing tasks, which can be connected through the database connection pool The database performs a data extraction task at corresponding intervals, extracting data of step size (time length) at a time. The extracted data is compressed, encrypted and uploaded to the document center for storage;

(4) In addition to the characteristics of ordinary data extraction and sharing tools, Go to Maning also has the following characteristics:

① Security: The platform must grant the developer permission to call the data sharing interface, which ensures the security of the interface; the data files are stored in the document center in an RSA encrypted manner, and the developer must apply to obtain the RSA private key to access the obtained data. Decrypt data files;

② High performance: Adopting distributed thinking and based on the Master-Slave mode (one master and multiple slaves mode), the master node is only used to generate and dispatch tasks, and the slave nodes are used to process these tasks. There can be multiple slave nodes, which can handle large-scale data extraction and sharing tasks at the same time;

③ High availability: The tasks dispatched by the master node are stored in the message middleware (hereinafter referred to as MQ). At the same time, the slave nodes have multiple nodes and are deployed on different machines to ensure the high availability of the system;

④ Convenient management: Through the present invention, the system administrator can add data sharing tasks at any time, modify data extraction rules at any time, start data sharing tasks at any time, stop data sharing tasks at any time, and delete data sharing tasks at any time;

⑤ Monitorable statistics: This invention can check the running status of tasks in real time and monitor the success and failure of tasks; at the same time, it provides various dimensional task statistical reports and visual charts to grasp the running status of tasks in real time;

⑥ Compensable: For failed tasks, the present invention has both automatic and manual compensation functions. When a task failure is detected, the compensation mechanism will be started immediately to compensate for the failed task. At the same time, the administrator can also manually compensate. Easy and fast operation;

⑦ Source database pressure reduction: The present invention provides the ability to split the same data sharing task, dividing a data sharing task into n multiple small tasks, realizing time sharing and segmenting the database to read data, so as to avoid putting pressure on the source database. , while improving the speed of data extraction and sharing;

⑧Reducing the pressure on the data sharing system: The data obtained from the source database is processed and sent to the document center for storage. System developers who need to share data directly download the data from the document center through the link, thus shifting the pressure on the data sharing system.

Description of the drawings

The present invention is further described below in conjunction with the accompanying drawings.

Figure 1 is a structural block diagram of a high-performance and highly reliable data sharing system based on distributed ideas;

Figure 2 is an architectural diagram of the working principle of the data sharing system in Figure 1;

Figure 3 is a screenshot of the interface for adding a new database connection;

Figure 4 is a screenshot of the interface of the new scheduled task module;

Figure 5 is the second screenshot of the interface of the new scheduled task module;

Figure 6 is a screenshot of the interface of the task log management unit;

Figure 7 is the second screenshot of the interface of the task log management unit;

Figure 8 is a screenshot of the interface of the task statistics unit;

Figure 9 is a screenshot of the interface of the line chart of the task monitoring unit;

Figure 10 is a screenshot of the pie chart interface of the task monitoring unit;

Figure 11 is a screenshot of the failure report interface.

Detailed ways

The high-performance and high-reliability data sharing system and method based on the distributed idea of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Example 1:

As shown in Figure 1, the high-performance and high-reliability data sharing system of the present invention is based on distributed thinking. The data sharing system includes:

The database connection management unit is used to add source databases to be shared and modify or delete database connections; the database connection management unit includes,

The new database connection module is used to add source databases to be shared through the database connection management function. It also provides a test connection button to test database connectivity and manage multiple database connections at any time, as shown in Figure 3;

The database connection modification and deletion module is used to modify or delete existing database connections at any time.

Among them, the new content of the source database to be shared includes connection name, connection address, connection port number, database name, database user name, database password and database type.

The scheduled task management unit is used to add, modify, start, stop, delete and manually compensate scheduled tasks; the scheduled task management unit includes,

The new scheduled task module is used to add scheduled tasks through the new functions of scheduled task management; the information that needs to be maintained for new scheduled tasks includes task name, agreed table name, connection name, cron expression, task step size, extraction time Format, extraction start time, time offset, key type, SQL type, task SQL and the full name of the timestamp field, as shown in Figures 4 and 5;

Among them, the agreed table name is used to protect the developer's underlying design strategy, and the developer can fill in this field at will;

The cron expression refers to the time command expression commonly used under the Linux operating system;

Task step size refers to how long the data needs to be fetched from the source database when executing this task, measured in minutes;

Time offset refers to how far forward it needs to be when selecting a task step. It is used to solve the problem of data entry delay when using database transactions to process complex businesses;

Key type refers to the ability to choose different public and private key pairs for encryption when encrypting files;

The SQL type allows the system administrator to tell the system the parameter filling strategy when executing SQL tasks, including automatic splicing parameters and manually defined parameters;

Task SQL refers to the structured query language for data sharing tasks;

The full name of the timestamp field is used to solve the problem of diverse naming of timestamp fields by various developers, and is also used to indicate which field in the task sql represents the timestamp.

The scheduled task modification module is used to modify existing data sharing scheduled tasks;

The scheduled task start, stop or delete module is used to start, stop or delete existing data sharing scheduled tasks accordingly;

The manual compensation task module is used to quickly and freely perform manual compensation on historical data.

The task log management unit is used to view the detailed execution process of each historical task, the execution process of each historical task, and the storage location of the generated encrypted data files in the document center. It also provides viewing of SQL, deletion of logs, and advanced operations, as attached. As shown in Figures 6 and 7; the task log management unit includes,

The task log clearing module is used to clear all the history of the selected task and clear unnecessary history records;

The SQL viewing module is used to view the specific execution SQL of the selected task and assist in analyzing problems that arise during the data sharing process;

Log deletion module, used to delete task logs one by one;

Advanced operation module, used to provide more advanced operations for log management.

The task statistics unit is used to collect statistics on the execution process of each task, as shown in Figure 8; the statistics of the task statistics unit include the number of shared data rows, the size of the shared data text file, the size of the shared data encryption file, and the length of the connection to the database. .

The task monitoring unit is used to monitor any task in real time, view and analyze the task running status in the form of visual charts, and provide administrators with a full range of data sharing task monitoring information, as shown in Figures 9 and 10;

The failure reporting unit is used to generate reports on failed tasks. Administrators can intuitively see the problem areas of failed tasks, as shown in Figure 11.

As shown in Figure 2, the data sharing system provides two working modes, as follows:

①. Single Master mode: The data sharing system works on one server and will automatically simulate multiple Slave sub-nodes through multi-threading;

②. Master-Slave mode: Manually split the data sharing system into Master and Slave. The Master node works on one server and the Slave node works on multiple other servers. The MQ message intermediate is used between the Master node and Slave node. to communicate.

The working process of the data sharing system is as follows:

(1) Log in as the system administrator and add a database to be shared through the database management function;

(2) Configure scheduled tasks and data sharing rules and start the tasks;

(3) The data sharing system will regularly extract data from the source database according to the defined rules according to the configuration of the system administrator to form a data text file;

(4) After the data text files are formed, the data sharing system will archive and compress these text files and encrypt them through the RSA asymmetric algorithm to form encrypted files. Then the system will upload the encrypted files to the enterprise's internal unstructured database management system (referred to as document Center) and record the access link;

(4) The data sharing system provides an external data sharing interface, and developers can obtain download links through the data sharing system interface;

(5) Call this link to download the encrypted file from the document center, and decrypt it with the pre-assigned key to obtain the data.

The invention is suitable for the retail industry.

Example 2:

The present invention's high-performance and high-reliability data sharing method based on distributed thinking is specifically as follows:

S1. The system administrator configures and starts a data sharing task. The task regularly extracts data from the source database through defined rules to form a data text file, compresses and encrypts the file, and then uploads it to the document center (internal unstructured database management system of the enterprise). and record the access link;

S2. The data sharing system provides a data sharing interface to the outside world, and the user obtains the download link through the data sharing interface;

S3. Go to the document center to download the encrypted file by calling the download link, and decrypt it with the pre-assigned key to obtain the data.

Among them, the data sharing system can automatically split a sharing task with a huge amount of data according to the time dimension according to the task step set by the administrator, and perform distributed execution to reduce the pressure on the source database;

The data sharing system forms a text file from the source database and encrypts it through the RSA asymmetric encryption algorithm to form an encrypted file. The encrypted file is sent to the document center for storage. Users who need to share data directly download the data from the document center through the link. pressure to shift data-sharing systems;

Users who need to share data need to obtain the target data in two steps, as follows:

① First call the interface provided by the data sharing system;

②. Based on the result obtained in the first step, call the interface of the document center to obtain the encrypted file.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features can be equivalently replaced; and these modifications or substitutions do not deviate from the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention. scope.

Claims (8)

1. A high-performance and high-reliability data sharing system based on a distributed idea is characterized in that the data sharing system comprises,

the database connection management unit is used for newly adding a source database to be shared and modifying or deleting database connection;

the timing task management unit is used for adding, modifying, starting, stopping, deleting and manually compensating the timing task;

the task log management unit is used for checking the detailed execution process of each historical task, the execution process of each historical task and the storage position of the generated encrypted data file in the document center, and simultaneously providing SQL checking, log deleting and advanced operation;

the task statistics unit is used for counting each task execution process; the content counted by the task counting unit comprises shared data line numbers, shared data text file sizes, shared data encryption file sizes and database connection time;

the task monitoring unit is used for monitoring any task in real time, checking and analyzing the running state of the task in a visual chart form and providing all-round data sharing task monitoring information for an administrator;

and the failure report unit is used for generating a report for the failed task, and an administrator visually sees the link of the failed task with a problem.

2. The high performance and high reliability data sharing system based on the distributed concept as claimed in claim 1, wherein said database connection management unit comprises,

the database connection newly-added module is used for newly adding a source database to be shared through a database connection management function, providing a test connection button, testing the connectivity of the database at any time and managing various database connections;

and the database connection modification and deletion module is used for modifying or deleting the existing database connection at any time.

3. The high-performance and high-reliability data sharing system based on the distributed idea according to claim 2, wherein the content newly added to the source database to be shared includes a connection name, a connection address, a connection port number, a database name, a database user name, a database password, and a database type.

4. The high performance high reliability data sharing system based on distributed architecture according to claim 1, wherein the timed task management unit comprises,

the timing task adding module is used for adding timing tasks through a new adding function of timing task management;

the timing task modification module is used for modifying the existing data sharing timing task;

the timing task starting, stopping or deleting module is used for carrying out corresponding starting, stopping or deleting operation on the existing data sharing timing task;

and the manual compensation task module is used for rapidly and freely carrying out manual compensation on the historical data.

5. The distributed idea based high performance and high reliability data sharing system according to claim 4, wherein the information to be maintained by the newly added time task includes task name, contract table name, connection name, cron expression, task step size, extraction time format, extraction start time, time offset, key type, SQL type, task SQL and timestamp field full name;

the contracted table name is used for protecting strategies designed by the bottom layer design of the developer, and the developer fills in the field at will;

the cron expression refers to a time instruction expression commonly used under a linux operating system;

the task step length refers to data of how long the task needs to be fetched from a source database when the task is executed, and the data is measured in minutes;

the time offset refers to how long the task step length needs to be shifted forward when being selected, and is used for solving the problem of data warehouse-in delay when complex business is processed by using database transactions;

the key type refers to that different public and private key pairs can be selected for encryption when the file is encrypted;

the SQL type allows a system administrator to tell the system of a parameter filling strategy when executing an SQL task, and specifically comprises automatic splicing parameters and manual definition parameters;

task SQL is a structured query language for data sharing tasks;

the timestamp field full name is used to solve the problem of multiple names of the timestamp fields of the respective developers, and is also used to indicate which field in the task sql represents the timestamp.

6. The high-performance high-reliability data sharing system based on the distributed concept as claimed in claim 1, wherein said task log management unit comprises,

the task log emptying module is used for emptying all histories of the selected tasks and emptying unnecessary history records;

the SQL checking module is used for checking the concrete execution SQL of the selected task and assisting in analyzing the problems in the data sharing process;

the log deleting module is used for deleting the task logs one by one;

and the advanced operation module is used for providing higher-level operation on log management.

7. The high-performance and high-reliability data sharing system based on the distributed idea according to claim 1, wherein the data sharing system provides two operation modes, specifically as follows:

(1) single Master mode: the data sharing system works on a server and simulates a plurality of Slave child nodes automatically in a multithreading mode;

(2) Master-Slave mode: and manually splitting the data sharing system into a Master and a Slave, wherein the Master node works on one server, the Slave node works on other multiple servers, and the Master node and the Slave node are communicated by adopting an MQ message middleware.

8. A high-performance and high-reliability data sharing method based on a distributed idea is characterized by comprising the following steps:

s1, a system administrator configures and starts a data sharing task, the task regularly extracts data from a source database through defined rules to form a data text file, compresses and encrypts the file, and then uploads the file to a document center and records access links;

s2, the data sharing system provides a data sharing interface to the outside, and a user obtains a downloading link through the data sharing interface;

s3, downloading the encrypted file through calling a downloading link to a document center, and decrypting through a key distributed in advance to obtain data;

the data sharing system can automatically split and execute a sharing task with huge data volume according to the task step length set by an administrator and the time dimension, so that the pressure of a source database is reduced;

the data sharing system forms a text file from the data acquired from the source database and encrypts the text file through an RSA asymmetric encryption algorithm to form an encrypted file, the encrypted file is transmitted to a document center for storage, and a user needing to share the data directly downloads the data from the document center through a link, so that the pressure of the data sharing system is transferred;

the user needing to share the data needs to acquire the target data in two steps, which is as follows:

(1) firstly, calling an interface provided by a data sharing system;

(2) and calling an interface of the document center according to the result obtained in the first step to obtain the encrypted file.