Disclosure of Invention
The invention solves the problem of how to realize the accurate and efficient consistency check of data.
In order to solve the above problems, the present invention provides a data synchronization comparison method based on random sampling, which comprises: generating a sampling comparison configuration table according to random sampling rule configuration; generating a sampling data table according to the sampling comparison configuration table; and generating a comparison verification result table according to the sampling data table and the target end data table, and judging whether comparison verification passes according to the comparison verification result table.
The data synchronization comparison method based on random sampling checks the data consistency of the source end and the target end of the data synchronization based on the random sampling rule configuration, directly matches and compares the sampling data table with the data table of the target end through the main key, realizes the comparison and verification of the data synchronization record, and accurately and efficiently ensures the consistency of the data record of the data source end and the target end.
Optionally, the generating a sampling comparison configuration table according to a random sampling rule configuration includes: determining the number of sampling data record samples, a primary key of a configuration data table and a configuration data type field according to a random sampling rule configuration item, and generating the sampling comparison configuration table according to the number of the sampling data record samples, the primary key of the data table and the data type field.
The data synchronization comparison method based on random sampling determines the number of the sample of the sampled data record, configures the main key of the data table and configures the data type field according to the random sampling rule configuration item to generate the sampling comparison configuration table, thereby realizing the comparison verification of the data synchronization record and accurately and efficiently ensuring the consistency of the data record of the data source end and the data record of the target end.
Optionally, the data type field includes a string field, a numeric type field, and a time type field, and the configuration data type field includes: and selecting one representative field from the character string field, the numerical type field and the time type field as a sampling comparison field.
According to the data synchronization comparison method based on random sampling, the difference possibly generated by different data types is covered by respectively selecting one representative field from the character string field, the numerical value type field and the time type field as the sampling comparison field, the coverage rate of the sampling field type is increased, and the accuracy of comparison verification is improved.
Optionally, the generating a sampling comparison configuration table according to a random sampling rule configuration further includes: and when a plurality of data tables needing to be compared exist, writing the configuration of the plurality of data tables needing to be compared into the sampling comparison configuration table.
According to the data synchronous comparison method based on random sampling, when a plurality of data tables needing to be compared exist, the configuration of the data tables needing to be compared is written into the sampling comparison configuration table, and batch comparison verification of the data tables can be achieved.
Optionally, the generating a comparison verification result table according to the sampling data table and the target end data table includes: and matching and comparing the sampling data sheet with the target terminal data sheet, and generating the comparison verification result sheet according to the field type and the value difference determined by matching and comparing.
The data synchronization comparison method based on random sampling carries out matching comparison on the sampling data sheet and the target end data sheet, generates a comparison verification result sheet according to the field type and the numerical difference determined by matching comparison, and further can judge whether comparison verification passes or not according to the comparison verification result sheet.
Optionally, the determining whether the comparison verification passes according to the comparison verification result table includes: if the comparison verification result table does not have a difference set result, the comparison verification is passed, and if the comparison verification result table has the difference set result, an abnormal result alarm is given.
The data synchronization comparison method based on random sampling judges whether the comparison verification is passed or abnormal result warning is carried out according to the difference set result of the comparison verification result table, and the consistency of the data records of the data source end and the target end is accurately and efficiently ensured.
Optionally, the difference set result includes a primary key association mismatch, a numeric field check not passed, a time field check not passed, and a character field check not passed.
The data synchronization comparison method based on random sampling realizes the high-efficiency check of the data records of the data source end and the target end by setting the data sampling consistency check content.
The invention also provides a data synchronization comparison device based on random sampling, which comprises: the configuration module is used for generating a sampling comparison configuration table according to random sampling rule configuration; the data table module is used for generating a sampling data table according to the sampling comparison configuration table; and the comparison module is used for generating a comparison verification result table according to the sampling data table and the target end data table and judging whether the comparison verification passes according to the comparison verification result table. Compared with the prior art, the random sampling-based data synchronous comparison device and the random sampling-based data synchronous comparison method have the same advantages, and are not described herein again.
The invention also provides a data synchronization comparison system based on random sampling, which comprises a computer readable storage medium and a processor, wherein the computer readable storage medium is used for storing a computer program, and the computer program is read by the processor and runs to realize the data synchronization comparison method based on random sampling. Compared with the prior art, the data synchronization comparison system based on random sampling and the data synchronization comparison method based on random sampling have the same advantages, and are not described herein again.
The invention also provides a computer readable storage medium, which stores a computer program, and when the computer program is read and executed by a processor, the data synchronization comparison method based on random sampling is realized. The advantages of the computer-readable storage medium and the random sampling-based data synchronization comparison method over the prior art are the same, and are not described herein again.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.
As shown in fig. 1, an embodiment of the present invention provides a data synchronization comparison method based on random sampling, including: generating a sampling comparison configuration table according to random sampling rule configuration; generating a sampling data table according to the sampling comparison configuration table; and generating a comparison verification result table according to the sampling data table and the target end data table, and judging whether the comparison verification passes according to the comparison verification result table.
Specifically, in this embodiment, the data synchronization comparing method based on random sampling includes: generating a sampling comparison configuration table according to random sampling rule configuration; generating a sampling data table according to the sampling comparison configuration table; and generating a comparison verification result table according to the sampling data table and the target end data table, and judging whether the comparison verification passes according to the comparison verification result table. The data consistency of the source end and the target end of the data synchronization is checked based on random sampling rule configuration, a sampling data table generated at the source end is configured according to the rule, and then the sampling data table is directly matched and compared with the data table of the target end through a main key, so that the comparison and verification of the data synchronization record are realized, and the data consistency of the data source end and the target end is accurately and efficiently ensured. By taking the financial big data as an example, the consistency check of the financial big data can reduce or even eliminate the data error condition, thereby improving the financial security.
In this embodiment, the data consistency of the source end and the target end of the data synchronization is checked based on the random sampling rule configuration, and the sampling data table is directly matched and compared with the data table of the target end through the main key, so that the comparison and verification of the data synchronization record are realized, and the consistency of the data records of the source end and the target end is accurately and efficiently ensured.
Optionally, the generating a sampling comparison configuration table according to a random sampling rule configuration includes: determining the number of sampling data record samples, a primary key of a configuration data table and a configuration data type field according to a random sampling rule configuration item, and generating the sampling comparison configuration table according to the number of the sampling data record samples, the primary key of the data table and the data type field.
Specifically, in this embodiment, configuring and generating the sampling comparison configuration table according to the random sampling rule includes: (1) determining the number of sample data records to be sampled according to the random sampling rule configuration items, namely the number of random sampling N, performing configuration management, wherein the number can be set to be 20, 50, 100 and the like, and the data records representing sampling are corresponding N records; by randomly extracting representative records, the resource consumption of comparison verification can be reduced to a certain extent; (2) configuring a data table main key according to a random sampling rule configuration item, matching and comparing a sampling data table and a data table of a target end through the main key, wherein the purpose of configuring the main key is to perform associated matching and use with the data table of the target end at the later time, if the data table is an increment synchronous table, random sampling increment partition data can be selected, and a partition field is additionally added in a configuration item; (3) the data type field is configured according to a random sampling rule configuration item.
In this embodiment, the number of samples of the sampled data record, the primary key of the configuration data table, and the type field of the configuration data are determined according to the random sampling rule configuration item to generate a sampling comparison configuration table, so that comparison verification of data synchronization records can be realized, and the consistency of data records of a data source end and a target end can be accurately and efficiently ensured.
Optionally, the data type field includes a string field, a numeric type field, and a time type field, and the configuration data type field includes: and selecting one representative field from the character string field, the numerical type field and the time type field as a sampling comparison field.
Specifically, in this embodiment, the data type field includes a character string field, a numerical value type field, and a time type field, and for the table primary key and the custom configuration data type field, the data table needs to be analyzed in advance, the data table primary key and the sampling field are configured, and instead of performing comparison verification on the full field of the data table, one representative is selected from each of the three types of fields of each table, that is, three fields other than the data table primary key are selected.
In the embodiment, one representative field is selected from the character string field, the numerical field and the time type field respectively to serve as the sampling comparison field, so that the difference possibly generated by different data types is covered, the coverage rate of the sampling field type is increased, and the accuracy of comparison verification is improved.
Optionally, the generating a sampling comparison configuration table according to a random sampling rule configuration further includes: and when a plurality of data tables needing to be compared exist, writing the configuration of the plurality of data tables needing to be compared into the sampling comparison configuration table.
Specifically, in this embodiment, for the sampling comparison configuration table, when there are a plurality of data tables that need to be compared, the configuration of the plurality of data tables that need to be compared is written into the sampling comparison configuration table, so that batch comparison verification of the plurality of data tables can be realized.
In this embodiment, when there are a plurality of data tables to be compared, the configuration of the plurality of data tables to be compared is written into the sampling comparison configuration table, so that batch comparison and verification of the plurality of data tables can be realized.
Optionally, the generating a comparison verification result table according to the sampling data table and the target end data table includes: and matching and comparing the sampling data table with the target end data table, and generating the comparison verification result table according to the field type and the value difference determined by matching and comparing.
Specifically, in this embodiment, the sampling data table and the target end data table are matched and compared, a difference set between the sampling data table and the target end data table is compared, whether a field type and a numerical value are different is determined, a corresponding data comparison verification result is generated, and whether comparison verification passes or not can be determined according to the comparison verification result table.
In this embodiment, the sampling data table and the target-end data table are matched and compared, a comparison verification result table is generated according to the field type and the value difference determined by matching and comparing, and whether comparison verification passes or not can be determined according to the comparison verification result table.
Optionally, the determining whether the comparison verification passes according to the comparison verification result table includes: if the comparison verification result table does not have a difference set result, the comparison verification is passed, and if the comparison verification result table has the difference set result, an abnormal result alarm is given.
Specifically, in this embodiment, the determining whether the comparison verification passes according to the comparison verification result table includes: if the comparison verification result table does not have the difference set result, the comparison verification is passed, and if the comparison verification result table has the difference set result, an abnormal inspection result exists, and an abnormal result alarm needs to be performed.
In this embodiment, whether the comparison verification passes or an abnormal result alarm is performed is determined according to whether a difference set result occurs in the comparison verification result table, so that the consistency of the data records of the data source end and the target end is accurately and efficiently ensured.
Optionally, the difference set result includes a primary key association mismatch, a numeric field check not passed, a time field check not passed, and a character field check not passed.
Specifically, in this embodiment, during comparison and verification, after the sample data table is generated, the full-table records or the full-table fields of the original table do not need to be compared and verified one by one, only whether the difference exists between the sample data table and the corresponding field records of the target table is compared, and the data sampling consistency check is mainly:
(1) if the main keys are consistent, if the main key association matching is not up, the checking is not passed, and the normal matching represents that the checking is successful;
(2) if the numerical field is legal, and if the field values are different, if the numerical field is legal, the task fails to pass, and if the numerical field is not legal, the task passes;
(3) the time field is used for checking whether the time field value is legal or not, comparing the time field values, judging whether a difference exists or not, if so, failing to pass the task, and if not, successfully passing the task;
(4) and character type fields, checking whether field values are consistent or not, recording whether differences exist or not, failing to pass the task if the differences exist, and succeeding to pass the task if the differences do not exist.
In this embodiment, by setting the data sample consistency check content, efficient checking of data records of the data source end and the target end is achieved.
The following examples are given.
For example, table tab01 has a primary key pkey of uuid, and a sample field of: and tzje, djxh, lrrq, UUID- > pkey, tzje- > col _ double, djxh- > col _ string, lrrq- > timestamp (common data field types: double, string, timestamp, select representative 3 fields, and configure the fields into the configuration table by comma segmentation).
And randomly sampling N records and storing the N records into a sampling data table. And then matching and comparing the sampled data table with the target end data table, associating through a key main key, comparing whether the field values of the three data types are consistent, if so, checking to be passed, otherwise, checking not to be passed.
Another embodiment of the present invention provides a data synchronization comparing apparatus based on random sampling, including: the configuration module is used for generating a sampling comparison configuration table according to random sampling rule configuration; the data table module is used for generating a sampling data table according to the sampling comparison configuration table; and the comparison module is used for generating a comparison verification result table according to the sampling data table and the target end data table and judging whether the comparison verification passes according to the comparison verification result table.
Another embodiment of the present invention provides a random sampling-based data synchronization comparison system, which includes a computer-readable storage medium storing a computer program and a processor, where the computer program is read by the processor and executed to implement the above random sampling-based data synchronization comparison method.
Another embodiment of the present invention provides a computer-readable storage medium, which stores a computer program, and when the computer program is read and executed by a processor, the computer program implements the random sampling-based data synchronization comparison method as described above.
Although the present disclosure has been described above, the scope of the present disclosure is not limited thereto. Various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the spirit and scope of the present disclosure, and these changes and modifications are intended to be within the scope of the present disclosure.