CN119938662A

CN119938662A - A data cleaning method and data cleaning engine based on rule data drive

Info

Publication number: CN119938662A
Application number: CN202510433442.3A
Authority: CN
Inventors: 宁黎; 孟庆国; 刘诗源; 王俭; 角帅涛; 王思两
Original assignee: Beijing Liujinnian Technology Co ltd
Current assignee: Beijing Liujinnian Technology Co ltd
Priority date: 2025-04-08
Filing date: 2025-04-08
Publication date: 2025-05-06
Anticipated expiration: 2045-04-08
Also published as: CN119938662B

Abstract

The present application relates to a data cleaning method and a data cleaning engine based on rule data drive, and belongs to the field of data processing technology. The data cleaning method includes: receiving the original data streams of multiple heterogeneous business systems; extracting multi-source data features and cross-system field association features, and constructing data feature fingerprints; performing similarity matching on the data feature fingerprints, screening rules with matching degrees higher than a preset threshold, and obtaining a candidate rule set; converting the candidate rule set into a directed acyclic graph; marking conflicting rule pairs, and obtaining a conflict warning list; inserting a virtual isolation layer into the directed acyclic graph, and generating a path identification table; topologically sorting the directed acyclic graph, generating a rule execution sequence and cleaning the original data stream; allocating data execution channels according to the path identification table; outputting the cleaned clean data stream and the cleaning log containing the rule execution path. The present application can efficiently and accurately process data from different business systems.

Description

Rule data driving-based data cleaning method and data cleaning engine

Technical Field

The application relates to the technical field of data processing, in particular to a data cleaning method and a data cleaning engine based on rule data driving.

Background

With the rapid development of information technology, data volume has increased exponentially, and particularly in various enterprises, various heterogeneous business systems generate a large amount of structured, semi-structured and unstructured data. Such data sources are diverse, including customer transaction records, product information, user behavior data, sensor data, social media text, and the like. Meanwhile, more and more enterprises gradually adopt emerging technologies such as big data technology, cloud computing technology, internet of things (IoT) and the like, and the technologies further promote the generation, transmission and storage of enterprise data. Enterprise business systems are also becoming increasingly complex, often using different technology stacks (e.g., relational databases, noSQL databases, file systems, log systems, etc.) to store and manage different types of data.

However, the heterogeneous nature, dispersion, and ever-expanding diversity requirements of data make data management and cleansing extremely complex. For various business systems, how to accurately extract useful information from a plurality of different data sources, and to reasonably integrate the useful information, so as to ensure the accuracy, consistency, integrity and high efficiency of data, is an important challenge in the current enterprise informatization process.

At present, common data cleaning methods often focus on cleaning and processing of a single data source, and the processing capacity and the intelligent level of multi-source data are still insufficient. In particular, processing data from different business systems, there is often a large variance in data formats, transmission protocols, and storage structures among these systems, resulting in high complexity, high cost, and high risk in data processing.

Therefore, how to efficiently and accurately process data from different service systems in the face of complex data sources, various service demands and large-scale distributed data processing is a key problem to be solved.

Disclosure of Invention

In order to efficiently and accurately process data from different service systems, the application provides a data cleaning method and a data cleaning engine based on rule data driving.

In a first aspect, the present application provides a rule data driving-based data cleaning method, which adopts the following technical scheme:

a data cleansing method based on rule data driving, the data cleansing method comprising:

receiving original data streams of a plurality of heterogeneous service systems;

extracting multi-source data characteristics and cross-system field association characteristics in the original data stream, and constructing to obtain data characteristic fingerprints;

Based on condition features in a preset rule base, similarity matching is carried out on the data feature fingerprints, and rules with matching degree higher than a preset threshold are screened to obtain a candidate rule set;

converting the candidate rule set into a directed acyclic graph, wherein nodes of the directed acyclic graph represent rules and edges represent data dependency relationships;

analyzing the input-output dependency relationship of each candidate rule in the candidate rule set, and marking conflict rule pairs to obtain a conflict early warning list;

inserting a virtual isolation layer into the directed acyclic graph according to the conflict early warning list, and binding the conflict rule pair with an isolation layer mark to generate a path mark table;

Performing topological sorting according to the directed acyclic graph inserted into the virtual isolation layer to generate a rule execution sequence;

cleaning the original data stream according to the rule execution sequence, and attaching a track label to each piece of data, wherein the track label comprises a rule path identifier, a field modification history and an isolation layer identifier;

a data execution channel is allocated according to the path identification table, wherein when the isolated layer identification in the track label is detected to be matched with the conflict early warning list, data rerouting is triggered;

And outputting the clean data stream with the cleaning completed and the cleaning log containing the regular execution path.

By adopting the technical scheme, a high-efficiency and reliable data cleaning system is formed through multisource data fusion, rule-dependent modeling, conflict isolation and topology execution. According to the technical scheme, when multi-source data are faced, the relation among different systems can be flexibly and accurately processed, rule conflicts can be effectively detected and processed, interference of conflict rule pairs is reduced, data quality is ensured, data cleaning throughput and efficiency are improved, and therefore accurate and stable data support is provided for various service systems.

Optionally, the original data stream comprises structured numeric data, text data or binary files, and the step of extracting the multi-source data features in the original data stream comprises:

performing dynamic distribution statistics on the structured numerical data to generate data distribution characteristics;

extracting semantic vector features from the text data;

analyzing the binary file and extracting metadata features;

And obtaining the multi-source data characteristic according to the data distribution characteristic, the semantic vector characteristic and the metadata characteristic.

By adopting the technical scheme, a multi-source data characteristic set is formed, rich information is provided for subsequent data cleaning, anomaly detection and pattern recognition, the accuracy and the efficiency of data processing are improved, diversified heterogeneous data can be intelligently processed, and the comprehensiveness and the high quality of data cleaning are ensured.

Optionally, the step of analyzing the input-output dependency relationship of each candidate rule in the candidate rule set, marking a conflict rule pair, and obtaining a conflict early warning list includes:

Receiving a candidate rule set, analyzing an input field set and an output field set of each candidate rule, and generating a rule input-output field table;

Constructing a rule dependency graph based on the input/output field table;

traversing the rule dependency graph, detecting rule pairs which have intersections in output fields and have no dependency paths, and marking the rule pairs as conflict rule pairs;

determining the confidence level of the conflict rule pair according to cross-system field association features in the data feature fingerprint;

and matching the corresponding preset routing strategy according to the confidence coefficient of each conflict rule pair, and generating a conflict early warning list.

By adopting the technical scheme, the rule dependency graph, the cross-system field association characteristic and the confidence adjustment mechanism are comprehensively used, so that the rule conflict problem in the data cleaning process is effectively identified and processed. Through the dependency relationship among the clear rules, potential conflicts are found in time, the confidence coefficient of the conflicts is adjusted, the processing flow is optimized according to a preset routing strategy, and finally an efficient and intelligent conflict early warning list and a processing path are generated, so that the accuracy, the processing efficiency and the flexibility of data cleaning are improved.

Optionally, inserting a virtual isolation layer in the directed acyclic graph according to the conflict early warning list, and identifying the conflict rule pair binding isolation layer, wherein the step of generating the path identification table includes:

positioning the nearest public ancestor node of the conflict rule pair according to the directed acyclic graph and the conflict early warning list;

inserting a virtual isolation layer node after the nearest public ancestor node points to the outgoing edge of the conflict rule pair, and generating an updated directed acyclic graph;

Injecting an isolation layer identifier, a conflict rule pair and a bypass copy generation mark into the virtual isolation layer node;

and generating a path identification table according to a preset routing strategy and cross-system association characteristics of the data characteristic fingerprints in the conflict early warning list.

By adopting the technical scheme, the root of the conflict rule is accurately found based on LCA positioning, the virtual isolation layer is inserted at a proper position, the conflict path is successfully isolated, the data can be ensured to flow according to the proper path through the analysis of the intelligent routing strategy and the cross-system characteristics, and the problems of incomplete data conflict isolation, poor cross-system consistency and the like are effectively solved.

Optionally, topological ordering is performed according to the directed acyclic graph after the virtual isolation layer is inserted, the step of generating a rule execution sequence includes:

Injecting node labels into the directed acyclic graph after the virtual isolation layer is inserted according to the path identification table;

initializing a high-priority queue and a low-priority queue according to node tag types, and sequencing nodes in the queues based on a service priority policy, wherein the node tag types comprise a main chain node, a branch node and a virtual isolation layer node;

Traversing the high priority queue and the low priority queue in a layering way to generate a main execution sequence and a branch execution sequence;

And obtaining a rule execution sequence according to the main execution sequence and the branch execution sequence.

By adopting the technical scheme, the system can clearly identify the main chain node and the branch node based on the injection of the node label and reasonably schedule the resource according to the priority strategy, and meanwhile, the hierarchical traversal ensures that the rule is executed according to the priority, so that the execution flow of the complex rule is more stable and efficient.

Optionally, the step of hierarchically traversing the high priority queue and the low priority queue to generate the main execution sequence and the branch execution sequence includes:

circularly extracting main chain nodes in the high-priority queue to add a main execution sequence;

when the high-priority queue is empty, extracting a current branch node in the low-priority queue to join in a branch execution sequence;

Traversing all outgoing edges of the current branch node, and screening the subsequent nodes with the node label type of the branch;

updating the degree of incidence of the subsequent node, and adding the subsequent node with the degree of incidence being zero and the node label type being a branch into a low-priority queue.

By adopting the technical scheme, the system can efficiently manage the execution sequence and the priority of the nodes, ensures the priority execution of the main chain nodes and the timely processing of the branch nodes after the main chain nodes are executed, flexibly schedules the tasks according to the dependency relationship and the resource state of the nodes, optimizes the resource use, avoids unordered execution and conflict, and improves the parallelism and the efficiency of the task execution.

Optionally, after the step of outputting the clean data stream and the associated clean log after the cleaning is completed, the method further comprises:

based on rule execution success rate and business feedback data in the cleaning log, eliminating rules with execution success rate lower than a preset success rate threshold and correcting feature extraction parameters;

And encoding the historical rule execution path into a rule gene library capable of genetic optimization, and iteratively updating the rule library version.

By adopting the technical scheme, in the rule execution process, the system can monitor the success rate of the rule through the cleaning log, eliminate the low-efficiency rule and continuously correct the feature extraction parameters, thereby ensuring the high efficiency and accuracy of the rule base. Meanwhile, the historical rule execution path is converted into a genetic optimizing gene library, and the genetic algorithm is utilized to carry out iterative updating of the rules, so that the rule library has self-optimizing capability. The technical scheme improves the self-adaptive capacity of the system in complex data processing and service execution, and ensures that the rule base always maintains a high-efficiency, accurate and sustainable optimized state in continuously changing service demands.

In a second aspect, the present application provides a rule data driven based data cleansing engine, which adopts the following technical scheme:

a rule data driven based data cleansing engine, the data cleansing engine comprising:

The receiving module is used for receiving the original data streams of a plurality of heterogeneous service systems;

The feature extraction module is used for extracting multi-source data features and cross-system field association features in the original data stream and constructing and obtaining data feature fingerprints;

the rule screening module is used for carrying out similarity matching on the data characteristic fingerprints based on the condition characteristics in the preset rule base, screening rules with matching degree higher than a preset threshold value and obtaining a candidate rule set;

the conversion module is used for converting the candidate rule set into a directed acyclic graph, wherein nodes of the directed acyclic graph represent rules, and edges represent data dependency relations;

the conflict marking module is used for analyzing the input-output dependency relationship of each candidate rule in the candidate rule set, marking conflict rule pairs and obtaining a conflict early warning list;

The path identification table generation module is used for inserting a virtual isolation layer into the directed acyclic graph according to the conflict early warning list, binding the conflict rule pair with an isolation layer identifier, and generating a path identification table;

The topological ordering module is used for carrying out topological ordering according to the directed acyclic graph inserted into the virtual isolation layer to generate a rule execution sequence;

the cleaning processing module is used for cleaning the original data stream according to the rule execution sequence and attaching a track label to each piece of data, wherein the track label comprises a rule path identifier, a field modification history and an isolation layer identifier;

The execution channel allocation module is used for allocating data execution channels according to the path identification table, wherein when the isolation layer identification in the track label is detected to be matched with the conflict early warning list, the data rerouting is triggered;

and the output module is used for outputting the clean data stream after cleaning and the cleaning log containing the regular execution path.

In a third aspect, the present application provides a computer device, which adopts the following technical scheme:

A computer device comprising a memory, a processor and a computer program stored on the memory, the processor executing the computer program to perform the steps of the method according to the first aspect.

In a fourth aspect, the present application provides a computer readable storage medium, which adopts the following technical scheme:

a computer readable storage medium storing a computer program capable of being loaded by a processor and executing any one of the methods of the first aspect.

In summary, the application has at least one of the following beneficial technical effects that the application can effectively extract and match data characteristics from a plurality of heterogeneous service systems, optimize the sequence of rule execution and process data dependency relations across systems. By constructing a directed acyclic graph of rules, marking conflict rule pairs and introducing virtual isolation layers, the reliability of rule execution and the accuracy of data are ensured. Meanwhile, the track label generation and data rerouting mechanism provides detailed execution record and dynamic control for the data cleaning process, so that the data processing path can be flexibly adjusted according to the service requirement. The technical scheme not only improves the efficiency and the precision of data cleaning, but also provides comprehensive traceability and transparency for data quality control, greatly enhances the intelligent and automatic level of data processing, can cope with complex heterogeneous data environments, improves the data quality, and supports more accurate data decision and business operation.

Drawings

FIG. 1 is a schematic flow chart of a rule data driven data cleansing method according to one embodiment of the present application.

FIG. 2 is a schematic diagram of a second flow chart of a rule data driven based data cleansing method according to one embodiment of the present application.

FIG. 3 is a third flow chart of a rule data driven based data cleansing method according to one embodiment of the present application.

Fig. 4 is a fourth flowchart of a rule data driven data cleansing method according to one embodiment of the present application.

FIG. 5 is a fifth flow chart of a rule data driven based data cleansing method according to one embodiment of the present application.

FIG. 6 is a sixth flow chart of a rule data driven based data cleansing method according to one embodiment of the present application.

Fig. 7 is a schematic diagram of a seventh flow chart of a rule data driven data cleansing method according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings 1 to 7 and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The embodiment of the application discloses a data cleaning method based on rule data driving.

Referring to fig. 1, a rule data driven based data cleansing method includes:

Step S101, receiving original data streams of a plurality of heterogeneous service systems;

the original data stream comprises a structured numerical data table, text data and a binary file;

Specifically, data is accessed from a plurality of heterogeneous service systems in a unified way, and as the service systems often use different technical stacks (such as a relational database, a NoSQL database, a log system or a binary file, etc.), the step needs to use an adapter mode to package different data source access interfaces so as to realize unified access of different data formats.

Illustratively, the e-commerce system pushes order data (structured data) through a Kafka message queue, the logistics system uploads and signs photos (binary files) through SFTP, the payment system returns transaction flow (Protobuf format) through gRPC interface, and after the data is accessed through an adapter, the data is uniformly converted into Parquet format for storage.

Step S102, multi-source data features and cross-system field association features in an original data stream are extracted, and data feature fingerprints are constructed;

Wherein, the data feature fingerprint (DFP) is a global feature set comprising data distribution features, semantic vector features and cross-system field association features of multiple business systems.

The method comprises the steps of calculating statistics of mean, variance, skewness, kurtosis and the like of data fields such as transaction amount, user age and the like, converting unstructured text into semantic vectors by using a pre-trained language model (such as BERT) to capture deep semantics of the text, and constructing a field association network among different service systems by cross-system field association features, wherein association relations between an order system and a payment system are recorded through a graph database.

It can be appreciated that through feature fusion of multiple dimensions, DFP can provide rich context information for subsequent rule matching, enhancing accuracy of data cleaning. The numerical characteristics can identify abnormal modes, the text characteristics can be accurately matched with specific contents, and cross-system association characteristics are used for guaranteeing consistency of rules in different business systems.

Step S103, performing similarity matching on the data characteristic fingerprints based on the condition characteristics in the preset rule base, and screening rules with matching degree higher than a preset threshold value to obtain a candidate rule set;

And performing similarity matching on the data feature fingerprints (DFP) based on the condition features in the preset rule base, and screening rules with matching degree higher than a preset threshold. Each rule may consist of a conditional portion (e.g., an amount greater than 1000) and an action portion (e.g., marked as suspicious transaction).

Specifically, the matching process includes structured condition matching, semantic condition matching, and cross-system field matching. The method comprises the steps of structuring condition matching, such as numerical condition in an amount field matching rule, semantic condition matching, such as semantic matching of text fields, calculating similarity between semantic vectors of data feature fingerprints and rule conditions, cross-system field matching, such as checking whether field association strength in the data feature fingerprints meets requirements when the rule depends on fields in two systems.

Step S104, converting the candidate rule set into a directed acyclic graph, wherein nodes of the directed acyclic graph represent rules and edges represent data dependency relations;

Specifically, the dependency relationship among the rules in the candidate rule set is converted into a Directed Acyclic Graph (DAG), the rule execution sequence is defined, the nodes represent the rules, and the edges represent the dependency relationship among the rules. The dependencies include explicit dependencies (e.g., the output of rule R1 is the input of rule R2) and implicit dependencies (e.g., business logic requires rule R1 to execute first).

Illustratively, in a credit-wind scenario, rules R201 (calculating credit) and R202 (approving the amount based on credit) constitute explicit dependencies, while rules R203 (verifying proof of work) and R204 (verifying revenue flowing) need to be executed in business logic order, although there is no dependency.

It will be appreciated that DAG modeling clearly shows dependencies between rules, ensuring that rules are executed in the correct order, avoiding execution errors. By detecting the cyclic dependence, the risk of rule design is reduced, and the failure rate in the production environment is reduced.

Step S105, analyzing the input-output dependency relationship of each candidate rule in the candidate rule set, marking conflict rule pairs, and obtaining a conflict early warning list;

Where conflicting rules refer to multiple rules modifying the same field without dependencies, which may lead to contradictions in data results, according to the method, through traversing the dependency relationship among the rules, the input and output of the rules are analyzed, all rule pairs which possibly have conflict are identified, and a conflict early warning list is obtained.

Illustratively, in a logistics system, rule R301 (time out labeled "delay") modifies the same "logistics status" field as rule R302 (manual customer service labeled "normal") and has no dependencies, and is therefore labeled as a conflicting rule pair.

Step S106, inserting a virtual isolation layer into the directed acyclic graph according to the conflict early warning list, and identifying the binding isolation layer by the conflict rule pair to generate a path identification table;

The virtual isolation layer is a logic marking node inserted in the directed acyclic graph and is used for dividing an execution path of the conflict rule pair. After inserting the nearest common ancestor (LCA) node that is located at the conflict rule pair, it is ensured that the conflict rule is executed at a different branch. The path identification table records the isolation layer ID, conflict rule pairs, rerouting targets (e.g., manual audit interfaces or cross-system verification services).

For example, the rule R7 collides with the rule R8, the LCA node is the rule R6, the virtual isolation layer L3 is inserted after the rule R6, and a path identification table is generated, which indicates that the conflicting rules R7 and R8 need to pass the manual auditing process.

It will be appreciated that the virtual isolation layer ensures that conflicting rules do not affect the main flow, avoiding erroneous data entering downstream. The insertion management of the path identification table enables the maintainability of the system to be stronger, and reduces the response time of manual intervention in production.

Step S107, topological sorting is carried out according to the directed acyclic graph inserted into the virtual isolation layer, and a rule execution sequence is generated;

In one embodiment of the application, the topological ordering is realized based on a Kahn algorithm, wherein nodes with the ingress degree of 0 (without pre-dependency rules) are initialized, the nodes are sequentially removed, and the subsequent node ingress degree is updated until a complete sequence is generated. For multiple-input nodes (e.g., multiple rules rely on the same pre-rule), the ordering is based on traffic priority. The conflict-free path (main execution sequence) and the conflict path (branch execution sequence) are respectively ordered, the main chain is executed according to the dependent sequence, and the branches are processed asynchronously.

For example, the final ordering result is R1-R2-R4-R5 (backbone), conflicting branches R3-L3-R6 execute asynchronously, the backbone rule is assigned to the high priority compute node, and the branch rule is assigned to the low priority queue.

It can be appreciated that the topology ordering can ensure the logic correctness of the main execution chain, and the conflict branch asynchronous processing avoids blocking the main flow.

Step S108, cleaning the original data stream according to the rule execution sequence, and attaching a track label to each piece of data, wherein the track label comprises a rule path identifier, a field modification history and an isolation layer identifier;

In particular, during a data cleansing process, track tags are key metadata used to track data streams. The rule path identification is generated by a hash algorithm and is generated based on the rule execution sequence dynamic generation of the topological ordering, so that the uniqueness and tamper resistance of the execution chain are ensured. The field modification history is a modification record for storing fields in a key value mode, and comprises an original value, a modification value, an execution rule ID and a time stamp, wherein the modification is updated in an increment mode through a version control mechanism, and data rollback and history version comparison are supported.

In addition, when data enters the virtual isolation layer, the isolation layer identification and the current processing state (such as L3→routing) are injected, and are bound with conflict rule pairs in the path identification table, so that subsequent rerouting decisions are triggered.

Step S109, a data execution channel is allocated according to a path identification table, wherein when the isolation layer identification in the track label is detected to be matched with a conflict early warning list, data rerouting is triggered;

The path identification table is used for recording the mapping relation between the virtual isolation layer and the conflict rule pair, and when the matching condition is met, the rerouting of the data is triggered according to a preset routing strategy in the path identification table, namely, a data copy is created and sent to a designated channel (such as manual audit), and the main thread caches intermediate state waiting results.

For example, after executing R1 (format verification) to R2 (wind control grading) on certain order data, the track label is [ R1, R2], if the track label enters the isolation layer L3, the copy is sent to manual verification, the label is updated to [ R1, R2, L3- & gt pending ], and after the manual verification, the track label continues to execute R6.

It will be appreciated that the track tags support data blood-edge tracking, helping to locate the source of the cleaning error. The rerouting mechanism ensures that conflicting data does not pollute the clean stream and the quality of the finally output data is guaranteed.

Step S110, clean data stream after cleaning is output and cleaning log containing regular execution path.

Specifically, the data flow of any conflict rule pair is not triggered in the cleaning process, and cleaning is carried out according to the topology sequence rule by rule, so that output data is ensured to accord with the preset quality standard. The purge log records the final rule execution path and field modification history for downstream system parsing.

Specifically, the purge log contains execution metadata (data ID, purge start-stop time, resource consumption), rule execution path details (success/failure rule list, conflict handling record), performance metrics (single rule time consuming, cross-system call delay).

In the embodiment, a high-efficiency and reliable data cleaning system is formed through multi-source data fusion, rule-dependent modeling, conflict isolation and topology execution. According to the technical scheme, when multi-source data are faced, the relation among different systems can be flexibly and accurately processed, rule conflicts can be effectively detected and processed, interference of conflict rule pairs is reduced, data quality is ensured, data cleaning throughput and efficiency are improved, and therefore accurate and stable data support is provided for various service systems.

Referring to fig. 2, as an embodiment of step S102, the step of extracting the multi-source data features in the original data stream includes:

step S201, performing dynamic distribution statistics on the structured numerical data to generate data distribution characteristics;

wherein the structured numeric data is typically data in tabular form, comprising a plurality of numeric fields. In this process, the system performs a dynamic statistical analysis of the distribution of each of the numeric fields. This means that the system will calculate statistical features of these data fields, such as mean, variance, skewness, kurtosis, maximum, minimum, etc.

It can be understood that the purpose of dynamic distribution statistics is to analyze the distribution characteristics of data, and understand the central tendency, the degree of dispersion, and whether abnormal values exist in the data. Through these statistical features, the system can identify potential problems in the data (e.g., deviant distributions or extreme outliers) and provide basis for subsequent data cleaning and processing.

Step S202, extracting semantic vector features from text data;

Where the text data is typically unstructured, such as log files, comments, articles, and the like. Text data may be converted to semantic vectors using pre-trained language models (e.g., BERT, word2Vec, etc.) using Natural Language Processing (NLP) techniques. Semantic vector features can capture deep semantic information in text, not just simple features based on word frequency. Through this step, the text data may be converted from abstract, unstructured content into digitized features that are available for computer processing.

Step S203, analyzing the binary file and extracting metadata features;

Wherein binary files (e.g., images, audio, video files, compressed files, etc.) contain structured binary data that cannot be used directly for conventional data analysis. Metadata features in these files may be extracted by specialized parsing tools (e.g., parse the EXIF data of the image, parse the metadata of the audio, etc.). Metadata typically includes information about the date of creation, size, format, resolution, author of the file, manner of encoding, etc. These metadata help understand the basic nature of binary data and provide key information for subsequent processing and analysis.

Step S204, multi-source data features are obtained according to the data distribution features, the semantic vector features and the metadata features.

Wherein, a comprehensive multi-source data feature set is generated by fusing the distribution features of the structured data, the semantic features of the text data and the metadata features of the binary data. The combination of the multi-source data features can strengthen the connection between the data and improve the intelligence and accuracy in the data cleaning process.

In the embodiment, a multi-source data characteristic set is formed, rich information is provided for subsequent data cleaning, anomaly detection and pattern recognition, the accuracy and the efficiency of data processing are improved, diversified heterogeneous data can be intelligently processed, and the comprehensiveness and the high quality of data cleaning are ensured.

Referring to fig. 3, as an embodiment of step S105, the step of analyzing the input-output dependency relationship of each candidate rule in the candidate rule set, marking the conflict rule pair, and obtaining the conflict pre-warning list includes:

step S301, receiving a candidate rule set, analyzing an input field set and an output field set of each candidate rule, and generating a rule input-output field table;

Where each rule is typically composed of conditions (input fields) and actions (output fields), these fields need to be extracted and categorized. The input field set refers to the condition fields required for the rule to trigger, and the output field set is the data fields that are changed or generated after the rule is applied. In the parsing process, a hybrid parsing engine is adopted, and a syntax tree parsing module and a semantic entity recognition module are combined, so that the structural and semantic understanding of rule conditions are ensured.

It can be understood that by generating the rule input/output field table, the system can clearly identify the dependent field and the target field of each rule, and lay a foundation for the subsequent construction of rule dependency graphs and conflict detection. The technical effect of this step is to provide accurate field mapping for rule management in the data cleaning process, so that subsequent rule dependency and conflict analysis is efficiently supported.

Step S302, constructing a rule dependency graph based on an input/output field table;

Wherein the rule dependency graph is a directed graph, nodes represent candidate rules, and edges represent explicit or implicit dependencies between rules. Explicit dependencies refer to rules A that explicitly depend on rules B when their output fields are used as input fields for rules B, and implicit dependencies are based on business logic constraints, where execution of certain rules must be performed in order. The establishment of this dependency helps the system understand the relationships between rules and provides support for subsequent collision detection and priority scheduling.

Illustratively, assume that the output field of rule A is a "suspicious transaction token," and the input field of rule B also includes a "suspicious transaction token. If rule A is executed first, rule B will rely on the output field of rule A, then in the rule dependency graph we will add an explicit dependency edge from rule A to rule B.

Step S303, traversing the rule dependency graph, detecting rule pairs with intersections and no dependency paths in output fields, and marking the rule pairs as conflict rule pairs;

The system traverses the rule dependency graph, analyzes the dependency relationship among the rules, and checks whether a rule pair of output field intersection exists. If there is an intersection of the output fields of two rules and there is no dependency path between the two rules (i.e., they are independent of each other), it is stated that the two rules may conflict and need to be marked as conflicting rule pairs. For example, when both rules attempt to modify the same field without performing a dependency on the order, data inconsistencies or uncertainties may result.

Step S304, determining the confidence level of conflict rule pairs according to cross-system field association features in the data feature fingerprints;

Wherein the system determines a confidence level of the conflict rule pair based on cross-system field association features in a data feature fingerprint (DFP). Data feature fingerprints evaluate the actual impact between pairs of conflict rules by modeling the strength of association between cross-system fields. For example, two rules may modify related fields in different systems, with greater risk of collision if their cross-system association strength is high, and conversely, with less likelihood of collision if the cross-system association between them is weak.

For example, assuming that rule A and rule B modify fields in two different business systems, respectively, but there is a strong cross-system correlation between the two fields (e.g., order status and payment status of the same user), the system will promote confidence in conflicting rule pairs based on this correlation.

Step S305, the corresponding preset routing strategies are matched according to the confidence coefficient of each conflict rule pair, and a conflict early warning list is generated.

Wherein, the conflict pre-warning list lists all conflict rule pairs and the confidence thereof. And then, the system can select a proper processing mode according to a preset routing strategy template, and allocate the conflict rule pair to a specific judging interface, and the routing strategy template decides whether to process the conflict rule through modes of manual auditing, rerouting and the like according to the confidence level and service requirement of the conflict rule pair.

For example, assuming that the confidence of conflicting rules for A and B is high, the system will route them to the manual review interface, while rules with low conflicting confidence go directly into the automated process flow.

In the embodiment, the rule dependency graph, the cross-system field association characteristic and the confidence adjustment mechanism are comprehensively used, so that the rule conflict problem in the data cleaning process is effectively identified and processed. Through the dependency relationship among the clear rules, potential conflicts are found in time, the confidence coefficient of the conflicts is adjusted, the processing flow is optimized according to a preset routing strategy, and finally an efficient and intelligent conflict early warning list and a processing path are generated, so that the accuracy, the processing efficiency and the flexibility of data cleaning are improved.

Referring to fig. 4, as an embodiment of step S106, inserting a virtual isolation layer in the directed acyclic graph according to the collision pre-warning list, and identifying the collision rule pair binding isolation layer, the step of generating the path identification table includes:

step S401, positioning the nearest public ancestor node of the conflict rule pair according to the directed acyclic graph and the conflict early warning list;

Wherein a nearest common ancestor (LCA) node of a conflicting rule pair is located by analyzing rule dependencies in a Directed Acyclic Graph (DAG). The LCA node is the last common node between two rules in the DAG, meaning that the two rules have a common execution path from that node.

In some embodiments, a Breadth First Search (BFS) algorithm may be employed to traverse, identify common paths between conflicting rule pairs, and find their LCA nodes. LCA positioning is a key step in resolving rule conflicts because all conflicting rules depend on the path from the LCA node, and a virtual isolation layer will be inserted after the LCA node to ensure that rule conflicts are isolated.

For example, assume that there is a conflict between rule A and rule B, and that they share a common node C as the LCA node in the DAG. Then the dependent paths of the conflict rules to a and B will be marked as paths from node C after LCA positioning.

Step S402, inserting a virtual isolation layer node after the nearest public ancestor node points to the outgoing edge of the conflict rule pair, and generating an updated directed acyclic graph;

wherein once the LCA node is determined, the system will modify the dependency path between the LCA node and the conflict rule pair. Specifically, the system may sever the LCA node to the original edge of the conflict rule pair and insert a virtual barrier node between the LCA node and the conflict rule pair. The virtual isolation layer is a logical node that separates conflict rules, ensuring that they do not share the same execution path, thereby avoiding conflicts.

For example, assuming that the LCA node is C, the collision rules a and B depend on the output path of node C, respectively. In this case, the system cuts off the original dependent edges of C through A, B and inserts a virtual barrier node L between the C node and A, B, creating a new branch path C→L→ A, C →L→B.

Step S403, injecting isolation layer identification, conflict rule pairs and bypass copy generation marks for the virtual isolation layer nodes;

After the virtual isolation layer node is inserted, the system needs to inject an isolation layer ID, a conflict rule pair and a bypass copy generation mark for the node. These identifiers ensure the uniqueness and traceability of the barrier layer. The isolation layer ID is a unique identification of the isolation layer and is used for tracking the execution path of the layer, conflict rules are used for recording rules isolated in the isolation layer, and bypass copy generation marks indicate whether a copy needs to be generated and processed through bypass. These labels are critical to subsequent routing decisions, data processing, and isolation management.

Illustratively, assuming a virtual barrier L is inserted in the DAG, the system assigns a barrier ID (e.g., "L1") to the L node and records the execution of conflict rules A and B in that barrier. If the L node triggers copy generation, the bypass copy flag indicates that the data needs to take another path to process.

Step S404, generating a path identification table according to a preset routing strategy and cross-system association characteristics of the data characteristic fingerprints in the conflict early warning list.

The system generates a path identification table according to a preset routing strategy in the conflict early warning list and cross-system association features in a data feature fingerprint (DFP). The path identification table contains the barrier ID of the conflict rule pair, the rerouting target, and the cross-system check rule.

Specifically, the routing policy decides whether the data needs to be manually checked or checked through cross-system service according to the confidence level of the conflict rule, and the cross-system association feature matrix provides evaluation of inter-system field association strength, and if the strength exceeds a preset threshold, the cross-system check service is triggered.

For example, assuming that the confidence of conflict rules A and B is high and the cross-system association strength between them is also high, the system will route it to the manual audit interface and mark "manual audit" in the path identification table as a goal. If the association strength is low, the system may pass the data directly to the automated processing path.

In the embodiment, the root of the conflict rule is precisely found based on LCA positioning, the virtual isolation layer is inserted into the proper position, the conflict path is successfully isolated, the data can be ensured to flow according to the proper path through the analysis of the intelligent routing strategy and the cross-system characteristics, and the problems of incomplete data conflict isolation, poor cross-system consistency and the like are effectively solved.

Referring to fig. 5, as an embodiment of step S107, the step of generating a rule execution sequence by topologically ordering according to the directed acyclic graph after inserting the virtual isolation layer includes:

step S501, node labels are injected into the directed acyclic graph after the virtual isolation layer is inserted according to the path identification table, wherein the node label types comprise main chain nodes, branch nodes and virtual isolation layer nodes;

Wherein a label is injected for each node in a Directed Acyclic Graph (DAG) after insertion of a virtual isolation layer to distinguish between different types of nodes. The label includes a main chain node, a branch node, and a virtual isolation layer node. The main chain node represents a rule node independent of the virtual isolation layer, the branch node represents a rule node branched by the isolation layer, and the virtual isolation layer node itself represents an isolated logic layer. The path identification table helps the system to know the execution state of each node and the conflict path of the rule, thereby providing input for subsequent rule execution sequence generation.

Step S502, initializing a high priority queue and a low priority queue according to the node label type, and sequencing nodes in the queues based on a service priority policy;

The system initializes two priority queues, namely a high priority queue and a low priority queue according to the types of the node labels (main chain node, branch node and virtual isolation layer node). Backbone nodes typically have higher traffic priorities because they are independent of other nodes, while branch nodes and virtual isolation layer nodes are given lower priorities. The nodes in the queue are ordered according to the service priority strategy (such as time requirement, calculation complexity and the like), so that the priority execution of the main chain node is ensured, and the branch nodes are processed subsequently according to the service requirement.

Specifically, the initialization step comprises the steps of screening a main chain node with the ingress degree of 0 to add a high-priority queue, and screening a branch node with the ingress degree of 0 to add a low-priority queue.

Step S503, traversing the high priority queue and the low priority queue in a layering way to generate a main execution sequence and a branch execution sequence;

The system extracts main chain nodes from the high-priority queue in a hierarchical traversal mode and adds the main chain nodes into a main execution sequence. When the high priority queue is empty, the system will extract the branch node from the low priority queue and add it to the branch execution sequence. During traversal, the system updates the degree of entry of the subsequent nodes, and determines whether to add the nodes to the low-priority queue according to the dependency relationship of the nodes (particularly, for the branch nodes, if the dependent nodes are already executed, their degree of entry is reset to zero, and the system adds the nodes to the low-priority queue to wait for execution).

Step S504, according to the main execution sequence and the branch execution sequence, a rule execution sequence is obtained.

In the embodiment, based on the injection of the node labels, the system can clearly identify the main chain nodes and the branch nodes and reasonably schedule the resources according to the priority strategy, and meanwhile, the hierarchical traversal ensures that the rules are executed according to the priority, so that the execution flow of the complex rules is more stable and efficient.

Referring to fig. 6, as an embodiment of step S503, the step of hierarchically traversing the high priority queue and the low priority queue to generate the main execution sequence and the branch execution sequence includes:

Step S601, circularly extracting main chain nodes in a high-priority queue to add a main execution sequence;

The system extracts the nodes in the high-priority queue through circulation and adds the nodes to the main execution sequence. Backbone nodes are preferentially executed because they typically do not rely on execution by other nodes, representing the backbone of the flow. Therefore, the main chain node in the high-priority queue is processed first, so that the most core part in the business process can be guaranteed to finish preferentially.

Illustratively, assume that the nodes in the high priority queue are A, B, C and that these nodes are all backbone nodes. The system extracts the main chain nodes from the queue in turn, and adds them to the main execution sequence in order. For example, rule A is extracted and added to the main execution sequence, after which rules B and C are added in sequence.

Step S602, when the high priority queue is empty, extracting the current branch node in the low priority queue to add the branch execution sequence;

When all execution of the main chain nodes in the high-priority queue is completed, the system extracts branch nodes in the low-priority queue and adds the branch nodes to the branch execution sequence. Low priority queues typically contain branch nodes that are more complex in dependencies or require less resources, so they execute after the backbone node processing is complete.

Step S603, traversing all outgoing edges of the current branch node, and screening the subsequent nodes with the node label type of the branch;

After the current branch node is extracted and added into the branch execution sequence, the system traverses all outgoing edges of the node, and screens out the subsequent nodes. These successor nodes need to be checked for their type, in particular to determine if the node tag type is a branch node. If the successor nodes are of a branch type, they will become the next branch nodes to be processed, continuing to add to the low priority queue waiting for execution.

Step S604, updating the degree of ingress of the subsequent node, and adding the subsequent node with the degree of ingress zeroed and the node label type of the branch into the low-priority queue.

Wherein, the system updates the ingress of the selected successor nodes, and reduces the ingress of all successor nodes by 1 each time a node is executed. When the degree of admission of a certain subsequent node is zero and the node is a branch node, the system adds the node into a low-priority queue to wait for subsequent execution. The invasiveness zeroing means that all the precursor nodes of the node are completed, and the node can execute.

For example, assume rule C is executed and the outgoing edge of rule C points to rule D and rule E. Let rule E be 1 and rule D be 2. When rule C is executed, the importances of rules D and E are respectively reduced by 1. Assuming rule D reduces to 0 and rule D is a branching node, rule D will be added to the low priority queue waiting for processing.

It will be appreciated that by doing the incoming updates, the system is able to ensure that nodes are executed in order according to dependencies. The judgment of zero entering degree can avoid deadlock or wrong execution sequence, and ensure that all the dependency relationships are correctly processed, thereby effectively organizing and optimizing the execution process.

In the embodiment, the system can efficiently manage the execution sequence and the priority of the nodes, ensures the priority execution of the main chain node and the timely processing of the branch nodes after the main chain node is executed, flexibly schedules the tasks according to the dependency relationship and the resource state of the nodes, optimizes the resource use, avoids unordered execution and conflict, and improves the parallelism and the efficiency of the task execution.

Referring to fig. 7, as a further embodiment of the data cleansing method, after the step of outputting the cleansing data stream and the associated cleansing log after cleansing, further includes:

Step S701, based on rule execution success rate and business feedback data in the cleaning log, eliminating rules with execution success rate lower than a preset success rate threshold and correcting feature extraction parameters;

And evaluating the validity of each rule by analyzing the rule execution success rate and the business feedback data recorded in the cleaning log. If the execution success rate of a rule is below a preset success rate threshold, the rule is considered to be inefficient and may need to be eliminated or adjusted. Eliminating the low-efficiency rules can avoid resource waste, and ensures that the rules in the rule base can effectively push the business flow. Meanwhile, by combining the business feedback data and the cleaning log, the system can also correct the characteristic extraction parameters, so that the accuracy and the efficiency of rule execution are improved.

Specifically, the success rate of executing the rule is calculated by the number of success and failure times in the execution log, and the service feedback data is usually derived from the result of the actual service, and reflects the actual effect after executing the rule. If the rules often fail or fail to achieve the desired effect, the system will optimize the rules based on this data. Correcting the feature extraction parameters refers to adjusting the conditions or algorithms used for data extraction in the rules to improve the performance of the rules.

Step S702, encoding the historical rule execution path into a rule gene library capable of genetic optimization, and iteratively updating the rule library version.

And converting the historical rule execution path into a rule gene library capable of genetic optimization, and iteratively updating the rule library version. By analyzing the historical rule execution paths, the system can identify the optimal execution paths and rule combinations, and the historical paths are coded into a 'gene' form, so that the rule base has the 'genetic optimization' capability, namely, the rule execution process is continuously optimized by simulating natural selection and genetic algorithm.

Specifically, in the process of encoding the execution path of the rule into the gene, the system extracts information such as key parameters, execution sequence, dependency relationship and the like executed by each rule, and then stores the information in a form of encoding the gene. These rule genes may be continuously evolved in subsequent executions through the optimization process of genetic algorithms to find the optimal rule combinations and execution sequences.

Illustratively, assuming that the rule execution path is rule a- > rule B- > rule C, in the genetic algorithm, each rule and its execution order will be encoded as one gene. With optimization of the execution path, the system may find a better path, for example, the order of rule B and rule C may be exchanged to improve the overall execution efficiency. Through genetic algorithm, the system phase out unsuitable rule combination and reserves optimal path.

In the above embodiment, in the rule execution process, the system can eliminate the inefficient rule by cleaning the success rate of log monitoring rules and continuously correct the feature extraction parameters, thereby ensuring the high efficiency and accuracy of the rule base. Meanwhile, the historical rule execution path is converted into a genetic optimizing gene library, and the genetic algorithm is utilized to carry out iterative updating of the rules, so that the rule library has self-optimizing capability. The technical scheme improves the self-adaptive capacity of the system in complex data processing and service execution, and ensures that the rule base always maintains a high-efficiency, accurate and sustainable optimized state in continuously changing service demands.

The embodiment of the application also discloses a data cleaning engine driven based on the rule data.

the path identification table generation module is used for inserting a virtual isolation layer into the directed acyclic graph according to the conflict early warning list, and binding the conflict rule pair with the isolation layer identification to generate a path identification table;

The data cleaning engine based on rule data driving in the embodiment of the application can realize any one of the data cleaning methods, and the specific working process of each module in the data cleaning engine can refer to the corresponding process in the embodiment of the method.

In several embodiments provided by the present application, it should be understood that the methods and systems provided may be implemented in other ways. For example, the system embodiments described above are merely illustrative, e.g., the partitioning of a module is merely a logical function partitioning, and there may be additional partitioning in actual implementation, e.g., multiple modules may be combined or integrated into another system, or some features may be omitted, or not performed.

The embodiment of the application also discloses computer equipment.

Computer apparatus comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing a rule data driven data cleansing method as described above when executing the computer program.

The embodiment of the application also discloses a computer readable storage medium.

A computer readable storage medium storing a computer program capable of being loaded by a processor and performing any one of the methods of rule data driven based data cleansing as described above.

Wherein the computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device, the program code contained on the computer readable medium can be transmitted over any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

In the foregoing embodiments, the descriptions of the embodiments are focused on, and for those portions of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.

The foregoing description of the preferred embodiments of the application is not intended to limit the scope of the application in any way, including the abstract and drawings, in which case any feature disclosed in this specification (including abstract and drawings) may be replaced by alternative features serving the same, equivalent purpose, unless expressly stated otherwise. That is, each feature is one example only of a generic series of equivalent or similar features, unless expressly stated otherwise.

Claims

1. A data cleansing method based on rule data driving, the data cleansing method comprising:

2. The method for rule-based data driven data cleansing of claim 1 wherein said raw data stream comprises structured numeric data, text data, or binary files, and wherein the step of extracting multi-source data features from said raw data stream comprises:

extracting semantic vector features from the text data;

analyzing the binary file and extracting metadata features;

3. The rule data driven based data cleansing method of claim 1 wherein the step of analyzing the input-output dependency of each candidate rule in the set of candidate rules, marking pairs of conflicting rules, and obtaining a list of conflicting pre-warnings comprises:

Constructing a rule dependency graph based on the input/output field table;

4. The method for cleaning data based on rule data driving according to claim 3, wherein the step of inserting a virtual isolation layer in the directed acyclic graph according to the collision pre-warning list, binding the collision rule pair to an isolation layer identification, and generating a path identification table comprises:

5. The method for rule-based data driven data cleansing of claim 4, wherein, performing topological sorting according to the directed acyclic graph after the virtual isolation layer is inserted, and generating a rule execution sequence comprises the following steps:

6. The method of claim 5, wherein hierarchically traversing the high priority queue and the low priority queue to generate a main execution sequence and a branch execution sequence comprises:

7. A method of data cleansing based on rule data driving according to any one of claims 1 to 6, further comprising, after the step of outputting the cleansing data stream and associated cleansing log after cleansing is completed:

8. A rule data driven based data cleansing engine, the data cleansing engine comprising:

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1 to 7 when the program is executed.

10. A computer-readable storage medium, characterized in that a computer program is stored that can be loaded by a processor and that performs the method according to any one of claims 1 to 7.