CN119862531A - Cross-domain data integration fusion method, terminal and storage medium based on large model - Google Patents
Cross-domain data integration fusion method, terminal and storage medium based on large model Download PDFInfo
- Publication number
- CN119862531A CN119862531A CN202411939614.6A CN202411939614A CN119862531A CN 119862531 A CN119862531 A CN 119862531A CN 202411939614 A CN202411939614 A CN 202411939614A CN 119862531 A CN119862531 A CN 119862531A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- domain
- integration
- standard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明提供一种基于大模型的跨域数据集成融合方法、终端及存储介质，所述方法包括：通过统一数据接入适配层对跨域原始数据进行格式转换和清洗，输出标准化的数据集；利用大模型进行语义理解集成，解析字段命名并结合业务术语知识库生成字段语义标签，通过计算相似度判定业务关联性，生成域间字段映射关系；基于所述映射关系执行技术标准转换和数据融合处理，并采用质量治理标准库与大模型相结合的方式进行智能化质量治理。本发明通过引入大模型辅助数据理解和治理，有效解决了现有跨域数据集成中语义理解不足、标准不统一等技术问题，显著提升了数据集成的准确性和自动化水平。
The present invention provides a cross-domain data integration and fusion method, terminal and storage medium based on a large model, the method comprising: performing format conversion and cleaning of cross-domain raw data through a unified data access adaptation layer, and outputting a standardized data set; using a large model for semantic understanding and integration, parsing field naming and generating field semantic labels in combination with a business terminology knowledge base, determining business relevance by calculating similarity, and generating field mapping relationships between domains; performing technical standard conversion and data fusion processing based on the mapping relationship, and performing intelligent quality management by combining a quality management standard library with a large model. The present invention effectively solves technical problems such as insufficient semantic understanding and inconsistent standards in existing cross-domain data integration by introducing a large model to assist in data understanding and management, and significantly improves the accuracy and automation level of data integration.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a large-model-based cross-domain data integration method, a terminal, and a storage medium.
Background
With the deep advancement of digital transformation, enterprises face increasing demands for cross-department, cross-system and cross-platform data integration and fusion. The cross-domain data integration refers to a technology for unified management and fusion utilization of heterogeneous data distributed in different data domains (such as different service systems, different departments and different organizations). In recent years, the rapid development of large model technology provides a new technical means for solving the complex problem in cross-domain data integration, and is hopeful to break through the limitation of the traditional method.
However, the existing cross-domain data integration scheme has the following key technical difficulties:
First, data semantics are not sufficiently understandable. The prior art relies mainly on predefined mapping rules and human experience for field matching, which makes it difficult to accurately understand and handle semantic differences between different data fields. In practical application, because of the diversity and complexity of service terms, the rule-based mapping method often requires a great deal of manual intervention, which reduces efficiency and makes it difficult to ensure the accuracy of mapping.
Second, the problem of non-uniform technical standards is highlighted. Different data fields often adopt different data formats, coding standards and technical specifications, and the consistency in the data integration process is difficult to ensure due to the heterogeneity. In the data conversion and fusion process, due to lack of unified standard specification, problems such as data distortion, format errors and the like are easy to occur, and the quality of data integration is affected.
Third, data quality management lacks intelligence. The traditional data management method mainly relies on a static rule base to carry out quality inspection and repair, and the method cannot effectively solve the newly-appearing data quality problem. With the increase of the data scale and complexity, the type of quality problems is evolving continuously, and the fixed governance rules are difficult to meet the governance requirements of dynamic changes.
Based on the above analysis, a need exists for a cross-domain data integration solution that can provide intelligent semantic understanding, unified technical standard conversion, and adaptive quality control. Aiming at the technical difficulties, the invention provides an innovative technical scheme based on a large model.
Disclosure of Invention
The invention aims to provide a large-model-based cross-domain data integration fusion method, which is used for solving the technical problems of insufficient semantic understanding, non-uniform technical standard, lack of intellectualization in quality control and the like in the prior art and improving the accuracy and automation level of cross-domain data integration.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
A cross-domain data integration fusion method based on a large model comprises the following steps:
The method comprises the steps of accessing integration, receiving cross-domain original data, performing format conversion and cleaning through a unified data access adaptation layer, and outputting a standardized data set, wherein the standardized data set adopts a unified naming format, the data type is mapped into a type supported by a system, the data content is complete and has no repeated value, and the missing value marks are unified;
Semantic understanding integration, analyzing the characteristics of a standardized data set in each data domain based on a large model, extracting field data characteristics in each domain, identifying the service meaning of fields in each domain by analyzing field naming and combining the service term knowledge base, generating field semantic tags, judging service relevance by calculating field name similarity and combining the field semantic tags, and generating field mapping relations between domains;
Technical standard integration, which is to execute unified technical standard conversion on the standardized data set based on the field semantic tag, wherein the unified technical standard conversion comprises unified data format specification and coding standard, and standardized data conforming to the unified technical specification is output;
Data fusion processing, namely, based on standardized data of the unified technical specification and the field mapping relation between domains, identifying data records of the same entity crossing domains and matching the data records, and carrying out merging processing on attributes of the matched entities according to the data domain authority relation to generate fused entity data;
And performing quality control, namely performing problem feature matching and repairing on the fused entity data based on a quality control standard library, performing standard repairing rules on successfully matched quality problems, analyzing unmatched problems by adopting the large model, identifying quality problems of entity attribute value consistency and entity relationship correctness, generating and performing repairing rules, updating the quality problems and a processing method to the quality control standard library, and outputting treated high-quality data.
The large model is a language model which is pre-trained by mass data and has cross-domain knowledge understanding capability, and can carry out intelligent analysis on data types, format specifications and business rules;
The further technical scheme is that the large model processing in the semantic understanding integration step specifically comprises the following steps:
Extracting field data characteristics in each field, classifying the field data into numerical value type, character type, date time type and Boolean type through analysis of the field data, and determining the effective range of each type field according to statistical distribution;
analyzing and semanteme identifying field names in each domain, and generating a field semantic tag containing a data domain, a service type and field attributes by combining the service term knowledge base;
generating field mapping relation among domains, calculating field name similarity among different data domains based on an edit distance algorithm, judging business association relation among fields among different data domains based on the field semantic label, converting the field name similarity and the business association degree into feature vectors, and determining field mapping relation among domains by calculating Euclidean distance between the field mapping feature vectors to be judged and standard mapping sample feature vectors in a verified field mapping set.
The method is characterized in that the following basic configuration is required to be completed before the method is executed:
Constructing a business term knowledge base which comprises standard term definitions, field naming rules and business mapping relations;
Establishing a verified field mapping set for guiding new field mapping;
configuring a technical specification conversion benchmark library, wherein the technical specification conversion benchmark library comprises numerical specification, text specification, time specification and classification specification;
Setting a priority order among data domains for conflict processing during data attribute combination;
Initializing a quality control standard library, wherein the quality control standard library comprises a problem feature pattern library, a standard repair rule library and a problem-rule mapping relation table, the problem feature pattern library is used for identifying data consistency, integrity, accuracy and relevance problems, the standard repair rule library comprises processing rules corresponding to problem features, and the problem-rule mapping relation table is used for realizing quick matching of quality problems and processing methods.
The further technical proposal is that the technical standard integration steps specifically comprise:
initializing a conversion environment, and reading numerical value specifications, text specifications, time specifications and classification specifications in the technical specification conversion reference library;
Retrieving and determining target conversion specifications of each field from the technical specification conversion reference library based on the field semantic tags;
and performing data format conversion operation according to the determined target specification.
The further technical proposal is that the data fusion processing steps specifically comprise:
Based on the field semantic tags, analyzing service attributes in each data field, and identifying a field with unique identification features, wherein the unique identification features refer to constraint features of which field values are not repeated;
associating unique identification fields through the inter-domain field mapping relation, generating an entity identification field mapping set, and executing matching of cross-domain data records based on the mapping set;
Combining attribute values of different data records of the same entity based on the field mapping relation and the data domain responsibility relation, and adopting effective values of other domains according to the priority of the data domain when the attribute values in the responsibility data domain are absent;
And integrating the combined attribute values to form the integrated entity data.
The further technical proposal is that the quality control step comprises the following steps:
Performing problem feature matching on the fused entity data, judging whether quality problems conforming to the problem features recorded in the quality control standard library exist or not, and executing a repair rule in the standard library for the quality problems of feature matching;
For quality problems which are not covered in a standard library, analyzing the fused entity data by adopting the large model, converting the consistency of the identification attribute values of the standard library in terms of numerical precision, character coding, time format and classification codes based on the technical specification, verifying the relation rule among entities, and outputting a quality problem list;
Generating and executing a repair rule containing problem positioning conditions and processing operations based on the field semantic tags and the inter-domain field mapping relation by adopting the large model;
And updating the quality problems and the processing method thereof to a quality control standard library for guiding subsequent quality control work.
The invention also provides a terminal device for realizing the method. The terminal device comprises a memory, a processor and a computer-readable storage medium, wherein the memory is used for storing computer program codes containing program instructions, the program instructions are used for realizing the large-model-based cross-domain data integration and fusion governance method, the processor is connected with the memory through a system bus and used for calling and executing the program instructions, and the computer-readable storage medium is connected with the processor and used for storing execution results of the method and intermediate data required in the execution process of the method. When the program instructions are executed by the processor, the processor performs the steps of a large model-based cross-domain data integration and fusion remediation method.
The beneficial effects of the invention are as follows:
(1) The invention is based on a semantic understanding integration mechanism of a large model, and generates the field semantic tags by intelligently analyzing field naming and combining a business term knowledge base, thereby realizing intelligent mapping of the cross-domain data fields, solving the problems of dependence on manual experience and low efficiency of field mapping in the traditional method, and remarkably improving the accuracy and efficiency of data integration.
(2) The technical standard integration scheme designed by the invention realizes the standardized processing of cross-domain heterogeneous data through the unified data access adaptation layer and the technical specification conversion reference library, solves the problem of non-uniform standards among different data domains, and effectively ensures the consistency and standardization in the data integration process.
(3) The invention creatively combines the quality control standard library with the large model, builds a self-adaptive quality control mechanism, can automatically identify and repair the data quality problem, continuously update the control rules, solve the problems of lack of intellectualization and poor expandability of the traditional quality control method, and realize continuous optimization of data control.
Drawings
FIG. 1 is a flow chart of an embodiment of a large model-based cross-domain data integration and fusion remediation method.
Fig. 2 is a semantic understanding integration flow chart.
Fig. 3 is a technical standard integration flow chart.
Fig. 4 is a flow chart of a data fusion process.
FIG. 5 is a schematic diagram of a closed loop process for quality control.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear and clear, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
As a possible embodiment of the present invention, before executing the core steps of the cross-domain data integration and fusion governance method, the following preset configuration needs to be completed:
SP, presetting configuration step for initializing basic environment required by system operation.
In the present invention, further, the SP step specifically includes:
SP01, constructing a business term knowledge base. Specifically, the library needs to include standard service term definition including standard terms of each service domain and definition descriptions thereof, field naming rules for specifying format standards of field names including naming methods (such as humps or underlining), naming factors and sequences thereof, service mapping relations, and definition of association rules and data flow constraints among different service domains.
SP02, establishing a verified set of field mappings. Specifically, the set is used for storing a field mapping relation passing through the history verification, including a corresponding relation between a source field and a target field, and is used as a reference sample for field mapping verification and used for evaluating the accuracy of newly built mapping.
And SP03, configuring a technical specification conversion reference library. The database comprises a numerical value specification, a text specification, a time specification, a classification specification, a standard for specifying state codes, enumeration values and the like, wherein the numerical value specification specifies precision requirements, measurement units, effective ranges and the like of numerical value types, the text specification defines rules such as character set codes, text length limitation and the like, the time specification unifies date and time formats and time zone processing modes, and the classification specification specifies standards for data such as state codes, enumeration values and the like.
SP04, setting the priority order of the data fields. Specifically, the configuration is used for defining the data management responsibility and authority of each data domain, prescribing the value priority sequence when the data conflict occurs, and determining the flow direction and rule of cross-domain data synchronization.
And SP05, initializing a quality control standard library. In particular, a complete quality control standard system needs to be established before the system operates, and the standard library is used as a basic support for quality problem identification and treatment. First, a problem feature pattern library is constructed, which covers data consistency problem features (for identifying normalization problems of attribute values in terms of format, precision, encoding, etc.), data integrity problem features (for identifying problems of missing characters to be filled, null values of key attributes, etc.), data accuracy problem features (for identifying data deviation problems of outliers, error values, etc.), and data relevance problem features (for identifying problems of entity relationship conflicts, referential integrity, etc.).
Secondly, a standard repair rule base is constructed, wherein the standard repair rule base comprises processing rules corresponding to problem characteristics, and the standard repair rule base comprises data conversion rules (used for processing problems such as format non-standardization and coding non-uniformity), data completion rules (used for processing integrity problems such as data missing and null values), data correction rules (used for processing accuracy problems such as abnormal values and error values) and relationship repair rules (used for processing relationship problems such as entity relationship conflict).
And finally, establishing a problem-rule mapping relation table, wherein the problem characteristics are associated with corresponding repair rules by the table, so that the quality problems are rapidly matched with the processing method, and rule support is provided for subsequent quality treatment work. By establishing a complete quality control standard library, the standardization and controllability of the quality control process are ensured.
In the embodiment, through perfect preset configuration, basic support is provided for subsequent data integration and fusion management. These configurations both standardize the data processing standards and provide a basic guarantee for ensuring data quality. Meanwhile, each configuration can be dynamically adjusted according to service requirements, so that the flexibility and adaptability of the system operation are ensured.
As a possible embodiment of the present invention, as shown in fig. 1, a large model-based cross-domain data integration and fusion governance method is provided, which includes the following steps:
And S100, an access integration step, which is used for receiving the cross-domain original data, performing format conversion and cleaning through a unified data access adaptation layer and outputting a standardized data set.
In the present invention, further, the step S100 specifically includes:
S101, receiving original data from different data fields. In particular, raw data may come from a number of different business systems or data sources, and the data formats, naming rules, and coding standards may all be different. For example, different business systems may employ different database types, such as Oracle, mySQL, etc., or different file formats, such as CSV, XML, etc.
S102, converting the data format through the unified data access adaptation layer. Specifically, according to the format characteristics of the original data, a corresponding adapter component is called to perform format conversion. For example, data in the relational database is read and converted through the database connection adapter, and file class data is parsed and converted through the file format adapter.
And S103, cleaning the converted data. Specifically, the cleaning process includes the operations of unified naming format, converting field names into system-specified unified format, such as hump naming, mapping data types, mapping original data types to standard types supported by the system, checking data integrity, checking and processing repeated values, and unified marking missing values.
And S104, outputting the standardized data set. Specifically, the data subjected to format conversion and cleaning processing is stored according to a predefined standard format to form a standardized data set. The data set has a uniform naming format, standard data types, complete and non-duplicate data content, and uniform missing value indicia.
In the embodiment, through setting a unified data access adaptation layer, the standardized processing of the original data with different sources and different formats is realized, and a foundation is laid for subsequent semantic understanding and data fusion. Meanwhile, the quality and consistency of the data are guaranteed and the availability of the data is improved through a standard data cleaning flow.
As a possible embodiment of the present invention, referring to fig. 2, after receiving the standardized dataset, a semantic understanding integration step is performed, specifically as follows:
And S200, a semantic understanding integration step, which is used for analyzing the characteristics of the standardized data set in each data domain based on the large model, identifying the business meaning of the field in the domain and generating the mapping relation between the fields.
In the present invention, further, the large model processing in step S200 specifically includes:
S201, extracting field data characteristics in each domain. Specifically, the field data types in each field are classified into a numerical type, a character type, a date-time type, and a boolean type. The effective range of the field data is set by carrying out statistical analysis, namely, an upper limit threshold and a lower limit threshold are set according to data distribution for a numerical field, a length range is determined for a character type field, and an effective time interval is set for a date and time type field. The feature extraction method based on the statistical distribution can effectively identify abnormal values and ensure the validity of data.
S202, performing field name resolution and semantic recognition. Specifically, the field names are analyzed according to a preset naming rule. And then matching the analysis result with standard terms in a business term knowledge base, and generating a field semantic tag through semantic recognition. The tag contains three key dimensions, a data field (identifying the service field to which the field belongs), a service type (illustrating the service purpose of the field), and a field attribute (describing the characteristic attribute of the field). This multi-dimensional semantic annotation provides a reliable semantic basis for subsequent field mapping.
S203, generating field mapping relation between domains. Specifically, the method comprises the following steps:
calculating the similarity of field names among different data domains by adopting an edit distance algorithm;
Judging the business association relationship between the fields based on the generated field semantic tags;
And converting the field name similarity and the service association degree into feature points in a vector space, and constructing a feature vector. The feature vector comprises a field name similarity component and a service semantic relevance component, and the value range of each component after normalization processing is [0,1].
S204, verifying field mapping relation. Specifically, based on the standard mapping sample of the verified field mapping set in SP02, the euclidean distance between the feature vector corresponding to the field mapping to be determined and the feature vector of the standard mapping sample is calculated. And when the calculated Euclidean distance is smaller than a preset threshold value, confirming that the field mapping relation between the domains exists.
For example, in a certain data integration scenario, the preset field mapping decision threshold is 0.1. When the euclidean distance between the field mapping feature vector <0.85,0.75> to be determined and the verified mapping sample feature vector <0.82,0.78> is calculated to be 0.058, since 0.058 is smaller than the preset threshold value 0.1, it can be confirmed that a valid mapping relationship exists between the two fields. Otherwise, if the calculated euclidean distance is 0.15, the calculated euclidean distance exceeds a preset threshold value, which indicates that no reliable mapping relation exists between the two fields.
In the embodiment, by adopting a large model to perform intelligent analysis and combining steps of feature extraction, semantic identification, mapping verification and the like, accurate understanding of field business meaning and reliable establishment of mapping relation are realized. The method not only improves the accuracy of data integration, but also lays a solid foundation for the subsequent data fusion processing.
As a possible embodiment of the present invention, referring to fig. 3, after the semantic understanding integration is completed, the technology standard integration steps are performed, specifically as follows:
And S300, a technical standard integration step, which is used for executing unified technical standard conversion on the standardized data set based on the field semantic tags to ensure the consistency of the data format specification and the coding standard.
In the present invention, further, the step S300 specifically includes:
S301, initializing a conversion environment. Specifically, the numerical value specification, the text specification, the time specification and the classification specification are loaded into a memory through the specification definition of the technical specification conversion reference library in the configuration file loading SP03, a specification index table is established, and the conversion parameters are initialized. The step improves the efficiency of subsequent standard retrieval through a caching mechanism and ensures the stability of the conversion process.
S302, determining a target conversion specification. Specifically, a target conversion specification is determined in a technical specification conversion reference library based on the data field, the service type and the field attribute information in the field semantic tag.
Firstly, constructing search conditions according to field semantic tags, sequentially matching data fields, service types and field attributes, and accurately positioning related specification definitions in a specification conversion reference library. And searching in a correct service range is ensured through data domain matching, a corresponding specification set is positioned based on the service type, and then the field attribute information is utilized to narrow the specification searching range.
And secondly, judging the applicability of the retrieved candidate specifications. The method comprises the steps of checking the matching degree of data types, ensuring that source data types can be converted into target types in a lossless mode, verifying the suitability of a data value range, ensuring that converted data meet the value requirement of target specifications, verifying the compatibility of data formats, and ensuring that the source data formats can be converted according to the specification requirements.
Finally, the determination of the target specification is completed. When a plurality of candidate specifications are present, the optimal specification is selected according to the priority order of the data fields. The determined field specification mapping relation is recorded into a conversion configuration table and used as the basis of the subsequent conversion operation. This structured specification determination procedure ensures that each field can find the most appropriate conversion specification.
And S303, performing data format conversion. Specifically, according to the determined target specification, the following conversion processing method is adopted:
The method comprises the steps of calling a numerical processing function to execute precision adjustment, unit conversion and interval normalization for numerical data, carrying out character set conversion, coding standardization and format standardization for text data by using a character string processing function, carrying out time zone conversion and standardization representation for time type data by unifying formats of date and time processing functions, and executing code conversion and enumeration value standardization processing for type data based on a mapping table.
In the embodiment, through a standardized technical standard integration step, the standard unification of the data in the technical level is realized, and a reliable basis is provided for subsequent data fusion and quality control. Meanwhile, through careful specification matching and conversion processing, the accuracy and consistency of data conversion are ensured.
As a possible embodiment of the present invention, as shown in fig. 4, after the integration of the technical standards is completed, the data fusion processing steps are performed, specifically as follows:
And S400, a data fusion processing step, which is used for identifying and matching data records of the same entity crossing the domains based on standardized data and field mapping relations among domains of the unified technical specification, and carrying out attribute merging processing.
In the present invention, further, the step S400 specifically includes:
S401, identifying a unique identification field. Specifically, the fields with unique identification features are identified based on the business attributes in each data domain being analyzed by the field semantic tags. The unique identification feature refers to a constraint feature that field values do not repeat within the data field. And determining a unique identification field set in each data domain by analyzing the value distribution characteristics and the business rule constraint of the fields.
S402, generating an entity identification field mapping set. Specifically, the method comprises the following processing steps:
Firstly, screening out the mapping relation related to the unique identification field based on the field mapping relation between the domains generated in the step S200, and secondly, verifying the screened mapping relation to ensure the consistency of the mapping field in a cross-domain scene. The verification process comprises the steps of checking format consistency of field values, verifying corresponding relation of the field values, and confirming timeliness of the field values, and finally integrating the verified mapping relation to form an entity identification field mapping set for matching of subsequent data records. The mapping set contains information such as source domain identification field, target domain identification field, mapping rule, validity period and the like.
S403, performing cross-domain data record matching. Specifically, based on the entity identification field mapping set, the following matching strategy is adopted:
For the identification field with direct mapping relation, adopting accurate matching mode, i.e. the records with identical field values are judged as identical entity, and for the identification field with format difference, firstly making format standardization according to mapping rule, then executing matching.
And S404, performing attribute value combination. Specifically, for the same entity data record with the matching confirmation, based on the field mapping relation and the data domain authority relation between domains, carrying out attribute value merging processing:
First, the master data domain of each attribute is determined according to the data domain priority order set in the SP04 step.
Next, the attribute values are processed according to the principle of "subject domain" and "subject domain. When the attribute value in the master domain is missing, the valid values in other domains are adopted according to the data domain priority order set in the SP04 step.
And finally, integrating the combined attribute values to form the integrated entity data. The data contains information such as uniform entity identification, valid attribute values from each domain, source domain identification of the attribute values, and the like.
In the embodiment, accurate fusion of cross-domain data is realized through strict unique identification field identification, entity record matching and attribute value merging processing. Meanwhile, by introducing the data domain authority relationship and the priority mechanism, the accuracy and the authority of the data fusion result are ensured. The method can effectively process the problems of entity identification and attribute combination in cross-domain data integration, and provides a reliable data base for subsequent quality control.
As a possible embodiment of the present invention, referring to fig. 5, after the data fusion process is completed, a quality improvement step is performed, specifically as follows:
And S500, a quality control step, which is used for intelligently checking and repairing the fused entity data based on a mode of combining a quality control standard library and a large model.
In the present invention, further, the step S500 specifically includes:
S501, performing problem feature matching. Specifically, feature detection is performed on the fused entity data based on the problem feature pattern in the quality control standard library initialized in the SP05 step. By matching the data features with the problem feature patterns in the standard library, quality problems such as consistency, integrity, accuracy, relevance and the like in the data are identified. And (3) for the quality problem matched with the characteristic pattern in the standard library, automatically calling the standard repairing rule preset in the SP05 step by the system to process.
And S502, analyzing the unmatched problem by the large model. Specifically, for quality problems not covered in the quality control standard library initialized in the SP05 step, the system uses a large model for intelligent analysis. First, based on the technical specification conversion reference library configured in the SP03 step, the system recognizes the consistency problem of the entity attribute values in terms of numerical accuracy, character encoding, time format, classification code, and the like. And secondly, verifying relation rules among the entities, and checking whether business constraints such as subordinate relations, mutual exclusion relations and the like among the entities are met. Through analysis of the large model, the system sorts all quality questions identified into a question list containing detailed information about the type of question, the fields involved, the degree of violation, etc.
S503, generating a repair rule. Specifically, the system adopts a large model to generate a repair rule aiming at various problems in a quality problem list based on field semantic tags and field mapping relations among fields. In the rule generation process, firstly, data characteristics and service scenes of problems are analyzed, and accurate problem positioning conditions are constructed. And secondly, designing corresponding problem processing operation according to the technical specifications and the business rules. And finally, assembling the problem positioning conditions and the processing operation to form a complete repair rule.
S504, executing repair processing. Specifically, the system processes the quality problem according to the generated repair rule. In the repairing process, firstly, the data record to be processed is screened out according to the problem locating condition, and then the repairing is carried out on the problem data according to the processing operation. The system synchronously records key information in the repairing process, including data states before and after repairing, applied rules and the like, so that traceability of the repairing process is ensured.
S505, updating a quality control standard library. Specifically, the system updates the problem features and the processing method found in the current quality control process to the quality control standard library initialized in the SP05 step. In the updating process, firstly, newly discovered quality problem features and an identification method thereof are added to a problem feature pattern library, secondly, newly generated repair rules are added to a standard repair rule library, and finally, the rules are scored and optimized according to the repair effect, so that the continuous improvement of quality control capability is realized.
In the embodiment, the intelligent quality control of the fusion data is realized by combining the quality control standard library initialized in the SP05 step with the large model. The method not only can process the known quality problem, but also can find and solve the newly-appearing quality problem through large model analysis, and simultaneously realizes dynamic optimization of the treatment capacity through continuously updating the quality treatment standard library, thereby providing effective guarantee for continuous improvement of data quality.
As a possible embodiment of the present invention, a specific device for implementing the above-mentioned cross-domain data integration and fusion management method is provided, and the technology is implemented as follows:
The apparatus includes a memory, a processor, and a computer-readable storage medium. The memory and the processor are connected through a system bus to form a data interaction path.
In the present invention, the memory may include, but is not limited to, high-speed RAM memory, nonvolatile memory (e.g., solid state disk, mechanical disk, etc.). The memory is used for storing the computer program of the invention, and the program comprises program codes for realizing all the steps of cross-domain data integration and fusion governance. Specifically, the following key program modules are provided in the memory:
(1) The data access adapter module comprises access adapter programs of various data sources and is used for realizing the data format conversion and cleaning functions in the step S100;
(2) The semantic understanding integrated module comprises a program code for calling a large model interface, and realizes field feature extraction, semantic recognition and mapping relation generation in the step S200;
(3) The technical standard conversion module comprises program codes for data standardization processing and realizes the technical standard unified conversion in the step S300;
(4) The data fusion processing module comprises program codes for entity identification matching and attribute merging, and realizes the data fusion function of the step S400;
(5) And the quality control module comprises program codes for quality inspection and repair processing and realizes data quality control in the step S500.
The processor may be a general-purpose processor (such as a CPU of Intel or AMD) or a dedicated data processing chip. The processor performs the specific functions of the steps described above by executing program code in the memory. During program execution, the processor may:
calling a data access adaptation module to finish format conversion and standardization processing of the cross-domain original data;
loading a semantic understanding integrated module, and realizing field semantic understanding and mapping through large model analysis;
Operating a technical standard conversion module, and executing unified conversion of the data specification;
starting a data fusion processing module to complete entity matching and attribute merging;
triggering a quality control module to realize intelligent checking and repairing of data quality.
The computer readable storage medium may take many forms, including magnetic disks, optical disks, solid state memories, etc., for storing the computer program described above. When loaded and executed by a processor, the program will implement all the steps of the cross-domain data integration and fusion governance method as described above. During the execution process of the program, various configuration information in the memory can be accessed, including a business term knowledge base, a technical specification conversion standard base, a quality control standard base and the like, so that the normal operation of the method is ensured.
Through the organic combination of the hardware environment and the software components, the device provided by the invention can efficiently realize intelligent integration and fusion management of cross-domain data, and provides reliable technical support for unified management of enterprise data assets.
Claims (7)
1. A large-model-based cross-domain data integration and fusion treatment method is characterized by comprising the following steps:
The method comprises the steps of accessing integration, receiving cross-domain original data, performing format conversion and cleaning through a unified data access adaptation layer, and outputting a standardized data set, wherein the standardized data set adopts a unified naming format, the data type is mapped into a type supported by a system, the data content is complete and has no repeated value, and the missing value marks are unified;
Semantic understanding integration, analyzing the characteristics of a standardized data set in each data domain based on a large model, extracting field data characteristics in each domain, identifying the service meaning of fields in each domain by analyzing field naming and combining the service term knowledge base, generating field semantic tags, judging service relevance by calculating field name similarity and combining the field semantic tags, and generating field mapping relations between domains;
Technical standard integration, which is to execute unified technical standard conversion on the standardized data set based on the field semantic tag, wherein the unified technical standard conversion comprises unified data format specification and coding standard, and standardized data conforming to the unified technical specification is output;
Data fusion processing, namely, based on standardized data of the unified technical specification and the field mapping relation between domains, identifying data records of the same entity crossing domains and matching the data records, and carrying out merging processing on attributes of the matched entities according to the data domain authority relation to generate fused entity data;
Performing quality control, namely performing problem feature matching and repairing on the fused entity data based on a quality control standard library, performing standard repairing rules on successfully matched quality problems, analyzing unmatched problems by adopting the large model, identifying quality problems of entity attribute value consistency and entity relationship correctness, generating and performing repairing rules, updating the quality problems and a processing method to the quality control standard library, and outputting treated high-quality data;
the large model is a language model which is pre-trained by mass data and has cross-domain knowledge understanding capability, and can carry out intelligent analysis on data types, format specifications and business rules.
2. The large-model-based cross-domain data integration and fusion governance method according to claim 1, wherein the large-model processing in the semantic understanding integration step specifically comprises:
Extracting field data characteristics in each field, classifying the field data into numerical value type, character type, date time type and Boolean type through analysis of the field data, and determining the effective range of each type field according to statistical distribution;
analyzing and semanteme identifying field names in each domain, and generating a field semantic tag containing a data domain, a service type and field attributes by combining the service term knowledge base;
generating field mapping relation among domains, calculating field name similarity among different data domains based on an edit distance algorithm, judging business association relation among fields among different data domains based on the field semantic label, converting the field name similarity and the business association degree into feature vectors, and determining field mapping relation among domains by calculating Euclidean distance between the field mapping feature vectors to be judged and standard mapping sample feature vectors in a verified field mapping set.
3. The large model-based cross-domain data integration and fusion governance method of claim 1, wherein the method is configured to:
Constructing a business term knowledge base which comprises standard term definitions, field naming rules and business mapping relations;
Establishing a verified field mapping set for guiding new field mapping;
Configuring a technical specification conversion reference library which comprises numerical specification, text specification, time specification and classification specification, setting a priority order among data domains and performing conflict processing when data attributes are combined;
Initializing a quality control standard library, wherein the quality control standard library comprises a problem feature pattern library, a standard repair rule library and a problem-rule mapping relation table, the problem feature pattern library is used for identifying data consistency, integrity, accuracy and relevance problems, the standard repair rule library comprises processing rules corresponding to problem features, and the problem-rule mapping relation table is used for realizing quick matching of quality problems and processing methods.
4. The large-model-based cross-domain data integration and fusion governance method of claim 1, wherein the technical standard integration step specifically comprises:
initializing a conversion environment, and reading numerical value specifications, text specifications, time specifications and classification specifications in the technical specification conversion reference library;
Retrieving and determining target conversion specifications of each field from the technical specification conversion reference library based on the field semantic tags;
and performing data format conversion operation according to the determined target specification.
5. The large-model-based cross-domain data integration and fusion governance method of claim 1, wherein the data fusion processing step specifically comprises:
based on the field semantic tags, analyzing service attributes in each data field, and identifying a field with unique identification features, wherein the unique identification features refer to constraint features of which field values are not repeated;
associating unique identification fields through the inter-domain field mapping relation, generating an entity identification field mapping set, and executing matching of cross-domain data records based on the mapping set;
Combining attribute values of different data records of the same entity based on the field mapping relation and the data domain responsibility relation, and adopting effective values of other domains according to the priority of the data domain when the attribute values in the responsibility data domain are absent;
And integrating the combined attribute values to form the integrated entity data.
6. The large model-based cross-domain data integration and fusion governance method of claim 1, wherein said quality governance step specifically comprises:
Performing problem feature matching on the fused entity data, judging whether quality problems conforming to the problem features recorded in the quality control standard library exist or not, and executing a repair rule in the standard library for the quality problems of feature matching;
For quality problems which are not covered in a standard library, analyzing the fused entity data by adopting the large model, converting the consistency of the identification attribute values of the standard library in terms of numerical precision, character coding, time format and classification codes based on the technical specification, verifying the relation rule among entities, and outputting a quality problem list;
Generating and executing a repair rule containing problem positioning conditions and processing operations based on the field semantic tags and the inter-domain field mapping relation by adopting the large model;
And updating the quality problems and the processing method thereof to a quality control standard library for guiding subsequent quality control work.
7. Cross-domain data integration and fusion treatment device based on large model is characterized by comprising:
A memory for storing program instructions comprising computer program code for implementing the large model-based cross-domain data integration and fusion remediation method of any one of claims 1-6;
The processor is connected with the memory through a system bus and is used for calling and executing the program instructions;
And the computer readable storage medium is connected with the processor and is used for storing the execution result of the cross-domain data integration and fusion treatment method and the data information required in the execution process of the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411939614.6A CN119862531B (en) | 2024-12-26 | 2024-12-26 | Cross-domain data integration fusion method, terminal and storage medium based on large model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411939614.6A CN119862531B (en) | 2024-12-26 | 2024-12-26 | Cross-domain data integration fusion method, terminal and storage medium based on large model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN119862531A true CN119862531A (en) | 2025-04-22 |
| CN119862531B CN119862531B (en) | 2025-09-23 |
Family
ID=95388569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411939614.6A Active CN119862531B (en) | 2024-12-26 | 2024-12-26 | Cross-domain data integration fusion method, terminal and storage medium based on large model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119862531B (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120124605A (en) * | 2025-05-15 | 2025-06-10 | 江西省公路科研设计院有限公司 | A method for automatically generating traffic safety facility statistics table |
| CN120316124A (en) * | 2025-04-27 | 2025-07-15 | 宜宾数字经济产业发展有限公司 | A data processing method and system for enterprise digital transformation platform |
| CN120447407A (en) * | 2025-05-06 | 2025-08-08 | 深圳信盈达科技有限公司 | Interactive control system of intelligent home appliances based on embedded software |
| CN120469991A (en) * | 2025-04-29 | 2025-08-12 | 北京大唐高鸿数字科技有限公司 | Multi-source heterogeneous data integration method and device integrating large model conversion operators |
| CN120596563A (en) * | 2025-08-05 | 2025-09-05 | 青岛大数华创科技有限公司 | Data integration and fusion method and system based on large model |
| CN120932840A (en) * | 2025-10-15 | 2025-11-11 | 恺恩泰(南京)科技有限公司 | Data asset management system and method for medical system |
| CN121071191A (en) * | 2025-11-07 | 2025-12-05 | 北京科杰科技有限公司 | Data type mapping method, related device and storage medium |
| CN121071045A (en) * | 2025-11-07 | 2025-12-05 | 北京能科瑞元数字技术有限公司 | Multi-source heterogeneous data integrated management method, medium and system based on digital main line |
| CN121144408A (en) * | 2025-11-06 | 2025-12-16 | 北京中数睿智科技有限公司 | A data weaving device that supports cross-domain data fusion analysis |
| CN121210474A (en) * | 2025-11-26 | 2025-12-26 | 中亿丰数字科技集团股份有限公司 | A Data Governance Method and System for Construction Projects Based on Multimodal and Large Models |
| CN121579582A (en) * | 2026-01-27 | 2026-02-27 | 深圳数阔信息技术有限公司 | Intelligent Field Mapping Method and System Based on Large Language Model |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853843A (en) * | 2014-03-20 | 2014-06-11 | 浪潮集团山东通用软件有限公司 | Method for realizing data concentration across security domains based on main data mapping |
| US20220207001A1 (en) * | 2020-12-31 | 2022-06-30 | International Business Machines Corporation | Cross-domain structural mapping in machine learning processing |
| CN115952770A (en) * | 2023-03-15 | 2023-04-11 | 广州汇通国信科技有限公司 | A data standardization processing method, device, electronic equipment and storage medium |
-
2024
- 2024-12-26 CN CN202411939614.6A patent/CN119862531B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853843A (en) * | 2014-03-20 | 2014-06-11 | 浪潮集团山东通用软件有限公司 | Method for realizing data concentration across security domains based on main data mapping |
| US20220207001A1 (en) * | 2020-12-31 | 2022-06-30 | International Business Machines Corporation | Cross-domain structural mapping in machine learning processing |
| CN115952770A (en) * | 2023-03-15 | 2023-04-11 | 广州汇通国信科技有限公司 | A data standardization processing method, device, electronic equipment and storage medium |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120316124A (en) * | 2025-04-27 | 2025-07-15 | 宜宾数字经济产业发展有限公司 | A data processing method and system for enterprise digital transformation platform |
| CN120469991A (en) * | 2025-04-29 | 2025-08-12 | 北京大唐高鸿数字科技有限公司 | Multi-source heterogeneous data integration method and device integrating large model conversion operators |
| CN120447407A (en) * | 2025-05-06 | 2025-08-08 | 深圳信盈达科技有限公司 | Interactive control system of intelligent home appliances based on embedded software |
| CN120124605A (en) * | 2025-05-15 | 2025-06-10 | 江西省公路科研设计院有限公司 | A method for automatically generating traffic safety facility statistics table |
| CN120596563A (en) * | 2025-08-05 | 2025-09-05 | 青岛大数华创科技有限公司 | Data integration and fusion method and system based on large model |
| CN120932840A (en) * | 2025-10-15 | 2025-11-11 | 恺恩泰(南京)科技有限公司 | Data asset management system and method for medical system |
| CN121144408A (en) * | 2025-11-06 | 2025-12-16 | 北京中数睿智科技有限公司 | A data weaving device that supports cross-domain data fusion analysis |
| CN121071191A (en) * | 2025-11-07 | 2025-12-05 | 北京科杰科技有限公司 | Data type mapping method, related device and storage medium |
| CN121071045A (en) * | 2025-11-07 | 2025-12-05 | 北京能科瑞元数字技术有限公司 | Multi-source heterogeneous data integrated management method, medium and system based on digital main line |
| CN121071045B (en) * | 2025-11-07 | 2026-03-20 | 北京能科瑞元数字技术有限公司 | A method, medium, and system for multi-source heterogeneous data integration and management based on a digital backbone |
| CN121210474A (en) * | 2025-11-26 | 2025-12-26 | 中亿丰数字科技集团股份有限公司 | A Data Governance Method and System for Construction Projects Based on Multimodal and Large Models |
| CN121579582A (en) * | 2026-01-27 | 2026-02-27 | 深圳数阔信息技术有限公司 | Intelligent Field Mapping Method and System Based on Large Language Model |
| CN121579582B (en) * | 2026-01-27 | 2026-04-03 | 深圳数阔信息技术有限公司 | Intelligent Field Mapping Method and System Based on Large Language Model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN119862531B (en) | 2025-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN119862531B (en) | Cross-domain data integration fusion method, terminal and storage medium based on large model | |
| US8346773B2 (en) | Product classification system | |
| CN112001047B (en) | Radar key part process design method based on PMI information | |
| CN118227599A (en) | Data standard treatment method, system, equipment and medium based on automatic flow | |
| US20220405235A1 (en) | System and method for reference dataset management | |
| CN113127460B (en) | Evaluation method of data cleaning frame, device, equipment and storage medium thereof | |
| CN121391167A (en) | Log analysis-based workflow analysis linkage verification method, device, equipment and medium | |
| CN121188103A (en) | Data processing and analysis methods, devices, equipment and media | |
| CN120872941A (en) | Data management enhancement method, device, equipment and storage medium | |
| CN119476213A (en) | Method, device, electronic device and storage medium for generating prompt words for text rewriting | |
| CN119829021A (en) | Method for generating front-end prototype based on generation countermeasure network | |
| CN119357378A (en) | A method, device, equipment and storage medium for entity relationship extraction based on domain adaptation transfer learning | |
| CN121029747B (en) | Data autonomy method, system and device based on artificial intelligence | |
| CN121301504B (en) | Industrial knowledge generation method and apparatus that integrates large models and knowledge graphs | |
| CN121832891A (en) | Pre-analysis method, device, medium and program product for business demand defect | |
| CN121880472A (en) | Data blood edge generation method and device, electronic equipment and storage medium | |
| CN121581168A (en) | A method for generating enterprise intelligent knowledge base based on large language model | |
| CN121210474A (en) | A Data Governance Method and System for Construction Projects Based on Multimodal and Large Models | |
| CN121029912A (en) | Methods for Automatic Identification and Rapid Structured Transformation of Insurance Terms and Conditions Documents | |
| CN120849287A (en) | A method and system for automatically generating test cases based on large models | |
| CN121579463A (en) | Methods, systems, devices, processors, and storage media for automatic filling of missing metadata attributes based on large models. | |
| CN121542355A (en) | A data management method, apparatus, electronic device, and computer program product | |
| CN120763182A (en) | Energy data registration method and related device | |
| CN121597576A (en) | Demand-protocol double-constraint test case generation method based on large model | |
| CN120950919A (en) | Method, apparatus, equipment and media for e-commerce order data quality inspection based on NL2SQL commands |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |