CN112115196A - Industrial big data visual modeling method and system - Google Patents
Industrial big data visual modeling method and system Download PDFInfo
- Publication number
- CN112115196A CN112115196A CN202010780964.8A CN202010780964A CN112115196A CN 112115196 A CN112115196 A CN 112115196A CN 202010780964 A CN202010780964 A CN 202010780964A CN 112115196 A CN112115196 A CN 112115196A
- Authority
- CN
- China
- Prior art keywords
- node
- data
- model
- processing
- canvas
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a visual modeling method and a visual modeling system for industrial big data, wherein the method comprises the steps of summarizing structured data and unstructured data; processing and storing the data by adopting an ETL technology; model control is quoted, a model analysis field is set, parameters are configured, and a model is constructed; and evaluating and testing the model, if the model meets the requirements, applying the model, otherwise, reconsidering the input field, adjusting the related parameters until the input field meets the requirements, reducing the requirements of the user programming capability, and facilitating modeling by the user.
Description
Technical Field
The invention relates to a visual modeling method and system for industrial big data.
Background
Industrial enterprises accumulate a large amount of data, and many enterprises want to mine data values by using data analysis tools, but the existing mining data values have the following problems: (1) the programming capability of business personnel is weak, and the model is difficult to be established quickly to analyze and mine industrial data; (2) the data analysis process is complex, and the data modeling capability of business personnel is insufficient.
Disclosure of Invention
The invention aims to solve the technical problem of providing a visual modeling method and system for industrial big data, which can reduce the requirement of user programming capability and facilitate modeling by users.
One of the present invention is realized by: an industrial big data visualization modeling method comprises the following steps:
step 1, summarizing structured and unstructured data;
step 2, processing and storing the data by adopting an ETL technology;
step 3, quoting a model control, setting a model analysis field, configuring parameters and constructing a model;
and 4, evaluating and testing the model, if the model meets the requirements, applying the model, otherwise, reconsidering the input field, and adjusting the relevant parameters until the model meets the requirements.
Further, the step 3 is further specifically:
newly building canvas nodes: selecting a type node menu so as to determine the type of the adopted node, drawing a corresponding node in the Canvas on the right side according to the mouse placement, and recording the node type and the node position information;
establishing node association: clicking nodes in Canvas, drawing arrows to perform node association, and storing the node association in a tree form;
parameter configuration: double-clicking a node in Canvas, opening an HTML configuration interface of a node of a corresponding type, acquiring a data source parameter configured by a previous node in the configuration interface by a user according to node association, sending a request to acquire a corresponding data resource, then performing corresponding parameter configuration according to a data source, and persistently storing parameter information of the node in a system configuration database;
performing a model: and sending an execution request, calling a corresponding function to perform operation according to the parameter information of the node, and then displaying an operation result at the browser end.
Further, the step 1 is further specifically: analyzing the demand service, summarizing the related data of the structure and the non-structure, or uploading the related data.
Further, the processing of the data by the ETL technique in step 2 includes missing value processing, abnormal value processing, discretization processing, and data normalization processing.
The second invention is realized by the following steps: an industrial big data visualization modeling system comprises the following modules:
the data source management module is used for summarizing structured and unstructured data;
the data processing module is used for processing and storing the data by adopting an ETL technology;
a model building module refers to a model control, sets a model analysis field, configures parameters and builds a model;
and the test module is used for evaluating and testing the model, performing model application if the model meets the requirements, and otherwise, reconsidering the input field and adjusting related parameters until the model meets the requirements.
Further, the construction model module is further specifically:
newly building canvas nodes: selecting a type node menu so as to determine the type of the adopted node, drawing a corresponding node in the Canvas on the right side according to the mouse placement, and recording the node type and the node position information;
establishing node association: clicking nodes in Canvas, drawing arrows to perform node association, and storing the node association in a tree form;
parameter configuration: double-clicking a node in Canvas, opening an HTML configuration interface of a node of a corresponding type, acquiring a data source parameter configured by a previous node in the configuration interface by a user according to node association, sending a request to acquire a corresponding data resource, then performing corresponding parameter configuration according to a data source, and persistently storing parameter information of the node in a system configuration database;
performing a model: and sending an execution request, calling a corresponding function to perform operation according to the parameter information of the node, and then displaying an operation result at the browser end.
Further, the data source management module is further specifically: analyzing the demand service, summarizing the related data of the structure and the non-structure, or uploading the related data.
Further, the processing of the data by the data processing module using the ETL technique includes missing value processing, abnormal value processing, discretization processing, and data normalization processing.
The invention has the following advantages: the visual modeling of business personnel on industrial data is realized, the dependence of the business personnel on a big data analysis program is reduced, the modeling speed is increased, the model application efficiency is improved, and the management level is improved.
Drawings
The invention will be further described with reference to the following examples with reference to the accompanying drawings.
FIG. 1 is a flow chart of the method of the present invention.
FIG. 2 is a schematic block diagram of the system of the present invention.
FIG. 3 is a system block diagram of an embodiment of the present invention.
Fig. 4 is a schematic diagram illustrating the effect of the embodiment of the present invention.
Detailed Description
As shown in FIG. 1, the invention relates to an industrial big data visualization modeling method, which comprises the following steps:
step 1, analyzing a demand service, summarizing structured and unstructured related data, or uploading related data;
step 2, processing and storing the data by adopting an ETL technology, wherein the processing of the data by adopting the ETL technology comprises missing value processing, abnormal value processing, discretization processing and data standardization processing;
step 3, newly building canvas nodes: selecting a type node menu so as to determine the type of the adopted node, drawing a corresponding node in the Canvas on the right side according to the mouse placement, and recording the node type and the node position information;
establishing node association: clicking nodes in Canvas, drawing arrows to perform node association, and storing the node association in a tree form;
parameter configuration: double-clicking a node in Canvas, opening an HTML configuration interface of a node of a corresponding type, acquiring a data source parameter configured by a previous node in the configuration interface by a user according to node association, sending a request to acquire a corresponding data resource, then performing corresponding parameter configuration according to a data source, and persistently storing parameter information of the node in a system configuration database;
performing a model: sending an execution request, calling a corresponding function to perform operation according to the parameter information of the node, and then displaying an operation result at a browser end;
and 4, evaluating and testing the model, if the model meets the requirements, applying the model, otherwise, reconsidering the input field, and adjusting the relevant parameters until the model meets the requirements.
As shown in FIG. 2, the industrial big data visualization modeling system of the invention comprises the following modules:
the data source management module analyzes the demand service, summarizes the related data of the structuring and the non-structuring, or uploads the related data;
the processing data module is used for processing and storing data by adopting an ETL technology, and the processing of the data by adopting the ETL technology comprises missing value processing, abnormal value processing, discretization processing and data standardization processing;
constructing a model module, and newly building canvas nodes: selecting a type node menu so as to determine the type of the adopted node, drawing a corresponding node in the Canvas on the right side according to the mouse placement, and recording the node type and the node position information;
establishing node association: clicking nodes in Canvas, drawing arrows to perform node association, and storing the node association in a tree form;
parameter configuration: double-clicking a node in Canvas, opening an HTML configuration interface of a node of a corresponding type, acquiring a data source parameter configured by a previous node in the configuration interface by a user according to node association, sending a request to acquire a corresponding data resource, then performing corresponding parameter configuration according to a data source, and persistently storing parameter information of the node in a system configuration database;
performing a model: sending an execution request, calling a corresponding function to perform operation according to the parameter information of the node, and then displaying an operation result at a browser end;
and the test module is used for evaluating and testing the model, performing model application if the model meets the requirements, and otherwise, reconsidering the input field and adjusting related parameters until the model meets the requirements.
One embodiment of the present invention:
platform architecture
As shown in fig. 3, the data source layer collects structured and unstructured data sources, processes the data using the ETL technique, and stores the data in a data warehouse, a distributed database, and a Hadoop platform. The preprocessed data can conveniently support multidimensional data display, real-time data query, data sharing, data modeling and the like, and industrial big data intelligent application with the advantages of reducing cost, improving production efficiency, improving product quality and creating more economic benefits is provided for enterprises based on data analysis and mining technology.
Part (II) core algorithm and industry application model
The core algorithm of the system comprises a traditional statistical analysis method, various classical data mining algorithms such as clustering, association, decision tree and prediction and machine learning algorithms, and can efficiently support industrial big data analysis modeling. Based on a classical analysis model, the big data analysis system is deeply applied to the aspects of logistics, equipment, production, management decision and the like, and an industrial algorithm model is precipitated.
(III) visual modeling Process
The big data analysis system follows CRISP-DM (cross-industry data mining) cross-industry data mining process standard, and industrial big data visual modeling and application are rapidly realized.
The system adopts a B/S architecture, the front end uses Vue + Element + G6 to provide a visual modeling function, the display of a modeling node network is carried out through G6, the back end uses Springboot + Mybatis to realize, a data source can support simple desktop data such as EXCEL, CSV, text files and the like, and also can support non-relational data such as MySQL, Oracle mainstream relational data, redis, mangold and the like.
The visual modeling process is as follows, as shown in FIG. 4.
(1) According to the analysis demand service, relevant data are gathered and sorted, and the user can also be supported to upload data by himself;
(2) preprocessing data, including missing value processing, abnormal value processing, discretization processing, data standardization processing and the like; the missing value processing mode comprises neglecting, directly removing samples, filling with maximum frequency, mean filling, interpolation and the like for repairing. The abnormal value judgment standard is a positive standard deviation and a negative standard deviation (1 time, 2 times and 3 times) of the mean value, and the processing mode comprises the steps of neglecting, directly removing the sample and restoring the mean value. The discretization processing comprises self-defining coding, one-hot coding and virtual variable coding. The data standardization mode comprises range standardization and logarithmic standardization.
(3) And (4) quoting the model control, setting a model analysis field, configuring relevant parameters and constructing a model. The main implementation process is as follows: newly building canvas nodes: and selecting a type node menu so as to determine the type of the adopted node, drawing a corresponding node in the Canvas on the right side according to the mouse placement, and recording the information such as the node type, the node position and the like. Establishing node association: and clicking the nodes in the Canvas, drawing arrows to perform node association, and performing record storage on the node association in a tree form. Parameter configuration: double-clicking a node in Canvas, opening an HTML configuration interface of a node of a corresponding type, acquiring a configured data source parameter of a previous node according to node association in the configuration interface by a user, sending a request to acquire a corresponding data resource, and then configuring the corresponding parameter according to the data source. The parameter information of the nodes is persistently stored in a system configuration database; executing the model: sending an execution request, calling a corresponding function to perform operation according to the parameter information of the node, and then displaying an operation result at a browser end;
(4) carrying out evaluation test on the model, wherein the main evaluation criteria comprise root mean square relative error, classification accuracy, contour coefficient and cross validation performance, if the model meets the requirements, the next step of model application can be carried out, otherwise, the input field is considered again, and relevant parameters are adjusted;
(5) and on the basis of satisfaction of the test model, applying the model to carry out service prediction.
Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.
Claims (8)
1. A visual modeling method for industrial big data is characterized in that: the method comprises the following steps:
step 1, summarizing structured and unstructured data;
step 2, processing and storing the data by adopting an ETL technology;
step 3, quoting a model control, setting a model analysis field, configuring parameters and constructing a model;
and 4, evaluating and testing the model, if the model meets the requirements, applying the model, otherwise, reconsidering the input field, and adjusting the relevant parameters until the model meets the requirements.
2. The industrial big data visualization modeling method according to claim 1, characterized in that: the step 3 is further specifically as follows:
newly building canvas nodes: selecting a type node menu so as to determine the type of the adopted node, drawing a corresponding node in the Canvas on the right side according to the mouse placement, and recording the node type and the node position information;
establishing node association: clicking nodes in Canvas, drawing arrows to perform node association, and storing the node association in a tree form;
parameter configuration: double-clicking a node in Canvas, opening an HTML configuration interface of a node of a corresponding type, acquiring a data source parameter configured by a previous node in the configuration interface by a user according to node association, sending a request to acquire a corresponding data resource, then performing corresponding parameter configuration according to a data source, and persistently storing parameter information of the node in a system configuration database;
performing a model: and sending an execution request, calling a corresponding function to perform operation according to the parameter information of the node, and then displaying an operation result at the browser end.
3. The industrial big data visualization modeling method according to claim 1, characterized in that: the step 1 is further specifically as follows: analyzing the demand service, summarizing the related data of the structure and the non-structure, or uploading the related data.
4. The industrial big data visualization modeling method according to claim 1, characterized in that: the step 2 of processing the data by using the ETL technology comprises missing value processing, abnormal value processing, discretization processing and data standardization processing.
5. An industrial big data visualization modeling system is characterized in that: the system comprises the following modules:
the data source management module is used for summarizing structured and unstructured data;
the data processing module is used for processing and storing the data by adopting an ETL technology;
a model building module refers to a model control, sets a model analysis field, configures parameters and builds a model;
and the test module is used for evaluating and testing the model, performing model application if the model meets the requirements, and otherwise, reconsidering the input field and adjusting related parameters until the model meets the requirements.
6. The industrial big data visualization modeling system according to claim 5, wherein: the construction model module is further specifically:
newly building canvas nodes: selecting a type node menu so as to determine the type of the adopted node, drawing a corresponding node in the Canvas on the right side according to the mouse placement, and recording the node type and the node position information;
establishing node association: clicking nodes in Canvas, drawing arrows to perform node association, and storing the node association in a tree form;
parameter configuration: double-clicking a node in Canvas, opening an HTML configuration interface of a node of a corresponding type, acquiring a data source parameter configured by a previous node in the configuration interface by a user according to node association, sending a request to acquire a corresponding data resource, then performing corresponding parameter configuration according to a data source, and persistently storing parameter information of the node in a system configuration database;
performing a model: and sending an execution request, calling a corresponding function to perform operation according to the parameter information of the node, and then displaying an operation result at the browser end.
7. The industrial big data visualization modeling system according to claim 5, wherein: the data source management module is further specifically: analyzing the demand service, summarizing the related data of the structure and the non-structure, or uploading the related data.
8. The industrial big data visualization modeling system according to claim 5, wherein: the processing data module adopts an ETL technology to process data, and comprises missing value processing, abnormal value processing, discretization processing and data standardization processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010780964.8A CN112115196A (en) | 2020-08-06 | 2020-08-06 | Industrial big data visual modeling method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010780964.8A CN112115196A (en) | 2020-08-06 | 2020-08-06 | Industrial big data visual modeling method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112115196A true CN112115196A (en) | 2020-12-22 |
Family
ID=73799461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010780964.8A Pending CN112115196A (en) | 2020-08-06 | 2020-08-06 | Industrial big data visual modeling method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112115196A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113110311A (en) * | 2021-03-05 | 2021-07-13 | 中海创科技(福建)集团有限公司 | Industrial big data online modeling and model sharing method, device, equipment and medium |
| CN113742432A (en) * | 2021-08-20 | 2021-12-03 | 广州市易工品科技有限公司 | Editable ER graph generation method and device |
| CN114254021A (en) * | 2021-12-30 | 2022-03-29 | 重庆允成互联网科技有限公司 | Multi-source data processing method and data factory |
| CN115826932A (en) * | 2023-02-10 | 2023-03-21 | 中国电子科技集团公司第十五研究所 | Assessment model and assessment model construction method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020109704A1 (en) * | 2000-12-20 | 2002-08-15 | Microsoft Corporation | Dynamic, live surface and model elements for visualization and modeling |
| CN103970109A (en) * | 2014-05-27 | 2014-08-06 | 重庆大学 | Flow reappearing visualization method based on industrial field data under Web environment |
| CN107450902A (en) * | 2016-04-29 | 2017-12-08 | 埃森哲环球解决方案有限公司 | System architecture with visual modeling tool |
| CN109523318A (en) * | 2018-11-16 | 2019-03-26 | 杭州市商务委员会（杭州市粮食局） | The analysis platform of electronic commerce data visual modeling transboundary |
-
2020
- 2020-08-06 CN CN202010780964.8A patent/CN112115196A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020109704A1 (en) * | 2000-12-20 | 2002-08-15 | Microsoft Corporation | Dynamic, live surface and model elements for visualization and modeling |
| CN103970109A (en) * | 2014-05-27 | 2014-08-06 | 重庆大学 | Flow reappearing visualization method based on industrial field data under Web environment |
| CN107450902A (en) * | 2016-04-29 | 2017-12-08 | 埃森哲环球解决方案有限公司 | System architecture with visual modeling tool |
| CN109523318A (en) * | 2018-11-16 | 2019-03-26 | 杭州市商务委员会（杭州市粮食局） | The analysis platform of electronic commerce data visual modeling transboundary |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113110311A (en) * | 2021-03-05 | 2021-07-13 | 中海创科技(福建)集团有限公司 | Industrial big data online modeling and model sharing method, device, equipment and medium |
| CN113742432A (en) * | 2021-08-20 | 2021-12-03 | 广州市易工品科技有限公司 | Editable ER graph generation method and device |
| CN114254021A (en) * | 2021-12-30 | 2022-03-29 | 重庆允成互联网科技有限公司 | Multi-source data processing method and data factory |
| CN115826932A (en) * | 2023-02-10 | 2023-03-21 | 中国电子科技集团公司第十五研究所 | Assessment model and assessment model construction method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112115196A (en) | Industrial big data visual modeling method and system | |
| CN118311914B (en) | A production line data acquisition control method and system for intelligent workshop | |
| CN113887749B (en) | Multi-dimensional monitoring and disposal method, equipment and platform of power Internet of Things based on cloud-edge collaboration | |
| CN111339375A (en) | Universal big data model configuration and analysis method | |
| CN108491991B (en) | Constraint analysis system and method based on industrial big data product construction period | |
| CN116599857B (en) | Digital twin application system suitable for multiple scenes of Internet of things | |
| CN108052020B (en) | A chemical process simulator for intelligent manufacturing | |
| CN120748172B (en) | Building engineering construction progress early warning method and system based on machine learning | |
| CN118982175A (en) | AIGC-based intelligent analysis system and method for business management data | |
| WO2025139186A1 (en) | Big data analysis and processing system based on deep learning | |
| CN119066056A (en) | An intelligent silicon rod cutting process management method and system | |
| CN118396606A (en) | Manufacturing industry product production optimization method based on multi-mode large model | |
| CN117972356A (en) | Engineering cost control method based on big data analysis | |
| CN113626285A (en) | Model-based job monitoring method and device, computer equipment and storage medium | |
| CN115562660A (en) | Lightweight knowledge base design applied to product quality analysis | |
| CN119719059A (en) | Educational resource sharing system and method based on cloud platform | |
| CN118586782A (en) | A mathematical and statistical analysis method based on quality management decision-making system | |
| CN114789557B (en) | Multi-nozzle printing method of 3d printer | |
| CN113515500B (en) | Visual data processing system and processing method | |
| CN117196523A (en) | Robot flow automation method and device | |
| CN116150989A (en) | Photovoltaic module manufacturing method and photovoltaic module | |
| CN113191569A (en) | Enterprise management method and system based on big data | |
| CN119539622B (en) | Quality monitoring system of full life cycle | |
| CN121351514B (en) | Method and System for Analyzing the Compressive Bearing Capacity of Composite Reinforced Sea Sand Concrete Columns | |
| CN110135647A (en) | A control method and control device for realizing trend prediction based on feature modeling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20201222 |
|
| WD01 | Invention patent application deemed withdrawn after publication |