CN114817425B - Method, device and equipment for classifying cold and hot data and readable storage medium - Google Patents

Abstract

The present invention provides a method, device, device and readable storage medium for classifying cold and hot data, and relates to the technical field of data storage. In this application, by counting the access characteristics of the inbound data and the value of the inbound data itself, the temperature of the inbound data at the current moment is calculated based on the improved Newton's law of cooling data classification model of data value, so that the temperature of the inbound data can be calculated. At any time, it is divided according to the size of the temperature value of the storage data, and the storage data is divided into hot data, warm data, and cold data in real time. Data with high data value and high access frequency is placed in high-speed storage devices, and data with low data value and low access frequency is placed in low-speed storage devices. In this way, the storage cost of massive data in data management terminals can be effectively reduced.

Description

A method, device, device and readable storage medium for classifying cold and hot data

technical field

The present invention relates to the technical field of data storage, and in particular, to a method, apparatus, device and readable storage medium for classifying cold and hot data.

Background technique

With the long-term data accumulation in the traffic structure monitoring scenario, the traditional solution is to continuously expand the storage capacity of the machine or add higher-performance storage devices. However, this method is accompanied by serious waste of storage resources and increased energy consumption. In view of the current situation, many experts and scholars divide the monitoring data cold and hot characteristics according to the frequency of access, and classify and store them based on the cold and hot characteristics of the data. However, at present, many hot and cold data judgment models F-type more consider the access characteristics of data, and calculate the temperature of data according to the access characteristics of data, without considering the value of the data itself.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method, apparatus, device and readable storage medium for classifying cold and hot data, so as to improve the above problems. In order to achieve the above object, the technical scheme adopted by the present invention is as follows:

In a first aspect, the present application provides a method for classifying hot and cold data, including:

At least two pieces of data flow information are acquired, and each of the data flow information includes storage data and a temperature parameter and a first time corresponding to the storage data.

Determine whether the inbound data is accessed, and if the inbound data is accessed, obtain the temperature parameter, data value increment, data corresponding to the second time and the inbound data at the first time temperature increment and data value weight; and according to the temperature parameter, the data value increment, the data temperature increment and the data value weight, update the corresponding data of the inbound data at the second moment. the temperature parameter; the second time is the time parameter when the inbound data is accessed.

Based on the updated temperature parameters corresponding to the inbound data, the data flow information is classified into cold and hot data.

In a second aspect, the present application also provides an apparatus for classifying cold and hot data, including an acquisition module, a judgment module and a classification module, wherein:

Acquisition module: used to acquire at least two pieces of data flow information, each of the data flow information includes storage data, and a temperature parameter and a first time corresponding to the storage data.

Judgment module: for judging whether the inbound data is accessed, if the inbound data is accessed, obtain the second moment and the temperature parameters and data corresponding to the inbound data at the first moment value increment, data temperature increment and data value weight; and update the inbound data in the second according to the temperature parameter, the data value increment, the data temperature increment and the data value weight The temperature parameter corresponding to the time; the second time is the time parameter when the inbound data is accessed.

Classification module: used for classifying cold and hot data on the data flow information based on the updated temperature parameter corresponding to the storage data.

In a third aspect, the present application also provides a device for classifying cold and hot data, including:

memory for storing computer programs;

The processor is configured to implement the steps of the method for classifying hot and cold data when executing the computer program.

In a fourth aspect, the present application further provides a readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method for classifying cold and hot data are implemented.

The beneficial effects of the present invention are:

In this application, by counting the access characteristics of the inbound data (such as access time, access frequency, etc.) and the value of the inbound data itself, the cold and hot data classification model based on the improved Newton's law of cooling based on the value of the data calculates the inbound and outbound data. The temperature of the data at the current moment, so that it can be divided at any time according to the temperature value (that is, the activity level) of the inbound data, and the inbound data can be divided into hot data, warm data, and cold data in real time, based on the characteristics of cold and heat It provides a solid foundation for the hierarchical storage of data, which helps to put data with high data value and high access frequency in high-speed storage devices, and data with low data value and low access frequency in low-speed storage devices. This method can effectively reduce the storage cost of massive data in the data management terminal.

Other features and advantages of the present invention will be set forth in the description which follows, and, in part, will be apparent from the description, or may be learned by practice of embodiments of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description, claims, and drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic flowchart of a method for classifying cold and hot data according to an embodiment of the present invention;

2 is a schematic structural diagram of an apparatus for classifying cold and hot data according to an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a device for classifying cold and hot data according to an embodiment of the present invention.

In the figure: 700 - apparatus for classifying hot and cold data; 710, acquisition module; 711, first processing unit; 712, second processing unit; 7121, first subunit; 7122, second subunit; 7123, third subunit unit; 7124, fourth subunit; 720, judgment module; 721, acquisition subunit; 722, first calculation unit; 723, second calculation unit; 724, third calculation unit; 730, classification module; 800, cold and heat 801, a processor; 802, a memory; 803, a multimedia component; 804, an I/O interface; 805, a communication component.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

First, the applicable application scenarios of this application and the system architecture under the application scenarios are introduced. The present application can be applied to a scenario where the data management terminal stores data based on the cold and hot attributes of the data. In the present application, monitoring data of a traffic structure is taken as an example for detailed description.

Example 1:

Referring to FIG. 1 , FIG. 1 shows a schematic flowchart of the method for classifying cold and hot data in this embodiment. The present application provides a method for classifying hot and cold data, including step S1, step S2 and step S3, wherein:

Step S1: Acquire at least two data flow information, each of which includes storage data, a temperature parameter and a first time corresponding to the storage data.

It can be understood that in this step, the data management terminal collects the storage data in real time through sensors, monitoring equipment and other tools, and the temperature parameters corresponding to the storage data and the first moment constitute data flow information, and the initial value of the first moment is The time from the data management terminal inspection to the storage data, the storage data is the experimental data detected by the sensor or monitoring equipment, and the temperature parameter when the storage data is just transmitted to the data management terminal is the user-defined temperature input by the client.

In another exemplary embodiment, step S1 further includes step S101 and step S102, wherein:

Step S101, respectively determine whether each of the data flow information is within the preset classification conditions, if the data flow information is within the preset classification conditions, then obtain at least one sub-database, the sub-database The data set having the same attribute after clustering processing is performed for the data flow information; the classification condition is the information of dividing the data of the same attribute.

It can be understood that, in this step, the data flow information is classified according to the classification conditions and the storage data included in the data flow information. The classification condition in this embodiment is the type information of the storage data. It is assumed that the classification condition is the two types of information of cable force and wind direction in the traffic structure scene, and according to the two classification conditions in the above classification conditions, Divide the data flow information into two sub-databases.

Step S102, based on the first moment, determine whether there is missing data in each of the sub-databases, if there is no missing data, remove each of the sub-databases respectively, and update each of the sub-databases as: The deleted subdatabase; if there is missing data, complete each of the subdatabases, and update the subdatabase to be the completed subdatabase.

It can be understood that, in this step, based on the time series, it is determined whether the two types of data of cable force and wind direction are missing in a certain period or time based on the first moment. Completion and update of sub-databases; data completion methods can be filled with numbers such as zero, average, and adjacent values of missing points, and can also be filled with algorithms such as linear interpolation and Lagrangian interpolation. If there is no missing, the outliers in the sub-database will be eliminated and the sub-database will be updated. For the elimination of outliers, the sub-database can be eliminated by setting the upper and lower limits of the elimination threshold.

In order to prevent these abnormal monitoring data from interfering with subsequent time series data prediction, it is very important to select the correct abnormal data processing method according to the specific conditions of the real-time test data set. Therefore, the above step S102 includes step S1021, step S1022, step S1023 and step S1024, wherein:

Step S1021: Acquire a sub-set, where the sub-set is a set of at least ten consecutive pieces of the data flow information intercepted from the sub-database based on the first moment.

It can be understood that, in this step, continuous at least ten pieces of data flow information are selected from the wind direction sub-database based on the time series to form a subset. The Soli sub-database intercepts sub-collections in the above-mentioned manner.

Step S1022, calculating the exclusion conditions: based on the subset, calculate the arithmetic mean and standard deviation of the storage data in the subset respectively; calculate according to the arithmetic mean and the standard deviation to obtain the first a condition.

It can be understood that, in this step, the first condition is obtained by calculating according to the arithmetic mean and standard deviation of all the inbound data in the subset, and calculating according to the arithmetic mean and the standard deviation. Assuming that in this embodiment, the abnormal data is eliminated by calculating the mean and standard deviation based on Laida's rule, then the first condition obtained by calculation is shown in formula (1):

（1）

(1)

为入库数据的算数平均数，

为入库数据的，

为入库数据的标准差值。in:

is the arithmetic mean of the inbound data,

For inbound data,

is the standard deviation of the incoming data.

Step S1023 , according to the first condition, remove the sub-database to obtain the removed sub-database.

It can be understood that, in this step, the incoming data that exceeds the range of the above-mentioned first condition is abnormal data, which is marked and then eliminated to obtain the eliminated sub-database.

Step S1024, judging whether the deleted sub-database obeys the normal distribution, and if it is judged that the deleted sub-database does not obey the normal distribution, restart the calculation of the elimination condition according to the deleted sub-database until after the elimination. The sub-database of is normally distributed.

It can be understood that, in this step, according to the deleted sub-database, it is judged whether the remaining data obeys the normal distribution based on the time series, and if it does not obey, the exclusion condition is recalculated according to the retained storage data, and according to the updated data. Elimination is carried out according to the subsequent elimination conditions until the sub-database after elimination obeys the normal distribution.

Due to many uncertain factors, such as monitoring equipment disconnection, sensor failure, network transmission interruption, replacement of monitoring equipment, etc., there are different degrees of missing data in monitoring time series data, which in turn affects the integrity of monitoring data and monitoring. Accuracy of subject assessment. Therefore, step S102 further includes step S1025, step S1026, step S1027 and step S1028, wherein:

Step S1025: Obtain missing segment information in the sub-database based on the first moment.

It can be understood that in this step, there are data missing in those places obtained based on the time series.

Step S1026: Acquire data parameters at both ends of the missing segment information according to the missing segment information, where the data parameters include the storage data and the first time.

It can be understood that, in this step, according to the missing segment information, the inbound data at the beginning and the end of the missing segment information and the first moment corresponding to the inbound data are obtained respectively.

Step S1027: Calculate and obtain an interpolation coefficient according to the data parameters at the first and last ends of the missing segment information.

It can be understood that, in this step, the data parameters at the beginning and end of the missing segment information are set to be (a ₁ , b ₁ ) and (a ₂ , b ₂ ) respectively, and the interpolation is calculated according to formula (2) according to the above two data parameters. coefficient, formula (2) is as follows:

（2）

(2)

为插值系数；a为缺失段信息中间任意一点对应的第一时刻，a₁为缺失段信息首端对应的第一时刻，a₂为缺失段信息尾端对应的第一时刻。in:

is the interpolation coefficient; a is the first moment corresponding to any point in the middle of the missing segment information, a ₁ is the first moment corresponding to the head end of the missing segment information, and a ₂ is the first moment corresponding to the tail end of the missing segment information.

Step S1028: Fill in the missing segment information according to the data parameter and the interpolation coefficient.

It can be understood that, in this step, the data parameters and the interpolation coefficients fill in the missing segment information according to formula (3), and formula (3) is as follows:

（3）

(3)

为插值系数，y₁为缺失段信息首端第一时刻对应的入库数据，y₂为缺失段信息尾端第一时刻对应的入库数据。Among them: y is the storage data corresponding to any first moment in the middle of the missing segment information,

is the interpolation coefficient, y ₁ is the storage data corresponding to the first moment at the head end of the missing segment information, and y ₂ is the storage data corresponding to the first moment at the tail end of the missing segment information.

Step S2, judging whether the inbound data is accessed, if the inbound data is accessed, obtain the second time and the corresponding temperature parameter and data value increase of the inbound data at the first time. data temperature increment and data value weight; and according to the temperature parameter, the data value increment, the data temperature increment and the data value weight, update the inbound data at the second moment The corresponding temperature parameter; the second time is the time parameter when the inbound data is accessed.

It can be understood that in this step, while transmitting data to the data management terminal, the client customizes the input temperature parameters, data value increment, data temperature increment and data value weight. Then, the above-mentioned custom parameters corresponding to the storage data are found, and the updated temperature parameters are calculated according to the above-mentioned custom parameters, and the updated temperature parameters correspond to the second moment when the storage data is accessed. Access to database data includes operations such as modification and query.

In this embodiment, step S2 further includes step S201, step S202 and step S203, wherein:

Step S201: Calculate according to the data value increment, the data temperature increment and the data value weight to obtain a data temperature change value.

It can be understood that in this step, the data temperature change value is calculated according to formula (4), and formula (4) is as follows:

（4）

(4)

为数据价值权重；R为数据价值增量，为自定义常量；

为数据温度增量，为自定义常量。Among them: T _Z is the data temperature change value;

is the data value weight; R is the data value increment, which is a custom constant;

is the data temperature increment, and is a custom constant.

Step S202: Calculate according to the temperature parameter, the data temperature change value and the Newton attenuation coefficient to obtain the updated temperature parameter.

It can be understood that in this step, the temperature parameter of the storage data is updated according to the formula (5), and the formula (5) is as follows:

（5）

(5)

为入库数据在t_n时刻被访问时对应的温度参数；

为入库数据在被访问前的t_n-1时刻对应的温度参数；e为自然常数；c为牛顿衰减系数；T_Z为数据温度变化值。in,

is the corresponding temperature parameter when the inbound data is accessed at time t _n ;

is the temperature parameter corresponding to the stored data at time t _n-1 before being accessed; e is a natural constant; c is the Newton attenuation coefficient; T _Z is the temperature change value of the data.

Step S203: Based on the second time, establish a mapping relationship between the storage data, the updated temperature parameter and the second time.

It can be understood that, in this step, according to the time when the storage data is accessed and the updated temperature parameter, a mapping relationship between the second moment, the storage data and the updated temperature parameter is established, which is convenient for the storage data in the later stage. When accessed again, iteratively update its corresponding temperature parameters.

The method for determining the value weight in the above steps further includes step S2011, step S2012, step S2013 and step S2014, wherein:

Step S2011: Obtain a first parameter, a second parameter and research collection information, where the first parameter is the number of users accessing the sub-database; the second parameter is an evaluation value of the data application value corresponding to the sub-database ; The research collection information is a data collection composed of the sub-databases with different attributes.

It is understandable that, in this step, if a piece of data is accessed by more users, the value of the data will be higher; at the same time, if a piece of data is accessed by multiple users, its changes and its access performance will affect the data. More users, therefore, need to obtain the first parameter from the access record of the data management terminal. The second parameter of the sub-database input by the client is obtained from the data management terminal, and the parameter is determined by the user's manual evaluation of the usage of the value. According to different research questions, the sub-databases associated with the research questions are retrieved from the data management terminal to constitute research collection information.

Step S2012: Calculate according to the first parameter and the second parameter to obtain a first data value, where the first data value is the sum of the first parameter and the second parameter corresponding to the sub-database.

Step S2013: Calculate according to the first data value and the research set information to obtain a second data value, where the second data value is the sum of the first data values of all the sub-databases in the research set information. and.

Step S2014: Calculate according to the first data value and the second data value to obtain the data value weight corresponding to the sub-database.

It can be understood that in this step, the data value weight corresponding to the sub-database is determined according to formula (6), and formula (6) is as follows:

（6）

(6)

为i子数据库对应的数据价值权重；

为i子数据库的第一数据价值；i为子数据库对应的入库数据的类型，n为子数据库的总个数。通过公式（4）可以基于每个研究问题中关键性因素的不同而重新确定每个入库数据的数据价值权重，更符合研究的实际情况，提升最终结果的精准度。in:

is the data value weight corresponding to the i sub-database;

is the first data value of the i sub-database; i is the type of inbound data corresponding to the sub-database, and n is the total number of sub-databases. Through formula (4), the data value weight of each database data can be re-determined based on the difference of key factors in each research question, which is more in line with the actual situation of the research and improves the accuracy of the final result.

Step S3 , classifying the data flow information based on the updated temperature parameter corresponding to the storage data, classifying the cold and hot data.

It can be understood that, in this step, based on the updated temperature parameters corresponding to the storage data, the data stream information is classified as cold and hot data, and after the classification is completed, the second time is updated to the first time. At one time, the updated first time, the storage data and the updated temperature parameters constitute the updated data flow information and are stored in the data management terminal.

Example 2:

Please refer to FIG. 2. FIG. 2 shows a schematic structural diagram of an apparatus 700 for classifying cold and hot data according to the present embodiment. The apparatus 700 for classifying cold and hot data includes an acquisition module 710, a judgment module 720 and a classification module 730, wherein:

Obtaining module 710: configured to obtain at least two data flow information, each of the data flow information includes storage data, temperature parameters and first time corresponding to the storage data.

Preferably, the above obtaining module 710 further includes a first processing unit 711 and a second processing unit 712, wherein:

The first processing unit 711 is configured to separately determine whether each of the data flow information is within the preset classification conditions, and if the data flow information is within the preset classification conditions, obtain at least one sub-database , wherein the sub-database is a data set with the same attribute after the data stream information is clustered and divided; the classification condition is the information of dividing the data with the same attribute.

The second processing unit 712 is configured to determine whether there is missing data in each of the sub-databases based on the first moment, and if there is no missing data, remove each of the sub-databases, and update each sub-database. The sub-database is the deleted sub-database; if there is missing data, each sub-database is completed, and the sub-database is updated to be the completed sub-database.

Preferably, the above-mentioned second processing unit 712 includes a first subunit 7121, a second subunit 7122, a third subunit 7123 and a fourth subunit 7124, wherein:

The first subunit 7121 is configured to acquire the missing segment information in the subdatabase based on the first moment.

The second subunit 7122 is configured to acquire, according to the missing segment information, data parameters at both ends of the missing segment information, where the data parameters include the storage data and the first moment.

The third subunit 7123 is configured to calculate and obtain an interpolation coefficient according to the data parameters at the first and last ends of the missing segment information.

Fourth subunit 7124: used to fill in the missing segment information according to the data parameter and the interpolation coefficient.

Judging module 720: for judging whether the inbound data is accessed, and if the inbound data is accessed, obtain the second moment and the temperature parameter corresponding to the inbound data at the first moment, Data value increment, data temperature increment and data value weight; and according to the temperature parameter, the data value increment, the data temperature increment and the data value weight, update the inbound data in the The temperature parameter corresponding to the second time; the second time is the time parameter when the inbound data is accessed.

Preferably, the above judgment module 720 further includes an acquisition subunit 721, a first calculation unit 722, a second calculation unit 723 and a third calculation unit 724, wherein:

Obtaining subunit 721: used to obtain a first parameter, a second parameter and study set information, where the first parameter is the number of users accessing the sub-database; the second parameter is the data application corresponding to the sub-database The evaluation value of the value; the research collection information is a data collection composed of the sub-databases with different attributes;

The first calculation unit 722 is configured to calculate according to the first parameter and the second parameter to obtain a first data value, where the first data value is the first parameter and the second parameter corresponding to the sub-database Sum;

Second calculation unit 723: configured to calculate according to the first data value and the research set information to obtain a second data value, where the second data value is the data of all the sub-databases in the research set information The sum of the first data value;

The third calculation unit 724 is configured to calculate according to the first data value and the second data value to obtain the data value weight corresponding to the sub-database.

Classification module 730: configured to classify the cold and hot data on the data flow information based on the updated temperature parameter corresponding to the storage data.

It should be noted that, with regard to the apparatus in the foregoing embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

Example 3:

Corresponding to the above method embodiments, a device 800 for classifying hot and cold data is also provided in this embodiment, a device 800 for classifying cold and hot data described below and a method for classifying hot and cold data described above can refer to each other.

FIG. 3 is a block diagram of a device 800 for classifying hot and cold data according to an exemplary embodiment. As shown in FIG. 3 , the apparatus 800 for classifying hot and cold data may include: a processor 801 and a memory 802 . The apparatus 800 for classifying hot and cold data may also include one or more of a multimedia component 803 , an I/O interface 804 , and a communication component 805 .

Wherein, the processor 801 is configured to control the overall operation of the apparatus 800 for classifying hot and cold data, so as to complete all or part of the steps in the above method for classifying hot and cold data. The memory 802 is used to store various types of data to support the operation of the hot and cold data classification device 800, such data may include, for example, instructions for any application or method operating on the hot and cold data classification device 800 , and application-related data such as contact data, messages sent and received, pictures, audio, video, and more. The memory 802 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (Static Random Access Memory, SRAM for short), electrically erasable programmable read-only memory ( Electrically Erasable Programmable Read-OnlyMemory, referred to as EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (Programmable Read-Only Memory, referred to as PROM), read-only memory ( Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. Multimedia components 803 may include screen and audio components. Wherein the screen can be, for example, a touch screen, and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in memory 802 or transmitted through communication component 805 . The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 804 provides an interface between the processor 801 and other interface modules, and the above-mentioned other interface modules may be a keyboard, a mouse, a button, and the like. These buttons can be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the device 800 for classifying the hot and cold data and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC for short), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 may include: Wi-Fi module, Bluetooth module, NFC module.

In an exemplary embodiment, the apparatus 800 for classifying hot and cold data may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), digital signal processing Device (Digital Signal Processing Device, referred to as DSPD), Programmable Logic Device (Programmable Logic Device, referred to as PLD), Field Programmable Gate Array (Field Programmable Gate Array, referred to as FPGA), controller, microcontroller, microprocessor or other An electronic component implements the method for performing the above-mentioned classification of hot and cold data.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned method for classifying hot and cold data are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions can be executed by the processor 801 of the apparatus 800 for classifying hot and cold data to complete the above method for classifying hot and cold data.

Example 4:

Corresponding to the above method embodiments, a readable storage medium is also provided in this embodiment, and a readable storage medium described below and a method for classifying cold and hot data described above can be referred to each other correspondingly.

A readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the method for classifying hot and cold data according to the above method embodiment.

The readable storage medium may specifically be a USB flash drive, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, etc. that can store program codes. Readable storage medium.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. A method for cold-hot data classification, comprising:

acquiring at least two data stream information, wherein each data stream information comprises warehousing data, a temperature parameter corresponding to the warehousing data and a first moment;

judging whether the warehousing data is accessed, if so, acquiring a second moment and the temperature parameter, data value increment, data temperature increment and data value weight corresponding to the warehousing data at the first moment; updating the temperature parameter corresponding to the warehousing data at the second moment according to the temperature parameter, the data value increment, the data temperature increment and the data value weight; the second moment is a time parameter when the warehouse entry data is accessed;

and classifying the cold and hot data of the data stream information based on the updated temperature parameter corresponding to the warehousing data.

2. The method for cold-hot data classification according to claim 1, wherein all the warehoused data need to be preprocessed before determining whether the warehoused data is accessed, and the preprocessing includes:

respectively judging whether each data stream information is in a preset classification condition, if so, obtaining at least one sub-database, wherein the sub-database is a data set with the same attribute after the data stream information is clustered and divided; the classification condition is information for dividing data with the same attribute;

judging whether each sub-database has missing data or not based on the first moment, if not, respectively removing each sub-database, and updating each sub-database to be the removed sub-database; and if the missing data exists, completing each sub-database, and updating the sub-databases into the completed sub-databases.

3. The method of claim 2, wherein complementing the sub-database comprises:

acquiring missing segment information in the sub database based on the first moment;

acquiring data parameters of the head end and the tail end of the missing segment information according to the missing segment information, wherein the data parameters comprise the warehousing data and the first moment;

calculating to obtain interpolation coefficients according to the data parameters at the head end and the tail end of the missing segment information;

and filling the missing section information according to the data parameters and the interpolation coefficient.

4. A method of cold-hot data classification according to claim 2, wherein the method of determining the data value weight comprises:

acquiring a first parameter, a second parameter and research set information, wherein the first parameter is the number of users accessing the sub-database; the second parameter is an evaluation value of the application value of the data corresponding to the sub database; the research set information is a data set formed by the sub databases with different attributes;

calculating according to the first parameter and the second parameter to obtain a first data value, wherein the first data value is the sum of the first parameter and the second parameter corresponding to the sub-database;

calculating according to the first data value and the research set information to obtain a second data value, wherein the second data value is the sum of the first data values of all the sub-databases in the research set information;

and calculating according to the first data value and the second data value to obtain a data value weight corresponding to the sub-database.

5. A device for cold-hot data classification, comprising:

an acquisition module: the system comprises a database, a database server and a database server, wherein the database server is used for acquiring at least two data stream information, and each data stream information comprises warehouse-in data, a temperature parameter corresponding to the warehouse-in data and a first moment;

a judging module: the system comprises a storage unit, a temperature parameter acquisition unit, a data value increment acquisition unit, a data temperature increment acquisition unit and a data value weight acquisition unit, wherein the storage unit is used for judging whether the storage data is accessed, and if the storage data is accessed, the temperature parameter, the data value increment, the data temperature increment and the data value weight corresponding to the storage data at the first moment are acquired at a second moment; updating the temperature parameter corresponding to the warehousing data at the second moment according to the temperature parameter, the data value increment, the data temperature increment and the data value weight; the second moment is a time parameter when the warehousing data is accessed;

a classification module: and the system is used for classifying the cold and hot data of the data flow information based on the updated temperature parameter corresponding to the warehousing data.

6. The device for cold-hot data classification according to claim 5, wherein the obtaining module further comprises:

a first processing unit: the data flow information processing device is used for respectively judging whether each piece of data flow information is in a preset classification condition, if so, at least one sub-database is obtained, and the sub-databases are data sets with the same attribute after the data flow information is clustered and divided; the classification condition is information for dividing data with the same attribute;

a second processing unit: the database management system is used for judging whether missing data exist in each sub-database or not based on the first moment, if not, each sub-database is removed, and each sub-database is updated to be the removed sub-database; and if the missing data exists, completing each sub-database, and updating the sub-databases into the completed sub-databases.

7. A device for cold and hot data classification as claimed in claim 6, wherein the second processing unit comprises:

a first subunit: the database management system is used for acquiring the missing segment information in the sub database based on the first moment;

a second subunit: the data processing device is used for acquiring data parameters of the head end and the tail end of the missing segment information according to the missing segment information, wherein the data parameters comprise the warehousing data and the first moment;

a third subunit: the interpolation coefficient is obtained by calculation according to the data parameters at the head end and the tail end of the missing segment information;

a fourth subunit: and the interpolation coefficient is used for filling the missing segment information according to the data parameters and the interpolation coefficient.

8. The device for cold-hot data classification according to claim 6, wherein the determining module further comprises:

an acquisition subunit: the system comprises a database, a database management module and a database management module, wherein the database management module is used for acquiring a first parameter, a second parameter and research set information, and the first parameter is the number of users accessing the sub-database; the second parameter is an evaluation value of the application value of the data corresponding to the sub database; the research set information is a data set formed by the sub databases with different attributes;

the first calculation unit: the database management system is used for calculating according to the first parameter and the second parameter to obtain a first data value, wherein the first data value is the sum of the first parameter and the second parameter corresponding to the sub-database;

a second calculation unit: the database management system is used for calculating according to the first data value and the research set information to obtain a second data value, wherein the second data value is the sum of the first data values of all the sub-databases in the research set information;

a third calculation unit: and the data value weight corresponding to the sub-database is obtained by calculating according to the first data value and the second data value.

9. An apparatus for hot and cold data classification, comprising:

a memory for storing a computer program;

a processor for implementing the steps of the method of hot and cold data classification as claimed in any one of claims 1 to 4 when said computer program is executed.

10. A readable storage medium, characterized by: the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of cold-hot data classification according to any one of claims 1 to 4.

Images