CN112699142A - Cold and hot data processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The invention relates to the field of data storage, and discloses a cold and hot data processing method, which comprises the following steps: the method comprises the steps of obtaining an original data set, classifying the original data set to obtain a classified data set, inquiring a key name set of the classified data set, obtaining an evaluation index set corresponding to the key name set, calculating the weight of the key name set based on the evaluation index set, evaluating whether the weight reaches a preset weight, marking the key name set as a hot data set if the weight reaches the preset weight, moving the key value set of the hot data set to a cache, marking the key name set as a cold data set if the weight does not reach the preset weight, and moving the key value set of the cold data set out of the cache. The invention also provides a cold and hot data processing device, electronic equipment and a storage medium. The invention also relates to blockchain techniques, the hot data sets may be stored in blockchain nodes. The invention can realize high-efficiency data query and reduce the data storage cost.

Description

Cold and hot data processing method, device, electronic device and storage medium

technical field

The present invention relates to the field of data storage, and in particular, to a method, device, electronic device and computer-readable storage medium for processing cold and hot data.

Background technique

In the process of continuous changes of enterprise business, the business configuration data in the enterprise configuration system shows an explosive growth. The data will be divided into cold data and hot data according to the access heat, and the cold and hot data have relative independence in logical space within a certain period of time. If both cold data and hot data are stored in the same storage method, the service performance of the system will be affected and the storage cost will be greatly increased. How to reduce the cost of data storage on the premise of ensuring the stability of system service performance is an urgent problem to be solved according to the degree of hot and cold data.

SUMMARY OF THE INVENTION

The present invention provides a method, device, electronic device and computer-readable storage medium for processing hot and cold data, the main purpose of which is to realize efficient data query and reduce data storage cost at the same time.

In order to achieve the above-mentioned purpose, a kind of cold and heat data processing method provided by the present invention comprises:

Obtain an original data set, classify the original data set, and obtain a classified data set;

query the key name set of the classification data set, and obtain the evaluation index set corresponding to the key name set;

Calculate the weight of the key name set based on the evaluation index set, and evaluate whether the weight reaches a preset weight;

If the weight reaches the preset weight, the key name set is marked as a hot data set, and when it is judged that the key value set of the hot data set is not in the cache of the preset server, the hot data set is The key-value set is moved to the cache of the default server;

If the weight does not reach the preset weight, the key name set is marked as a cold data set, and when it is judged that the key value set of the cold data set is in the cache of the preset server, the cold data set is The key-value set is moved out of the preset server's cache.

Optionally, classifying the original data set to obtain a classified data set, including:

obtaining the business scenario category corresponding to the original data set;

According to the scene category of the business scene, the original data set is classified to obtain a classified data set.

Optionally, the obtaining the evaluation index set corresponding to the key name set includes:

Obtain an access log according to the access result of the key name set within the preset time period in the database instance;

The evaluation index set of the key name set is extracted from the access information in the access log.

Optionally, calculating the weight of the key name set based on the evaluation index set includes:

The weight of the key name set is periodically calculated based on the evaluation index set according to a preset time threshold, and the weight of the key name set is updated according to the recalculation result.

Optionally, the evaluation indicator set includes access data length, access frequency, update frequency and recent access time, and the calculation of the weight of the key name set based on the evaluation indicator set regularly according to a preset time threshold includes:

An aggregation operation is performed on the regularly updated access frequency, the access data length, the most recent access time, and the update frequency to obtain the weight of the key name set.

Optionally, when judging that the key-value set of the hot data set is not in the cache of the preset server, moving the key-value set of the hot data set to the cache of the preset server, including:

If the cache does not contain the key-value set of the hot data set, extract the corresponding key-value set from the database, and save the key-value set into the cache.

Optionally, after marking the key name set as a hot data set, the method further includes:

back up the hot data set in the blockchain node;

When the server storing the hot data set is down or restarted, after confirming that the server restarts successfully, the hot data set is obtained from the blockchain node, and the hot data set is moved to the in the server's cache.

In order to solve the above problems, the present invention also provides a cold and heat data processing device, the device includes:

an acquisition module for acquiring an original data set, classifying the original data set, and obtaining a classified data set;

a query module, used to query the key name set of the classified data set, and obtain the evaluation index set corresponding to the key name set;

A calculation module, used for calculating the weight of the key name set based on the evaluation index set, and evaluating whether the weight reaches a preset weight;

The processing module is configured to mark the key name set as a hot data set if the weight reaches the preset weight, and determine that the key value set of the hot data set is not in the cache of the preset server, The key value set of the hot data set is moved to the cache of the preset server. If the weight does not reach the preset weight, the key name set is marked as a cold data set, and the key value of the cold data set is judged When the set is stored in the cache of the preset server, the key-value set of the cold data set is moved out of the cache of the preset server.

In order to solve the above problems, the present invention also provides an electronic device, the electronic device includes:

at least one processor: and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the above-described method for processing hot and cold data.

In order to solve the above problems, the present invention also provides a computer-readable storage medium, comprising a storage data area and a storage program area, wherein the storage data area stores created data, and the storage program area stores a computer program, wherein the When the computer program is executed by the processor, the above-mentioned cold and hot data processing method is realized.

The cold and hot data processing method provided by the embodiment of the present invention obtains the key name of the data, calculates the weight of the key name based on the evaluation index set corresponding to the key name, and defines the degree of hot and cold data according to the weight judgment result of the key name. , and perform corresponding operations according to the degree of hot and cold data. The embodiment of the present invention stores the hot data in the cache by performing different storage processing on the hot and cold data, so as to realize more efficient access to the data and ensure the service performance of the system. Move cold data out of the cache, reduce the waste of cache resources, and reduce the cost of data storage.

Description of drawings

1 is a schematic flowchart of a method for processing cold and hot data according to an embodiment of the present invention;

FIG. 2 is a schematic block diagram of a cold and heat data processing apparatus according to an embodiment of the present invention;

3 is a schematic diagram of an internal structure of an electronic device for implementing a method for processing cold and hot data according to an embodiment of the present invention;

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Embodiments of the present application provide a method for processing cold and heat data. The execution body of the cold and hot data processing method includes, but is not limited to, at least one of electronic devices that can be configured to execute the method provided by the embodiments of the present application, such as a server and a terminal. In other words, the cold and hot data processing method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.

Referring to FIG. 1 , it is a schematic flowchart of a method for processing cold and heat data according to an embodiment of the present invention. In this embodiment, the cold and heat data processing method includes:

S1. Obtain an original data set, and classify the original data set to obtain a classified data set.

In a preferred embodiment of the present invention, the original data set may be a set of all business data existing in the configuration system under a specific application scenario, and the business data may be obtained by interacting with each business system to access data sources. The specific application scenarios include banking, insurance, etc. For example, the business data of the configuration system of the bank includes liability business data, intermediate business data, asset business data, and the like.

Further, the described original data set is classified to obtain a classified data set, including:

obtaining the business scenario category corresponding to the original data set;

Classify the original data set according to the business scenario category to obtain a classified data set.

In this embodiment of the present invention, the business parameters in the original data set can be collected, and the business parameters can be decoupled to obtain the information exchange process and logical framework information of the business scenario reflected in the original data set, and the information exchange process and logical framework information is loaded into Storm for calculation, and the business scenario category corresponding to the original data set is restored.

Further, in the embodiment of the present invention, the original data set is classified according to the business scenario category to obtain a classified data set. If the business scenarios corresponding to the business scenario restoration of the debt business data include deposit business scenarios, loan business scenarios, inter-bank business scenarios, etc., the debt business data is divided into deposit business data, loan business data, inter-bank business data, etc.

S2. Query the key name set of the classification data set, and obtain the evaluation index set corresponding to the key name set.

In one embodiment of the present invention, the data attribute of the classified data set can be identified by js script, and according to the data attribute, the corresponding key name of the classified data set in the database table is queried by using the select query command based on the SQL statement.

A preferred embodiment of the present invention extracts the evaluation index set of the key name set from the access information in the access log by obtaining the access log of the key name set within a preset time period in the database instance by using a python script. The evaluation indicator set includes evaluation indicators such as access data length, access frequency, update frequency, and access data length at the most recent access time. In one embodiment of the present invention, the preset time period may be 30 days, and the preset time period is not limited to the values listed in this embodiment, and can also be set according to actual needs.

S3. Calculate the weight of the key name set based on the evaluation index set.

Specifically, the embodiment of the present invention performs an aggregation operation on the access frequency, access data length, recent access time, and update frequency to obtain the weight of the key name set. Preferably, in the embodiment of the present invention, the aggregation operation is to sequentially calculate the evaluation index set of the key name set by using a preset weight aggregation function to obtain the weight of the key name set. In one embodiment of the present invention, the weight aggregation function may be:

Wherein, the w represents the weight, n represents the number of evaluation indicators in the evaluation indicator set, λ is a hyperparameter, representing the influence ability of each evaluation indicator, f represents the evaluation indicator, and in the embodiment of the present invention, the λ may be 0.1 , other embodiments of the present invention are not limited to the values listed above, and can also be set according to actual needs.

Further, in the embodiment of the present invention, the evaluation index set of the key name set will change in real time with the access of the key name set in the database instance, so the weight of the key name set needs to be based on the real-time changing evaluation index. set for real-time computation. However, in order to save calculation loss and calculation delay caused by real-time calculation, one embodiment of the present invention may preset a time threshold, periodically calculate the weight of the key name set based on the evaluation index set, and calculate the weight of the key name set according to the recalculated result. The weight of the key name set is updated to facilitate subsequent corresponding processing of the classified data set. In one embodiment of the present invention, the time threshold may be 7 days. The setting of the time threshold is not limited to the values listed in this embodiment, and may also be set according to actual needs.

S4. Evaluate whether the weight reaches a preset weight.

In a preferred embodiment of the present invention, the preset weight (Fix Weight) may be comprehensively set according to the memory threshold of Redis. In one embodiment of the present invention, the preset weight may be 0.7. When the weight of the key name set is not less than 0.7, the weight of the key name set reaches the preset weight. When the weight of the key name set is less than 0.7, the weight of the key name set does not reach the preset weight.

In one embodiment of the present invention, if the weight reaches the preset weight, S5 is performed to mark the key name set as a hot data set.

In a preferred embodiment of the present invention, when the weight evaluation result of the key name set reaches the preset weight, it indicates that the key name set has a high access rate on the database instance, and at this time, the key name set is marked is a hot dataset.

In a preferred embodiment of the present invention, after step S5, the method further includes: backing up the hot data set in the blockchain node, when the server storing the hot data set crashes or restarts, after confirming that the server restarts successfully , obtain the hot data set from the blockchain node, and re-move it to the cache of the server.

In a preferred embodiment of the present invention, the cache refers to a data cache layer based on Redis.

S6. Determine whether the key-value set of the hot data set is in the cache of the preset server.

In a preferred embodiment of the present invention, the set command of the Redis is used to determine whether the key-value set of the hot data set is in the cache of the preset server.

When the key-value set of the hot data set is in the cache of the preset server, S11 is performed, and the key-value set of the hot data set is not processed.

When the key-value set of the hot data set is not in the cache of the preset server, perform S7 to move the key-value set of the hot data set to the cache of the preset server.

In a preferred embodiment of the present invention, the key-value set corresponding to the hot data set is extracted from the database and stored in the cache.

Further, in another embodiment of the present invention, if the weight does not reach the preset weight, S8 is performed to mark the key name set as a cold data set.

In a preferred embodiment of the present invention, when the weight evaluation result of the key name set does not reach the preset weight, it means that the access popularity of the key name set on the database instance is low, and at this time, the key name set is Sets are marked as cold datasets.

In a preferred embodiment of the present invention, after step S8, the method further includes: periodically calculating the weight of the key name set based on the evaluation index set according to a preset time threshold, and updating the weight of the key name set according to the recalculation result. Weights. The weight of the key name set is updated according to the time threshold, and the update basis is based on the updated access frequency, the length of the access data, the latest access time and the update frequency, performing an aggregation operation to obtain the weight of the key name set, which is convenient for The classification data set is then processed accordingly.

S9. Determine whether the key-value set of the cold data set is in the cache of the preset server.

In a preferred embodiment of the present invention, the set command of the Redis is used to determine whether the key-value set of the cold data set is in the cache of the preset server.

When the key-value set of the cold data set is not in the cache of the preset server, S11 is executed, and the key-value set of the cold data set is not processed.

When the key-value set of the cold data set is in the cache of the preset server, perform S10 to move the key-value set of the cold data set out of the cache of the preset server.

In a preferred embodiment of the present invention, the key-value set corresponding to the cold data set contained in the cache is deleted.

The cold and hot data processing method provided by the embodiment of the present invention obtains the key name of the data, calculates the weight of the key name based on the evaluation index set corresponding to the key name, and defines the degree of hot and cold data according to the weight judgment result of the key name. , and perform corresponding operations according to the degree of hot and cold data. The embodiment of the present invention stores the hot data in the cache by performing different storage processing on the hot and cold data, so as to realize more efficient access to the data and ensure the service performance of the system. Move cold data out of the cache, reduce the waste of cache resources, and reduce the cost of data storage.

As shown in FIG. 2 , it is a schematic diagram of a module of the cold and heat data processing device of the present invention.

The cold and heat data processing apparatus 100 of the present invention can be installed in an electronic device. According to the realized functions, the cold and heat data processing apparatus may include an acquisition module 101 , a query module 102 , a calculation module 103 and a processing module 104 . The modules described in the present invention can also be called units, which refer to a series of computer program segments that can be executed by the electronic device processor and can perform fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The obtaining module 101 is configured to obtain an original data set, classify the original data set, and obtain a classified data set;

In a preferred embodiment of the present invention, the original data set may be a set of all business data existing in the configuration system under a specific application scenario, and the business data may be obtained by interacting with each business system to access data sources. The specific application scenarios include banking, insurance, etc. For example, the business data of the configuration system of the bank includes liability business data, intermediate business data, asset business data, and the like.

The obtaining module 101 is configured to classify the original data set to obtain a classified data set, including obtaining a business scene category corresponding to the original data set, and classifying the original data set according to the business scene category , to get a classification dataset.

In this embodiment of the present invention, the business parameters in the original data set can be collected, and the business parameters can be decoupled to obtain the information exchange process and logical framework information of the business scenario reflected in the original data set, and the information exchange process and logical framework information is loaded into Storm for calculation, and the business scenario category corresponding to the original data set is restored.

Further, in the embodiment of the present invention, the original data set is classified according to the business scenario category to obtain a classified data set. If the business scenarios corresponding to the business scenario restoration of the debt business data include deposit business scenarios, loan business scenarios, inter-bank business scenarios, etc., the debt business data is divided into deposit business data, loan business data, inter-bank business data, etc.

The query module 102 is configured to query the key name set of the classified data set, and obtain the evaluation index set corresponding to the key name set;

The query module 102 can identify the data attribute of the classified data set through js script, and according to the data attribute, use the select query command based on the SQL statement to query the corresponding key name of the classified data set in the database table.

A preferred embodiment of the present invention extracts the evaluation index set of the key name set from the access information in the access log by obtaining the access log of the key name set within a preset time period in the database instance by using a python script. The evaluation indicator set includes evaluation indicators such as access data length, access frequency, update frequency, and access data length at the most recent access time. In one embodiment of the present invention, the preset time period may be 30 days, and the preset time period is not limited to the values listed in this embodiment, and can also be set according to actual needs.

The calculation module 103 is configured to calculate the weight of the key name set based on the evaluation index set, and evaluate whether the weight reaches a preset weight;

The calculation module 103 performs an aggregation operation on the access frequency, access data length, latest access time, and update frequency to obtain the weight of the key name set. Preferably, in the embodiment of the present invention, the aggregation operation is to sequentially calculate the evaluation index set of the key name set by using a preset weight aggregation function to obtain the weight of the key name set. The calculation module 103 uses the following method to calculate the weight of the key name set:

Wherein, the w represents the weight, n represents the number of evaluation indicators in the evaluation indicator set, λ is a hyperparameter, representing the influence ability of each evaluation indicator, f represents the evaluation indicator, and in the embodiment of the present invention, the λ may be 0.1 , other embodiments of the present invention are not limited to the values listed above, and can also be set according to actual needs.

Further, in the embodiment of the present invention, the evaluation index set of the key name set will change in real time with the access of the key name set in the database instance, so the weight of the key name set needs to be based on the real-time changing evaluation index. set for real-time computation. However, in order to save calculation loss and calculation delay caused by real-time calculation, one embodiment of the present invention may preset a time threshold, periodically calculate the weight of the key name set based on the evaluation index set, and calculate the weight of the key name set according to the recalculated result. The weight of the key name set is updated to facilitate subsequent corresponding processing of the classified data set. In one embodiment of the present invention, the time threshold may be 7 days. The setting of the time threshold is not limited to the values listed in this embodiment, and may also be set according to actual needs.

In a preferred embodiment of the present invention, the preset weight (Fix Weight) may be comprehensively set according to the memory threshold of Redis. In one embodiment of the present invention, the preset weight may be 0.7. When the weight of the key name set is not less than 0.7, the weight of the key name set reaches the preset weight. When the weight of the key name set is less than 0.7, the weight of the key name set does not reach the preset weight.

The processing module 104 is configured to mark the key name set as a hot data set if the weight reaches the preset weight, and determine when the key value set of the hot data set is not in the cache of the preset server , move the key value set of the hot data set to the cache of the preset server, if the weight does not reach the preset weight, mark the key name set as a cold data set, and judge the cold data set When the key-value set of the cold data set is in the cache of the preset server, the key-value set of the cold data set is moved out of the cache of the preset server.

The processing module 104 is configured to mark the key name set as a hot data set if the weight reaches the preset weight.

In a preferred embodiment of the present invention, when the weight evaluation result of the key name set reaches the preset weight, it indicates that the key name set has a high access rate on the database instance, and at this time, the key name set is marked is a hot dataset.

In a preferred embodiment of the present invention, after marking the key name set as a hot data set, the method further includes: backing up the hot data set in the blockchain node, and when the server storing the hot data set crashes Or when restarting, after confirming that the server restarts successfully, the hot data set is obtained from the blockchain node and moved to the cache of the server again.

In a preferred embodiment of the present invention, the cache refers to a data cache layer based on Redis.

Determine whether the key-value set of the hot data set is in the cache of the preset server. In a preferred embodiment of the present invention, the set command of the Redis is used to determine whether the key-value set of the hot data set is in the cache of the preset server. .

When the key-value set of the hot data set is in the cache of the preset server, the key-value set of the hot data set is not processed.

When the key-value set of the hot data set is not in the cache of the preset server, the key-value set of the hot data set is moved to the cache of the preset server. In a preferred embodiment of the present invention, the key-value set of the hot data set is extracted from the database. The key-value set corresponding to the hot data set is saved to the cache.

The processing module 104 is further configured to mark the key name set as a cold data set if the weight does not reach a preset weight.

In a preferred embodiment of the present invention, when the weight evaluation result of the key name set does not reach the preset weight, it means that the access popularity of the key name set on the database instance is low, and at this time, the key name set is Sets are marked as cold datasets.

In a preferred embodiment of the present invention, after marking the key name set as a cold data set, the method further includes: periodically calculating the weight of the key name set based on the evaluation index set according to a preset time threshold, and calculating the weight of the key name set according to the new The result of the calculation updates the weight of the set of key names. The weight of the key name set is updated according to the time threshold, and the update basis is based on the updated access frequency, the length of the access data, the latest access time and the update frequency, performing an aggregation operation to obtain the weight of the key name set, which is convenient for The classification data set is then processed accordingly.

Determine whether the key-value set of the cold data set is in the cache of the preset server. In a preferred embodiment of the present invention, the set command of the Redis is used to determine whether the key-value set of the cold data set is in the cache of the preset server. .

When the key-value set of the cold data set is not in the cache of the preset server, the key-value set of the cold data set is not processed.

When the key-value set of the cold data set is in the cache of the preset server, the key-value set of the cold data set is moved out of the cache of the preset server. The key-value set corresponding to the cold data set.

As shown in FIG. 3 , it is a schematic structural diagram of an electronic device implementing the method for processing cold and hot data according to the present invention.

The electronic device 1 may include a processor 10 , a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a thermal data processing program 12 .

Wherein, the memory 11 includes at least one type of computer-readable storage medium, and the computer-readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, Disks, CDs, etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1 . In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a pluggable mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) device equipped on the electronic device 1. , SD) card, flash memory card (Flash Card) and so on. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software installed in the electronic device 1 and various types of data, such as codes of the hot and cold data processing program 12, etc., but also can be used to temporarily store data that has been output or will be output.

In some embodiments, the processor 10 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more integrated circuits. Central processing unit (Central Processing Unit, CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and by running or executing the program or module (for example, executing the program) stored in the memory 11. Hot and cold data processing programs, etc.), and call the data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

The bus may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (extended industry standard architecture, EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to implement connection communication between the memory 11 and at least one processor 10 and the like.

FIG. 3 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation on the electronic device 1, and may include fewer or more components than those shown in the figure. components, or a combination of certain components, or a different arrangement of components.

For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) for powering the various components, preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that the power management The device implements functions such as charge management, discharge management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device 1 may also include a network interface, optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used in the electronic device 1 Establish a communication connection with other electronic devices.

Optionally, the electronic device 1 may further include a user interface, and the user interface may be a display (Display), an input unit (eg, a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display may also be appropriately called a display screen or a display unit, which is used for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

The cold and hot data processing program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, and when running in the processor 10, it can realize:

Obtain an original data set, classify the original data set, and obtain a classified data set;

query the key name set of the classification data set, and obtain the evaluation index set corresponding to the key name set;

Calculate the weight of the key name set based on the evaluation index set, and evaluate whether the weight reaches a preset weight;

If the weight reaches the preset weight, marking the key name set as a hot data set, and moving the key value set of the hot data set to the cache of the preset server;

If the weight does not reach the preset weight, the key name set is marked as a cold data set, and the key value set of the cold data set is removed from the cache of the preset server.

Further, if the modules/units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory) .

Further, the computer usable storage medium may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, and the like; using the created data, etc.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, but that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the invention.

Therefore, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the invention is to be defined by the appended claims rather than the foregoing description, which are therefore intended to fall within the scope of the claims. All changes within the meaning and range of the equivalents of , are included in the present invention. Any accompanying reference signs in the claims should not be construed as limiting the involved claims.

The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the system claims can also be realized by one unit or means by means of software or hardware. Second-class terms are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present invention.

Claims (10)

1. A method for processing cold and hot data, the method comprising:

acquiring an original data set, and classifying the original data set to obtain a classified data set;

inquiring a key name set of the classified data set, and acquiring an evaluation index set corresponding to the key name set;

calculating the weight of the key name set based on the evaluation index set, and evaluating whether the weight reaches a preset weight;

if the weight reaches the preset weight, marking the key name set as a hot data set, and when judging that the key value set of the hot data set is not in the cache of a preset server, moving the key value set of the hot data set to the cache of the preset server;

if the weight does not reach the preset weight, marking the key name set as a cold data set, and shifting the key value set of the cold data set out of the cache of the preset server when judging that the key value set of the cold data set is in the cache of the preset server.

2. A method as claimed in claim 1, wherein said classifying said original data set to obtain a classified data set comprises:

acquiring a service scene category corresponding to the original data set;

and classifying the original data set according to the service scene category to obtain a classified data set.

3. The method for processing cold and hot data according to claim 1, wherein the acquiring a set of evaluation metrics corresponding to the set of key names includes:

obtaining an access log according to an access result of the key name set in a database instance within a preset time period;

and extracting an evaluation index set corresponding to the key name set from the access information in the access log.

4. The method of cold-hot data processing according to claim 1, wherein said calculating weights for said set of key names based on said set of evaluation metrics comprises:

and calculating the weight of the key name set based on the evaluation index set according to a preset time threshold, and updating the weight of the key name set according to a recalculated result.

5. The method for cold and hot data processing according to claim 4, wherein the evaluation index set includes an access data length, an access frequency, an update frequency, and a latest access time, and the calculating of the weight of the key name set based on the evaluation index set according to a preset time threshold timing comprises:

and performing aggregation operation on the access frequency, the access data length, the latest access time and the update frequency which are updated regularly to obtain the weight of the key name set.

6. The method according to any one of claims 1 to 5, wherein the moving the key value set of the hot data set to the cache when determining that the key value set of the hot data set is not in the cache of the predetermined server comprises:

and if the cache does not contain the key value set of the hot data set, extracting the corresponding key value set from the database and storing the key value set into the cache.

7. A hot-cold data processing method according to any one of claims 1 to 5, wherein after said labeling said set of key names as a hot data set, said method further comprises:

backing up the hot data set in a block link point;

and when the server storing the hot data set is detected to be down or restarted, after the server is confirmed to be restarted successfully, acquiring the hot data set from the block chain node, and moving the hot data set to the cache of the server again.

8. A cold-hot data processing apparatus, characterized in that the apparatus comprises:

the system comprises an acquisition module, a classification module and a processing module, wherein the acquisition module is used for acquiring an original data set and classifying the original data set to obtain a classified data set;

the query module is used for querying the key name set of the classified data set and acquiring an evaluation index set corresponding to the key name set;

the calculation module is used for calculating the weight of the key name set based on the evaluation index set and evaluating whether the weight reaches a preset weight;

and the processing module is used for marking the key name set as a hot data set if the weight reaches the preset weight, shifting the key value set of the hot data set to the cache of the preset server when judging that the key value set of the hot data set is not in the cache of the preset server, marking the key name set as a cold data set if the weight does not reach the preset weight, and shifting the key value set of the cold data set out of the cache of the preset server when judging that the key value set of the cold data set is in the cache of the preset server.

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and the number of the first and second groups,

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of cold and hot data processing as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium comprising a stored data area storing created data and a stored program area storing a computer program, wherein the computer program, when executed by a processor, implements a method of cold and hot data processing according to any one of claims 1 to 7.

Images