CN115269621B - Database data consistency detection method and equipment based on examples - Google Patents
Database data consistency detection method and equipment based on examplesInfo
- Publication number
- CN115269621B CN115269621B CN202210924605.4A CN202210924605A CN115269621B CN 115269621 B CN115269621 B CN 115269621B CN 202210924605 A CN202210924605 A CN 202210924605A CN 115269621 B CN115269621 B CN 115269621B
- Authority
- CN
- China
- Prior art keywords
- data
- mdcc
- detection
- consistency detection
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明提供了一种基于实例的数据库数据一致性检测方法及设备，当某个物联实例的数据需要一致性检测时，本发明通过数据库的主节点上的MDCC主服务生成对该物联实例的数据进行数据一致性检测的指令，并将所述指令发送给数据存储节点上的MDCC子服务，所述MDCC子服务将该物联实例的数据所在的数据存储节点上与该物联实例的ID相同的数据分片的检测任务分配给同一个MDCC线程；MDCC线程基于分配到的检测任务，对与该物联实例的ID相同的数据分片上的数据进行一致性检测，以得到检测结果，可以实现当数据库集群节点宕机恢复后，集群的守护进程会启动基于物联实例的数据检测程序扫描整个数据库，以保障数据恢复的一致性，降低集群部分节点无法服务的时间，有效地提升了数据库的数据一致性检测的性能。
This invention provides an instance-based database data consistency detection method and device. When the data of a certain IoT instance needs consistency detection, this invention generates an instruction to perform data consistency detection on the data of the IoT instance through the MDCC main service on the database master node, and sends the instruction to the MDCC sub-service on the data storage node. The MDCC sub-service assigns the detection tasks of the data shards with the same ID as the IoT instance on the data storage node where the IoT instance's data is located to the same MDCC thread. The MDCC thread performs consistency detection on the data shards with the same ID as the IoT instance based on the assigned detection tasks to obtain the detection results. This enables the cluster daemon to start the IoT instance-based data detection program to scan the entire database after the database cluster nodes recover from a crash, ensuring data recovery consistency, reducing the time when some cluster nodes are unavailable, and effectively improving the performance of database data consistency detection.
Description
Technical Field
The invention relates to a database data consistency detection method and device based on an instance.
Background
With the development of the Internet of things, the distributed database is increasingly widely applied, and the reliability requirement on the data is continuously improved. Particularly, after the database cluster node is down and restored, the consistency detection of the data is very important, and the detection performance directly influences the service restoration time.
The current industry approach is a full Table walk from the root index data node to the leaf data node, and the walk object may be a full Table (Table) or a Partition of data (Partition).
Because the system storage hierarchy of the industry database class is generally a data Table (Table) - > data Partition) - > data Page (Page), the data consistency is generally based on concurrent detection of the data Table and the data Partition, and the detection efficiency is lower.
Disclosure of Invention
The invention aims to provide an instance-based database data consistency detection method and device.
In order to solve the above problems, the present invention provides an example-based database data consistency detection method, including:
The storage level of the database is sequentially set from large to small to be a data table, a data partition and a data storage block, the data of each Internet of things instance is stored into one or more corresponding data partitions, wherein each data partition is stored on a data storage node of the database, each data partition only stores all or part of data of one corresponding Internet of things instance, and the ID of each data partition is the ID of the corresponding Internet of things instance;
When a certain data fragment needs data consistency detection, an MDCC main service on a main node of a database generates an instruction for carrying out data consistency detection on the data fragment, and sends the instruction to an MDCC sub-service on a corresponding data storage node, wherein the MDCC sub-service distributes detection tasks of the data fragment, which is the same as the ID of the data fragment needing data consistency detection, on the corresponding data storage node to the same MDCC thread;
The MDCC thread carries out consistency detection on the data fragments with the same ID and needing consistency detection based on the allocated detection task so as to obtain a detection result;
and the MDCC sub-service gathers the detection result to the MDCC main service on the main node.
Further, in the above method, when a certain data fragment needs data consistency detection, an MDCC main service on a main node of the database generates an instruction for performing data consistency detection on the data fragment, and sends the instruction to an MDCC sub-service on a corresponding data storage node, including:
The MDCC master service on the master node of the database accesses the metadata of the master node to acquire the data fragments needing to be subjected to data consistency detection and the data storage nodes where the data fragments are positioned;
The MDCC main service on the main node of the database generates an instruction for carrying out data consistency detection on the data fragments needing to be subjected to data consistency detection based on the data fragments needing to be subjected to data consistency detection and the data storage node where the data fragments are located, and sends the instruction to the MDCC sub service on the data storage node where the data fragments needing to be subjected to data consistency detection are located.
Further, in the above method, the MDCC sub-service allocates the detection task of the data fragment on the corresponding data storage node, which is the same as the ID of the data fragment that needs to be subjected to data consistency detection, to the same MDCC thread, including:
the MDCC sub-service hashes the IDs of the data fragments needing to be subjected to data consistency detection on the corresponding data storage nodes to obtain corresponding hash values;
The MDCC sub-service distributes detection tasks of the data fragments with the same hash value to the same MDCC thread.
Further, in the above method, the MDCC sub-service allocates the detection task of the data fragment on the corresponding data storage node, which is the same as the ID of the data fragment that needs to be subjected to data consistency detection, to the same MDCC thread, including:
the MDCC sub-service hashes the ID of the data fragment needing to be subjected to data consistency detection on the corresponding data storage node and the data partition to which the data fragment needing to be subjected to data consistency detection belongs so as to obtain a corresponding hash value;
The MDCC sub-service distributes detection tasks of the data fragments with the same hash value to the same MDCC thread.
Further, in the above method, the MDCC thread performs consistency detection on data on the data slices of the same ID that need consistency detection based on the allocated detection task, so as to obtain a detection result, including:
The MDCC thread carries out binary quick traversal detection on the data fragments needing consistency detection of the same ID according to the sequence of the key values based on the allocated detection tasks so as to obtain detection results including whether the key values are ordered, whether the data are complete and whether the time stamps of the data are reasonable.
According to another aspect of the present invention, there is also provided an instance-based database data consistency detection apparatus including:
The device is used for sequentially setting the storage hierarchy of the database from large to small into a data table, a data partition and a data storage block, and storing the data of each Internet of things instance into one or more corresponding data partitions, wherein each data partition is stored on a data storage node of the database, each data partition only stores all or part of data of one corresponding Internet of things instance, and the ID of each data partition is the ID of the corresponding Internet of things instance;
The MDCC main service on the main node is used for generating an instruction for carrying out data consistency detection on a certain data fragment when the data fragment needs the data consistency detection, and sending the instruction to the MDCC sub service on the corresponding data storage node;
the MDCC sub-service is used for distributing detection tasks of the data fragments, which are the same as the IDs of the data fragments to be subjected to data consistency detection, on the corresponding data storage nodes to the same MDCC thread, and summarizing detection results of the MDCC thread to the MDCC main service on the main node;
And the MDCC thread is used for carrying out consistency detection on the data fragments with the same ID and needing the consistency detection based on the allocated detection task so as to obtain a detection result.
Further, in the device, the MDCC main service on the main node of the database is used for accessing the metadata of the main node to acquire the data fragments needing to be subjected to data consistency detection and the data storage node where the data fragments are located, generating an instruction for carrying out data consistency detection on the data fragments needing to be subjected to data consistency detection based on the data fragments needing to be subjected to data consistency detection and the data storage node where the data fragments are located, and sending the instruction to the MDCC sub service on the data storage node where the data fragments needing to be subjected to data consistency detection.
Further, in the above device, the MDCC sub-service is configured to hash the IDs of the data fragments that need to be detected for data consistency on the corresponding data storage node to obtain corresponding hash values, and allocate the detection tasks of the data fragments with the same hash values to the same MDCC thread.
Further, in the above device, the MDCC sub-service is configured to hash the ID of the data partition to be detected for data consistency on the corresponding data storage node and the data partition to which the data partition to be detected for data consistency belongs, so as to obtain a corresponding hash value, and allocate the detection tasks of the data partition with the same hash value to the same MDCC thread.
Furthermore, in the above device, the MDCC thread is configured to perform binary fast traversal detection on the data fragments requiring consistency detection of the same ID according to the key value order based on the allocated detection task, so as to obtain a detection result including whether the key value is ordered, whether the data is complete, and whether the timestamp thereof is reasonable.
Compared with the prior art, the method has the advantages that the storage hierarchy of the database is sequentially set as the data table, the data partition, the data fragments and the data storage block from large to small, the data of each Internet of things instance is stored into one or more corresponding data fragments, each data fragment is stored on a data storage node of the database, each data fragment only stores all or part of data of one corresponding Internet of things instance, the ID of each data fragment is the ID of the corresponding Internet of things instance, when the data of a certain Internet of things instance needs to be detected in consistency, the MDCC main service on a main node of the database generates an instruction for detecting the data of the Internet of things instance, the instruction is sent to the MDCC sub-service on the data storage node, the MDCC sub-service distributes detection tasks of the data fragments with the same ID of the Internet of things instance on the data storage node of the Internet of things instance to the same MDCC thread, the MDCC thread can detect the data fragments with the same ID of the Internet of things instance on the basis of the allocated detection tasks, the consistency detection of the data fragments with the ID of the Internet of things instance can be ensured, the consistency of the data can be restored on the basis of the data nodes of the data of the Internet of things instance can be effectively detected, the data can be restored on the basis of the data detection results of the data of the Internet of things instance, the data can be reduced, and the consistency of the data can be effectively restored, and the cluster can not be restored.
Drawings
FIG. 1 is a schematic diagram of a data storage distributed system of an example-based database data consistency detection method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data storage block according to one embodiment of the present invention;
FIG. 3 is a schematic diagram of data slicing according to one embodiment of the present invention;
Fig. 4 is a schematic diagram of an example of an internet of things according to an embodiment of the invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
As shown in fig. 1, the present invention provides an example-based database data consistency detection method, which includes:
Step S1, sequentially setting a storage hierarchy of a database from large to small into a data table, data partitions, data fragments and data storage blocks, wherein the data of each Internet of things instance is stored into one or more corresponding data fragments (SliceTable), each data fragment is stored on a data storage node (Ranger) of the database, each data fragment (SliceTable) only stores all or part of the data of one corresponding Internet of things instance, and the ID of each data fragment is the ID of the corresponding Internet of things instance;
The data of the Internet of things instance can comprise an ID, a time stamp and a data value of the Internet of things instance;
wherein one data table may include a plurality of data partitions, one data partition includes a plurality of data fragments, and one data fragment includes a plurality of data storage blocks.
Step S2, when a certain data fragment (SliceTable) needs data consistency detection, an MDCC main service on a main node (Master) of a database generates an instruction for carrying out data consistency detection on the data fragment (SliceTable) and sends the instruction to an MDCC sub-service on a corresponding data storage node (range), wherein the MDCC sub-service distributes detection tasks of the data fragment which is the same as the ID of the data fragment (SliceTable) on the corresponding data storage node (range) to the same MDCC thread;
The data consistency detection service (Maxtable Data Consistency Checker) of the invention is hereinafter referred to as MDCC, the MDCC main service operates on a Master node, and the MDCC sub-service operates on each ranger node;
the detection tasks of the data fragments belonging to the same Internet of things instance can be divided into a group through the ID of the data fragments, namely the ID of the Internet of things instance, and the group of detection tasks are distributed to the same MDCC thread;
step S3, the MDCC thread carries out consistency detection on the data fragments (SliceTable) which need to be subjected to consistency detection and have the same ID based on the allocated detection task so as to obtain a detection result;
and S4, the MDCC sub-service gathers the detection result to the MDCC main service on the main node (Master).
The Maxtable of the invention is a mass data storage and processing distributed system, which solves the database requirements of high concurrency, large data volume and low delay reading and writing. And supporting rich class database query functions, including creation and use of secondary indexes. The method supports the full memory calculation of data, has quick response time of data reading and writing, can process a large number of concurrent requests, and manages mass data. The expandability is good, capacity expansion only needs to be achieved by adding machines in the cluster, any node fails, and system paralysis and data loss are avoided.
Because the system storage hierarchy of the existing database class is generally a data Table (Table) - > data Partition (Partition) - > data Page (Page), the data consistency is generally based on concurrent detection of the data Table and the data Partition, and the granularity of the detected data objects is larger.
The data storage hierarchy of Maxtable of the invention is a data Table (Table) - > data partition (Table) - > data slice (SliceTable) - > data storage Block (Block),
The data storage Block (Block) is the minimum basic unit of Maxtable bottom layer storage, the size of the data storage Block can be configured into storage blocks with different sizes of 64KB, 128KB, 1MB and the like according to requirements, multiple lines of data are stored in the blocks, the data are ordered according to line key values, and the format of the data storage Block is shown in figure 2.
One data partition (SliceTable) is composed of 64 storage blocks, each Block can be connected with each other according to meta information of a Block header, sliceTable is a minimum basic unit of load balancing and multi-copy disaster recovery fault tolerance of each range (data storage node) of a Maxtable cluster, and a storage format of the minimum basic unit is shown in fig. 3.
The data Table (Table), data Partition (Table) and data storage Block (Block) of the present invention basically correspond to the concepts of the data Table (Table), data Partition (Partition) and data Page (Page) in the existing database.
However, the inventive data slicing (SliceTable) is different from the Page and Partition concepts of the existing database, and the inventive data slicing (SliceTable) is larger than both the data pages (pages) and the inventive data storage blocks (blocks) in the existing database and smaller than the data partitions (partitions and tableets) of the existing database from the storage size perspective.
From the data storage allocation logic, the data slicing (SliceTable) of the invention is a basic unit based on the storage allocation of the Internet of things instance, and one data slicing (SliceTable) can only store the data of one Internet of things instance. The data for an instance of an Internet of things is stored by one or more data fragments (SliceTable).
The data storage of the Internet of things instance is composed of a series of SliceTable with the ID of the Internet of things instance as an index, and the basic structure is shown in figure 4.
As shown in FIG. 1, the Maxtable system of the present invention has two roles, master and Ranger. The Master-slave in fig. 1 is a backup Master node, and after the Master is down, the Master-slave can serve as the Master to continue to provide service.
The Master node is responsible for managing the Ranger and providing global services such as load balancing, schema management, service restoration after the Ranger is down, and the like.
Master (Master node) has two main responsibilities:
1. management element operations such as building a table, changing a table structure, etc.;
2. And managing the range (data storage node), detecting the working state of the range (data storage node), adjusting the data of the range service to realize load balancing, and carrying out service migration when the range fails.
Ranger (data storage node) is a unit for providing services, and a working machine can deploy a plurality of Ranger services to fully utilize the processing capacity of the multi-core cpu. Each Ranger manages SliceTable data within a certain range of the user table, and provides memory cache at the same time, and the latest hot spot SliceTable is loaded into the memory to provide service for outside.
In the data storage system of the internet of things device Maxtable provided by the invention, each internet of things device accessed to Maxtable is called an internet of things instance, each instance has an ID of a globally unique internet of things instance, and each internet of things device can upload instance state data according to a certain time interval. The internet of things device may be, for example, a camera, an industrial robot, a garbage truck, etc. When the internet of things equipment is garbage collection, example state data are uploaded according to a certain time interval, and can be data such as vehicle speed, mileage, oil consumption and the like.
According to the method, the storage hierarchy of the database is sequentially set to be a data table, a data partition and a data storage block from large to small, data of each Internet of things instance is stored into one or more corresponding data partitions, each data partition is stored on a data storage node of the database, each data partition only stores all or part of data of one corresponding Internet of things instance, the ID of each data partition is the ID of the corresponding Internet of things instance, when the data of one Internet of things instance needs to be detected in consistency, an MDCC main service on a main node of the database generates an instruction for detecting the data of the Internet of things instance, the instruction is sent to an MDCC sub-service on the data storage node, the MDCC sub-service distributes detection tasks of the data partitions, which are identical to the IDs of the Internet of things instance, on the data storage node of the Internet of things instance to the same MDCC thread, the MDCC thread detects the data of the data partitions, which are identical to the IDs of the Internet of things instance, on the basis of things instance, the ID of the data partitions is identical to the ID of the Internet of things instance, the data of things instance is detected in consistency, when the data of one Internet of things instance needs to be detected in consistency, the data of things instance is detected, the data of the Internet of things instance can be restored, the data of the data instance can be effectively can be achieved, and the cluster can not be restored, and the cluster can be restored.
In an embodiment of the method for detecting data consistency of a database based on an instance, step S2, when a certain data slice (SliceTable) needs data consistency detection, an MDCC main service on a main node (Master) of the database generates an instruction for detecting data consistency of the data slice (SliceTable), and sends the instruction to an MDCC sub-service on a corresponding data storage node (range), includes:
Step S211, the MDCC Master service on the Master node (Master) of the database accesses the metadata of the Master node (Master) to acquire the data fragments (SliceTable) required to be subjected to data consistency detection and the data storage node (Ranger) nodes where the data fragments are positioned;
In step S212, the MDCC main service on the main node (Master) of the database generates an instruction for performing data consistency detection on the data fragment (SliceTable) to be subjected to data consistency detection based on the data fragment (SliceTable) to be subjected to data consistency detection and the data storage node (range) node where the data fragment is located, and sends the instruction to the MDCC sub service on the data storage node (range) where the data fragment (SliceTable) to be subjected to data consistency detection is located.
Here, the metadata of the access Master node (Master) of the MDCC Master service is used to obtain the data fragments (SliceTable) needing to be subjected to data consistency detection and the data storage node (range) node where the data fragments (SliceTable) needing to be subjected to data consistency detection, so that the instructions can be accurately and efficiently sent to the MDCC sub-service on the data storage node (range) where the data fragments (SliceTable) needing to be subjected to data consistency detection are located.
In an embodiment of the method for detecting database data consistency based on an instance of the present invention, step S2, the MDCC sub-service allocates the detection task of the data fragment on the corresponding data storage node (range) that is the same as the ID of the data fragment (SliceTable) that needs to perform data consistency detection to the same MDCC thread, including:
step S221, the MDCC sub-service hashes the IDs of the data fragments (SliceTable) on the corresponding data storage node (range) that need to perform data consistency detection, so as to obtain corresponding hash values;
In step S222, the MDCC sub-service allocates the detection tasks of the data slices with the same hash value to the same MDCC thread.
The method comprises the steps that an MDCC main service on a main node (Master) of a database sends an instruction for detecting data consistency of the data fragments to an MDCC sub-service on a data storage node (Ranger) where the data fragments are located, the MDCC sub-service on the data storage node (Ranger) where the data fragments are located carries out hash classification according to an ID (identity) of the data fragments recorded by a first Block of the data fragments, detection tasks of the data fragments with the same hash value are distributed to the same MDCC thread, and therefore granularity and concurrent design of data processing of the MDCC thread are very fine, and high-performance and high-reliability service applied on a line of a large-scale Internet of things data storage scene is guaranteed.
In an embodiment of the method for detecting database data consistency based on an instance of the present invention, step S2, the MDCC sub-service allocates the detection task of the data fragment on the corresponding data storage node (range) that is the same as the ID of the data fragment (SliceTable) that needs to perform data consistency detection to the same MDCC thread, including:
step S231, the MDCC sub-service hashes the ID of the data partition (SliceTable) on the corresponding data storage node (range) that needs to perform data consistency detection and the data partition to which the data partition belongs, so as to obtain a corresponding hash value;
in step S232, the MDCC sub-service allocates the detection tasks of the data fragments with the same hash value to the same MDCC thread.
When the master node is to detect the data consistency of each data storage node, the data fragments to be detected can be classified through the data partition and the hash two-layer index and then distributed to the corresponding threads in each node for parallel detection. As SliceTable is stored in different Ranger nodes according to the IDs of the data partition and the Internet of things instance, and the data partition can be set to be within hundred megabytes or even ten megabytes, the granularity and concurrent design of data processing of the MDCC thread are very fine, and the high-performance and high-reliability service of online application of a large-scale Internet of things data storage scene is ensured.
In an embodiment of the method for detecting consistency of database data based on an instance of the present invention, step S3, the MDCC thread performs consistency detection on data on a data fragment (SliceTable) with the same ID that needs consistency detection based on the allocated detection task, so as to obtain a detection result, including:
The MDCC thread performs binary quick traversal detection on the data fragments (SliceTable) requiring consistency detection of the same ID according to the key value size sequence based on the allocated detection task so as to obtain a detection result comprising whether the key value is ordered, whether the data is complete and whether the time stamp is reasonable.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (4)
1. An instance-based database data consistency detection method, comprising:
The method comprises the steps of sequentially setting a storage level of a database from large to small into a data table, data partitions, data fragments and data storage blocks, storing data of each object connection instance into one or more corresponding data fragments, wherein each data fragment is stored on a data storage node of the database, each data fragment only stores all or part of data of one corresponding object connection instance, and an ID (identity) of each data fragment is an ID of the corresponding object connection instance;
When a certain data fragment needs data consistency detection, an MDCC main service on a main node of a database generates an instruction for carrying out data consistency detection on the data fragment, and sends the instruction to an MDCC sub-service on a corresponding data storage node, wherein the MDCC sub-service distributes detection tasks of the data fragment, which is the same as the ID of the data fragment needing data consistency detection, on the corresponding data storage node to the same MDCC thread;
The MDCC thread carries out consistency detection on the data fragments with the same ID and needing consistency detection based on the allocated detection task so as to obtain a detection result;
the MDCC sub-service gathers the detection result to the MDCC main service on the main node;
when a certain data fragment needs data consistency detection, an MDCC main service on a main node of a database generates an instruction for carrying out data consistency detection on the data fragment, and sends the instruction to an MDCC sub-service on a corresponding data storage node, wherein the method comprises the following steps:
The MDCC master service on the master node of the database accesses the metadata of the master node to acquire the data fragments needing to be subjected to data consistency detection and the data storage nodes where the data fragments are positioned;
The MDCC main service on the main node of the database generates an instruction for carrying out data consistency detection on the data fragments needing to be subjected to data consistency detection based on the data fragments needing to be subjected to data consistency detection and the data storage node where the data fragments are positioned, and sends the instruction to the MDCC sub service on the data storage node where the data fragments needing to be subjected to data consistency detection are positioned;
The MDCC sub-service distributes the detection task of the data fragment which is the same as the ID of the data fragment which needs to be subjected to data consistency detection on the corresponding data storage node to the same MDCC thread, and the method comprises the following steps:
the MDCC sub-service hashes the IDs of the data fragments needing to be subjected to data consistency detection on the corresponding data storage nodes to obtain corresponding hash values;
The MDCC sub-service distributes detection tasks of the data fragments with the same hash value to the same MDCC thread;
or the MDCC sub-service distributes the detection task of the data fragment which is the same as the ID of the data fragment which needs to be subjected to data consistency detection on the corresponding data storage node to the same MDCC thread, including:
the MDCC sub-service hashes the ID of the data fragment needing to be subjected to data consistency detection on the corresponding data storage node and the data partition to which the data fragment needing to be subjected to data consistency detection belongs so as to obtain a corresponding hash value;
The MDCC sub-service distributes detection tasks of the data fragments with the same hash value to the same MDCC thread.
2. The instance-based database data consistency detection method according to claim 1, wherein the MDCC thread performs consistency detection on data on the data slices requiring consistency detection for the same ID based on the assigned detection task to obtain a detection result, comprising:
The MDCC thread carries out binary quick traversal detection on the data fragments needing consistency detection of the same ID according to the sequence of the key values based on the allocated detection tasks so as to obtain detection results including whether the key values are ordered, whether the data are complete and whether the time stamps of the data are reasonable.
3. An instance-based database data consistency detection apparatus, comprising:
the device is used for sequentially setting the storage hierarchy of the database from large to small, wherein the data table comprises a plurality of data partitions, one data partition comprises a plurality of data partitions, the data of each data partition comprises a plurality of data storage blocks, the data of each data partition is stored in one or more corresponding data partitions, each data partition is stored on a data storage node of the database, each data partition only stores all or part of data of one corresponding data partition, and the ID of each data partition is the ID of the corresponding data partition;
The MDCC main service on the main node is used for generating an instruction for carrying out data consistency detection on a certain data fragment when the data fragment needs the data consistency detection, and sending the instruction to the MDCC sub service on the corresponding data storage node;
the MDCC sub-service is used for distributing detection tasks of the data fragments, which are the same as the IDs of the data fragments to be subjected to data consistency detection, on the corresponding data storage nodes to the same MDCC thread, and summarizing detection results of the MDCC thread to the MDCC main service on the main node;
The MDCC thread is used for carrying out consistency detection on the data fragments which need to be subjected to the consistency detection and have the same ID based on the allocated detection task so as to obtain a detection result;
The MDCC main service on the main node of the database is used for accessing the metadata of the main node to acquire the data fragments needing to be subjected to data consistency detection and the data storage nodes where the data fragments are positioned; generating an instruction for carrying out data consistency detection on the data fragments needing to be subjected to the data consistency detection based on the data fragments needing to be subjected to the data consistency detection and the data storage nodes where the data fragments are located, and sending the instruction to an MDCC sub-service on the data storage nodes where the data fragments needing to be subjected to the data consistency detection are located;
the MDCC sub-service is used for hashing the IDs of the data fragments needing to be subjected to data consistency detection on the corresponding data storage nodes to obtain corresponding hash values;
or the MDCC sub-service is used for hashing the ID of the data fragment needing to be subjected to data consistency detection on the corresponding data storage node and the data partition to which the data fragment needing to be subjected to data consistency detection belongs so as to obtain a corresponding hash value, and distributing the detection task of the data fragment with the same hash value to the same MDCC thread.
4. The instance-based database data consistency detection apparatus according to claim 3, wherein the MDCC thread is configured to perform binary fast traversal detection on the data slices requiring consistency detection of the same ID according to the order of the key values based on the assigned detection tasks, so as to obtain a detection result including whether the key values are ordered, whether the data are complete, and whether the time stamps thereof are reasonable.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210924605.4A CN115269621B (en) | 2022-08-02 | 2022-08-02 | Database data consistency detection method and equipment based on examples |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210924605.4A CN115269621B (en) | 2022-08-02 | 2022-08-02 | Database data consistency detection method and equipment based on examples |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115269621A CN115269621A (en) | 2022-11-01 |
| CN115269621B true CN115269621B (en) | 2025-10-31 |
Family
ID=83747307
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210924605.4A Active CN115269621B (en) | 2022-08-02 | 2022-08-02 | Database data consistency detection method and equipment based on examples |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115269621B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731858A (en) * | 2015-01-29 | 2015-06-24 | 无锡科技职业学院 | Multi-client-side parallel detection database content consistency system and method |
| CN105045917A (en) * | 2015-08-20 | 2015-11-11 | 北京百度网讯科技有限公司 | Example-based distributed data recovery method and device |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100496138C (en) * | 2006-07-26 | 2009-06-03 | 华为技术有限公司 | System and method for processing communication operation |
| KR101527634B1 (en) * | 2012-08-10 | 2015-06-10 | 네이버 주식회사 | Method and apparatus for providing sharding service |
| CN106096038A (en) * | 2016-06-28 | 2016-11-09 | 浪潮软件股份有限公司 | A kind of distributed database architecture under cloud computing framework |
| CN106775470B (en) * | 2016-12-08 | 2020-04-10 | 武汉斗鱼网络科技有限公司 | Data storage method and system |
| US10467365B1 (en) * | 2017-04-10 | 2019-11-05 | Cadence Design Systems, Inc. | Systems and methods for calculating common clock path pessimism for hierarchical timing analysis in an electronic design |
| CN107508901B (en) * | 2017-09-04 | 2020-12-22 | 北京京东尚科信息技术有限公司 | Distributed data processing method, apparatus, server and system |
| CN109656911B (en) * | 2018-12-11 | 2023-08-01 | 江苏瑞中数据股份有限公司 | Distributed parallel processing database system and its data processing method |
| CN109669929A (en) * | 2018-12-14 | 2019-04-23 | 江苏瑞中数据股份有限公司 | Method for storing real-time data and system based on distributed parallel database |
| WO2021207923A1 (en) * | 2020-04-14 | 2021-10-21 | 深圳市欢太科技有限公司 | Cluster expansion method and apparatus, storage medium, and electronic device |
| CN112052141B (en) * | 2020-09-02 | 2022-04-01 | 平安科技（深圳）有限公司 | Data fragment verification method and device, computer equipment and readable storage medium |
| CN113722099B (en) * | 2021-09-01 | 2025-07-04 | 烽火通信科技股份有限公司 | A method and device for processing 10 million-level data fragments |
| CN114510539B (en) * | 2022-04-18 | 2022-06-24 | 北京易鲸捷信息技术有限公司 | Method for generating and applying consistency check point of distributed database |
-
2022
- 2022-08-02 CN CN202210924605.4A patent/CN115269621B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731858A (en) * | 2015-01-29 | 2015-06-24 | 无锡科技职业学院 | Multi-client-side parallel detection database content consistency system and method |
| CN105045917A (en) * | 2015-08-20 | 2015-11-11 | 北京百度网讯科技有限公司 | Example-based distributed data recovery method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115269621A (en) | 2022-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20240370460A1 (en) | Replicating Big Data | |
| AU2017225107B2 (en) | System-wide checkpoint avoidance for distributed database systems | |
| US10229011B2 (en) | Log-structured distributed storage using a single log sequence number space | |
| CA2906511C (en) | Fast crash recovery for distributed database systems | |
| US9507843B1 (en) | Efficient replication of distributed storage changes for read-only nodes of a distributed database | |
| CN102307206B (en) | Caching method of caching system for quickly accessing virtual machine mirror image based on cloud storage | |
| CN102904949B (en) | Replica-based dynamic metadata cluster system | |
| CN103186554B (en) | Distributed data mirror method and storage back end | |
| US20150095282A1 (en) | Multi-site heat map management | |
| US12511254B2 (en) | Key-value store and file system | |
| CN109299056B (en) | A kind of method of data synchronization and device based on distributed file system | |
| CN104750757B (en) | A kind of date storage method and equipment based on HBase | |
| Liu et al. | Cfs: A distributed file system for large scale container platforms | |
| CN105740334A (en) | System and method for asynchronous and batched file creation in file system | |
| CN118860993B (en) | Metadata processing method and device based on distributed storage system | |
| CN102984256B (en) | Processing method and system for metadata based on authorization manner | |
| CN117687970A (en) | Metadata retrieval method and device, electronic equipment and storage medium | |
| Barry et al. | Implementing journaling in a linux shared disk file system | |
| CN115269621B (en) | Database data consistency detection method and equipment based on examples | |
| CN118550477B (en) | Data deduplication methods, products, computer devices and storage media | |
| WO2025256379A1 (en) | Data management | |
| Zhang et al. | Cheetah: Metadata aggregation for fast object storage without distributed ordering | |
| Klein et al. | Dxram: A persistent in-memory storage for billions of small objects | |
| Zuo et al. | Data Distribution Algorithms in Distributed File Systems | |
| CN120560825A (en) | Performance optimization method, device, computer equipment, readable storage medium and program product of power data system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |