CN118606104A - A file detection and repair method, device and system - Google Patents

Abstract

The disclosed embodiments provide a method, device and system for detecting and repairing files, and relate to the technical field of distributed file systems. The specific implementation of the method includes: receiving one or more file detection tasks; in response to the file detection task, accessing the metadata server according to the metadata server address in the endpoint configuration, and reading the metadata information of the file to be detected corresponding to the file detection task from the metadata server; comparing the metadata information of the file to be detected with the file copy stored on the corresponding service node, determining the damaged copy in each file copy, and the damage type of each damaged copy; and backing up and repairing or synchronously repairing the damaged copy according to the damage type. This implementation can shorten the detection time and computing resource consumption, improve detection efficiency, reduce system load, improve repair efficiency, cover the extreme full damage scenario, ensure the accuracy and integrity of the data content, and improve the user access experience.

Description

A file detection and repair method, device and system

Technical Field

The present disclosure relates to the technical field of distributed file systems, and in particular to a method, device and system for detecting and repairing files.

Background Art

With the continuous development of big data technology, distributed file systems (DFS) with the advantages of high security and reliability, strong scalability, high access efficiency and user-imperceptibleness have been widely used. Taking glusterFS (GFS for short) as an example, the user sends a data processing request to the GFS client through the virtual port. In response to the data processing request, the client interacts with the corresponding GFS server to perform data read/write, modify, delete and other operations.

In order to ensure the accuracy and integrity of the data, the damaged data can only be repaired using the built-in file damage repair command Heal. The repair modes of Heal include automatic scanning repair and specified path repair. Automatic scanning repair refers to regularly scanning the entire disk data, locating the damaged data and repairing it; specified path repair means that when data damage is found during user access, the path where the damaged data is located is actively reported, and GFS calls the Heal command to repair it.

However, since GFS does not have a centralized metadata management center, the automatic scanning and repair method requires a full disk scan. On the one hand, the repair process is time-consuming, which greatly increases the repair cost. On the other hand, the regular scanning method is too slow to perceive file damage. At the same time, the repair requires system resources, which increases the server load and slows down the response to normal user requests, all of which lead to poor user access experience. Faced with massive files (hundreds of millions or even larger), the specified path repair method is too random and has high uncertainty. It is not only delayed but also unable to cover all damaged data, resulting in low data integrity and accuracy.

Summary of the invention

In view of this, the embodiments of the present disclosure provide a file detection and repair method, device and system, which can solve the problems that the full disk scanning and repair takes a long time, the cost of resources required for repair is too high, the system load is heavy and the performance decreases accordingly, which affects the user's normal data access, the normal user request response is slow, and the files whose copies are all missing cannot be repaired; the specified path repair has high uncertainty and cannot cover all damaged data, resulting in low data integrity and accuracy, all of which make the repair too delayed and the user access experience poor.

To achieve the above object, according to one aspect of the present disclosure, a method for detecting and repairing a file is provided, the method being applied to a GFS service node of a GFS server, the method comprising:

Receive one or more file detection tasks;

In response to the file detection task, access the metadata server according to the metadata server address in the endpoint configuration, and read metadata information of the file to be detected corresponding to the file detection task from the metadata server;

Comparing the metadata information of the file to be detected with the file copies stored on the corresponding service node, determining the damaged copies among the file copies and the damage type of each damaged copy;

According to the damage type, the damaged copy is backed up and repaired or synchronously repaired.

According to another aspect of the present disclosure, a file detection and repair device is provided, the device is applied to a GFS service node of a GFS server, and the device includes:

A receiving module, used for receiving one or more file detection tasks;

A reading module, configured to access the metadata server in response to the file detection task according to the metadata server address in the endpoint configuration, and read metadata information of the file to be detected corresponding to the file detection task from the metadata server;

A comparison module, used to compare the metadata information of the file to be detected with the file copies stored on the corresponding service node, to determine the damaged copies among the file copies and the damage type of each damaged copy;

The repair module is used to perform backup repair or synchronous repair on the damaged copy according to the damage type.

According to another aspect of the present disclosure, a file detection and repair system is provided, including: a GFS server consisting of multiple GFS service nodes including file detection and repair devices, a GFS client, and a metadata server.

According to another aspect of the present disclosure, there is provided an electronic device, including:

Processor; and

Memory for storing programs,

The program includes instructions, which, when executed by the processor, cause the processor to execute the file detection and repair method.

According to another aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the file detection and repair method.

One or more technical solutions provided in the embodiments of the present application, by introducing a metadata server for storing metadata information of all files, can locate damaged files based on the metadata information in the subsequent detection process, and determine different repair methods according to the damage type of the damaged copy, thereby shortening the detection time and computing resource consumption, improving the detection efficiency, reducing the system load, and achieving the dual effects of reducing the detection cost and improving the system performance; and, by using concurrent repair to improve the repair speed and efficiency, while backing up and repairing to cover the extreme full damage scenario, the accuracy and integrity of the data content are guaranteed, the user's normal data access and the immediate response to the request are guaranteed, and the user's access experience is improved.

BRIEF DESCRIPTION OF THE DRAWINGS

Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

FIG1 shows a flow chart of a method for detecting and repairing a file according to an exemplary embodiment of the present disclosure;

FIG2 shows a schematic diagram of a GFS server according to an exemplary embodiment of the present disclosure;

FIG3 is a schematic diagram showing interaction between a user and a GFS according to an exemplary embodiment of the present disclosure;

FIG4 shows a flow chart of a method for triggering a file detection task according to an exemplary embodiment of the present disclosure;

FIG5 shows a schematic diagram of a file detection and repair system according to an exemplary embodiment of the present disclosure;

FIG6 shows a flow chart of a method for generating metadata information according to an exemplary embodiment of the present disclosure;

FIG7 shows a flowchart of a method for storing a file according to an exemplary embodiment of the present disclosure;

FIG8 shows a flow chart of a method for detecting a file according to an exemplary embodiment of the present disclosure;

FIG9 shows a flow chart of a method for repairing a file according to an exemplary embodiment of the present disclosure;

FIG10 shows a flowchart of a backup and restoration method according to an exemplary embodiment of the present disclosure;

FIG11 shows a flow chart of a synchronous repair method according to an exemplary embodiment of the present disclosure;

FIG12 shows a flow chart of a method for monitoring system indicators according to an exemplary embodiment of the present disclosure;

FIG13 shows a schematic block diagram of a file detection and repair device according to an exemplary embodiment of the present disclosure;

FIG. 14 shows a block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. In addition, the method embodiments may include additional steps and/or omit the steps shown. The scope of the present disclosure is not limited in this respect.

The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "in the embodiment of the present disclosure" means "at least one embodiment"; the term "another exemplary embodiment" means "at least one other embodiment". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

MurMurHash algorithm: is a widely tested and fast non-cryptographic hash function with many variants. Its name comes from two basic operations, multiply and rotate (although the algorithm actually uses shift and xor instead of rotate). MurMurHash algorithm includes multiple versions, with different versions corresponding to the same or different number of bits of hash value. For example, MurmurHash2 produces 32-bit or 64-bit hash values, MurmurHash3 can produce 32-bit or 128-bit hash values, and MurmurHash2-160 produces 160-bit hash values.

CityHash algorithm: It is a string hash algorithm released by Google. Like murmurhash, it is a non-encrypted hash algorithm. CityHash algorithms include CityHash64, CityHash128, etc., which can generate 64-bit, 128-bit or 256-bit hash values (also called hash values).

Endpoint: is an abstraction of a network address, which can be expressed as "IP address + port" or "domain name + port". It can be used for server request monitoring and/or traffic routing. For example, the server creates an endpoint object, and the network address is bound to the endpoint object. Therefore, when the server receives a service request, it can access the network address in the endpoint object.

concurrentHashMap: is a hash table that supports high-concurrency updates and queries. It guarantees high performance for both read and write operations. It (almost) does not require locking during read operations, and during write operations, it uses lock segmentation technology to lock only the segment being operated without affecting the client's access to other segments.

When data in the existing GFS is damaged, it can usually only be repaired through the built-in Heal function. Due to the lack of a centralized metadata management center, the automatic scanning of the Heal command requires a full disk scan, which takes a long time to repair and the cost of resources required for repair is too high. In addition, the mutual exclusion of file locks will increase the load on the entire system, reduce GFS performance, and affect normal users' data access. Faced with massive files, the full disk scan method is not only too slow to handle damaged files, but also unable to repair files with all copies missing, resulting in low data integrity and accuracy.

Through the file detection and repair method disclosed in the present invention, a metadata server is introduced, and for responsive detection tasks triggered by abnormal errors in response to regular detection tasks and data processing requests, metadata comparison is used to locate damaged files, which greatly shortens the detection time and reduces the detection cost. For different repair types, synchronous repair is used to improve the repair rate, and backup repair is used to provide a bottom-line repair for extreme scenarios, thereby achieving efficient file repair.

Aspects of the present disclosure are described below with reference to the accompanying drawings.

FIG. 1 shows a flow chart of a method for detecting and repairing a file according to an exemplary embodiment of the present disclosure. As shown in FIG. 1 , the method for detecting and repairing a file of the present disclosure includes the following steps:

In the disclosed embodiment, the file detection and repair method of the disclosed embodiment is applied to the GFS server. As shown in FIG2 , the GFS server includes multiple GFS service nodes, each of which includes a glusterfsd process, a BMST thread (Background Master Scanning Thread, i.e., background thread) and a thread pool composed of multiple BWST threads (Background Worker Scanning Thread, i.e., worker threads) generated by the BMST thread, and multiple BWST threads are uniformly managed by the BMST thread. The file detection and repair method of the disclosed embodiment is executed by the BMST thread and the BWST thread.

Step 101: Receive one or more file detection tasks.

In the disclosed embodiment, the BMST thread of each GFS service node receives the file detection task, and the file detection task can be periodically triggered by the glusterfsd process of each GFS service node. The periodic frequency can be selectively set as needed, for example, the periodic frequency is triggered once every hour; or, the periodic frequency can be adaptively set according to the idle period of each GFS service node. For example, the idle period of GFS service node 1 is 10:00-23:00 on weekdays, and the idle period of GFS service node 2 is 21:00-8:00 on non-working days. The periodic frequency of GFS service node 1 is set to 11:00-22:00 on weekdays, and the periodic frequency of GFS service node 2 is set to 22:00-7:00 on non-working days.

Alternatively, the file detection task may be triggered by each GFS client in response to a user's data processing request. As shown in FIG3 , the GFS includes a GFS client and a GFS server. Accordingly, as shown in FIG4 , the triggering method of the file detection task of the present disclosure includes the following steps:

Step 401: Receive a request operation and/or an operation object sent by one or more GFS clients.

In the disclosed embodiment, the user issues a data processing request through a user terminal. The types of data processing requests are various. For example, the data processing request may be a data write request, a data read request, a data modification request, a data deletion request, etc. The corresponding request operations may be a write operation, a read operation, a modification operation, a deletion operation, etc.

Furthermore, the user terminal will send the received data processing request to the GFS client. After receiving the data processing request, the GFS client will analyze and process the data processing request, and send the corresponding request operation and/or operation object to the glusterfsd process of the corresponding GFS service node. For example, for a data write request, the operation object is the file to be written and the request operation is write; for another example, for a data read request, the operation object is the file to be read and the request operation is read.

Step 402: Determine the file processing result in response to the request operation and/or the operation object, and return it to the GFS client.

In the disclosed embodiment, it should be noted that, when responding to a user data processing request, there must be at least two or more normal copies of the file, and consistency verification between the copies is implemented to ensure the accuracy of the data content of the file returned to the GFS client, thereby ensuring the correctness of the data processing result returned by the GFS client, that is, at least two or more file copies are normal copies, and the user data processing request can be correctly responded to. Therefore, once the user's data processing request is not responded to, the GFS client can determine that the number of normal copies of the request file corresponding to the data processing request is insufficient, and thus directly trigger the file detection task corresponding to the request file of the data processing request to repair the damaged copy and respond to the user.

Furthermore, in response to the request operation and/or operation object sent by the GFS client, the glusterfsd process of each GFS service node performs the request operation on the operation object, determines the file processing result and returns it to the GFS client. When the GFS client does not receive the file processing result within the preset time, or the received file processing result does not meet the requirements, it is determined that there is a problem with the file copy of the requested file, and the file detection task is triggered. In other words, before correctly responding to the user's data processing request, the file copy of the problematic requested file is first detected, the damaged copy is located and repaired, so as to avoid sending the wrong processing result to the user, or not responding for a long time, resulting in a poor user access experience, etc.

Furthermore, the detection object of the file detection task triggered by the GFS client is usually the operation object. For example, for a data read request, the operation object is the file to be read, and the detection object of the file detection task is the file to be read; for another example, for a data modification request, the operation object is the file to be modified, and the detection object of the file detection task is the file to be modified.

Step 403: Receive the file detection task triggered by the GFS client.

In the disclosed embodiments, the triggering method of the file detection task of the disclosed embodiment is used to trigger the responsive detection task of the file on the GFS server. Subsequently, the file requested by the user can be detected to determine in advance whether there is any damage and repair it in time to prevent repair delays, ensure that the data returned to the user is complete and accurate, ensure the user's normal access and normal request response, and improve the user's access service experience.

Step 102: In response to the file detection task, access the metadata server according to the metadata server address in the endpoint configuration, and read metadata information of the file to be detected corresponding to the file detection task from the metadata server.

In the embodiment of the present disclosure, as shown in FIG5 , the file detection and repair system of the present disclosure includes GFS, a metadata server and a backup server. The metadata server and the backup server are both independent of the GFS configuration. The GFS server, the GFS client, the metadata server and the backup server can be in various forms such as a physical machine, a virtual machine, a container on a physical machine, and/or a container on a virtual machine. For example, the GFS server is a physical machine, and the physical machine includes multiple disks, which can be used to configure multiple GFS service nodes. The number of disks used by a GFS service node can be selectively set as needed.

Furthermore, in order to ensure the robustness of the metadata service, the metadata server adopts the same multi-computer room and multi-node layout mode as the GFS server. The number of nodes is an odd number greater than or equal to 3, for example, 3 nodes or 5 nodes. The nodes communicate based on the rAFT protocol. The master node performs operations such as adding, modifying, and updating. After the master node completes the operation, it synchronizes it to other slave nodes. When the slave node receives commands such as adding, modifying, and updating, it can only perform operations such as forwarding and distributing, and forwarding and distributing them to the master node for processing.

In an embodiment of the present disclosure, as shown in Figure 5, in the file detection and repair system of the present disclosure, the glusterfsd process of each GFS service node creates an endpoint object for the corresponding BMST thread, and the object information includes the metadata server address of the metadata server and the backup server address of the backup server. The address format of the metadata server address and the backup server address is the domain name of the metadata server and the domain name of the backup server. In this way, the BMST thread can access the metadata server and the backup server based on the domain name. No matter how the actual IP address of the metadata server address and the backup server address changes, it is imperceptible and transparent to the BMST thread, and will not affect the GFS service node's access to the metadata server and the backup server.

Furthermore, in response to a file detection task that is triggered periodically or based on a data processing request, the GFS service node accesses the metadata server according to the metadata server address configured in the endpoint object. The metadata server stores metadata information corresponding to each file on the GFS server. As shown in FIG6 , the method for generating metadata information disclosed in the present invention includes the following steps:

In the embodiment of the present disclosure, the method for generating metadata information of the present disclosure is executed by a GFS client.

Step 601, receiving a file to be written.

In the disclosed embodiment, the GFS client receives the file to be written, and the file to be written can be obtained by a data writing request sent by a user through a terminal.

Step 602: Calculate the name hash value of the file name to be written as the storage key.

In the disclosed embodiment, the file name of each file on each service node is unique, so the corresponding name hash value is naturally unique. If the file name of the file to be written is the same as the file name of an existing file, the existing file will be directly overwritten when written.

Furthermore, the GFS client calculates the name hash value of the file name to be written according to a preset hash algorithm as the key of the metadata information of the file to be written. The hash algorithm of the name hash value can be selectively set as needed, for example, the hash algorithm of the name hash value is murmurhash algorithm, cityhash algorithm, etc.

Step 603: divide the file to be written into multiple file blocks according to a preset step size, calculate the file block hash value of each file block of the file to be written, and form a block hash array.

In the disclosed embodiment, the preset step size can be selectively set as needed. For example, if the preset step size is 1M and the file to be written is 100M, the GFS client will divide the file to be written into 100 file blocks (hereinafter referred to as "chunk").

Furthermore, the GFS client calculates the block hash value of each chunk after segmentation according to a preset hash algorithm, obtains the chunk hash value of each chunk, and forms a block hash array of the file to be written.

Furthermore, each chunk can be numbered, for example, chunk numbers are 0-99, "0" corresponds to the chunk hash value of the first chunk, "1" corresponds to the chunk hash value of the second chunk, ..., "99" corresponds to the chunk hash value of the hundredth chunk, and the chunk number serves as the offset of each chunk, which can be subsequently used for verification and checking of damaged copy detection and repair.

Step 604: Use the combination of the file name, file size, creation time, modification time, and block hash array of the file to be written as a key value.

In the disclosed embodiment, after obtaining the chunk hash value of each chunk, the GFS client combines other information of the file to be written with the chunk hash array to obtain the key value of the metadata information of the file to be written. The other information can be selectively set as needed, for example, the other information includes file size, creation time, modification time, etc.

Furthermore, the GFS client combines the file name, file size, creation time, modification time, and block hash array of the file to be written as the value of the metadata information of the file to be written.

Step 605: Determine that the key-value pair consisting of the storage key and the key value is the metadata information of the file to be written.

In the disclosed embodiment, the GFS client combines the key of the file to be written obtained in step 602 and the value of the file to be written obtained in step 604 into a key-value pair [key, value], and determines that the key-value pair [key, value] is metadata information of the file to be written.

In an embodiment of the present disclosure, through the metadata information generation method of the present disclosure, a hash calculation is performed on the file name of each file to obtain the key of the metadata information of the file; the file is then divided into multiple blocks, the hash value of each file block is calculated respectively, and the other information of the file is combined to obtain the value of the metadata information of the file; the metadata information of the file is generated by the key-value key-value pair, thereby realizing the unique identification of the metadata information of each file, and the subsequent detection process can be compared based on the metadata information without the need to compare the contents of the entire file, thereby greatly reducing the repair cost such as the time and resources required for repair.

Furthermore, after the GFS client obtains the metadata information of the file to be written, it sends the metadata information of the file to be written to the metadata server, which stores it. At the same time, the GFS client stores the file to be written to the GFS server and the backup server respectively. As shown in FIG7 , the file storage method disclosed in the present invention includes the following steps:

In the embodiment of the present disclosure, the file storage method of the present disclosure is executed by a GFS client.

Step 701: Send the metadata information of the file to be written to each service node of the metadata server.

Step 702 , receiving the writing results returned by each service node of the metadata server, and determining whether the writing results of more than half of the metadata information are successful. If yes, go to step 703 ; if not, go to step 705 .

In the disclosed embodiment, when storing metadata information, it is stored in the form of multiple copies of information, the number of copies of information is an odd number greater than 1, one of the multiple copies of information is a master copy, and the service node where the master copy is located is the master node. In the subsequent metadata information reading process, based on the same computer room priority logic, the master node where the master copy of the multiple copies of information is located will be prioritized to interact with the GFS server, etc. Among them, each service node of the metadata server stores the metadata information of each file in the form of concurrentHashMap.

Furthermore, the service nodes where the multiple information copies are located return the writing result of the metadata information of the file to be written to the GFS client, and the GFS client determines whether to perform the writing operation of the file to be written according to the number of service nodes whose writing results are successful.

Furthermore, when the writing result of more than half of the information copies is successful, it is determined that the metadata information of the file to be written is written successfully, and the GFS client then performs the writing operation of the file to be written to the GFS server and the backup server.

Step 703: Determine the multiple service nodes of the GFS server and the multiple service nodes of the backup server corresponding to the file to be written.

In the disclosed embodiment, when the metadata information of the file to be written is written successfully, the GFS client determines the node identifiers of multiple service nodes of the GFS server that can be written according to the file size of the file to be written, the number of copies, and the node capacity of each GFS service node. The number of multiple service nodes is the same as the number of copies of the file copy, which is also an odd number greater than 1. One of the multiple file copies is the primary copy, and the GFS service node corresponding to the primary copy is the primary node. In the subsequent metadata information, backup copy reading, etc., based on the same computer room priority logic, the BMST thread of the primary node where the primary copy of the file copy is located will interact with the metadata server, backup server, etc.

Similarly, when the metadata information of the file to be written is written successfully, the GFS client determines the node identifiers of multiple service nodes of the backup server that can be written based on the file size of the file to be written, the number of copies, and the node capacity of each service node of the backup server. The number of multiple service nodes is the same as the number of copies of the backup copy, which is also an odd number greater than 1. One of the multiple backup copies is the primary copy, and the service node of the backup server corresponding to the primary copy is the primary node. In the subsequent backup copy reading process, based on the same computer room priority logic, the primary node where the primary copy in the backup copy is located will interact with the GFS server first.

Furthermore, the node identifier of the service node of the GFS server corresponding to the file to be written and the node identifier of the service node of the backup server may be added to the metadata information of the file to be written.

Step 704: Send the file blocks to be written to the file to be written to the master node among the multiple service nodes of the GFS server and the master node among the multiple service nodes of the backup server respectively.

In the disclosed embodiment, the GFS client sends the node identifiers of the slave nodes other than the master node among the determined multiple service nodes and the file blocks of the files to be written to the master node among the determined multiple service nodes of the GFS server, so that the master node writes the file blocks of the files to be written to the disk, and synchronizes with other slave nodes, so that each slave node writes the file blocks of the files to be written to the disk, thereby realizing the storage of multiple file copies of the files to be written.

Furthermore, the writing of the backup copy of the backup server is the same as that of the GFS server, which will not be repeated here.

Step 705: Determine that the writing result of the metadata information of the file to be written is a writing failure, and roll back the metadata server.

In the disclosed embodiment, when the writing result of more than half of the information copies is write failure, it is determined that the writing result of the metadata information of the file to be written is write failure, and the metadata server is rolled back to the state before the metadata information of the file to be written was written.

In the disclosed embodiment, through the file storage method of the disclosed embodiment, it is possible to realize the storage of metadata information of the file to be written and the placement of file copies of the file to be written on the GFS server and the backup server. As a result, the metadata information of the file is stored in the metadata server, and the file copy is stored in the GFS server and the backup server respectively. The metadata information of the metadata server can be used for subsequent file damage detection, and there is no need to compare the contents of the entire file, which greatly reduces the repair time and repair costs such as the resources required for repair; the files written by the GFS server are used to execute subsequent data processing requests, file detection tasks and other operations, and the files written by the backup server are used to cover the extreme case where the file copy of the GFS server is completely damaged.

Step 103: Compare the metadata information of the file to be detected with the file copies stored on the corresponding service nodes to determine the damaged copies among the file copies and the damage type of each damaged copy.

In the disclosed embodiment, the metadata information of the file to be detected is compared one by one with the file copy stored on the corresponding service node, and the file size of the metadata information is first compared with the copy size for coarse screening; when the file size and the copy size are the same, the hash value of the text block is traversed for fine screening until the text blocks of each file copy are traversed, and the existence of damaged copies and the damage type of the damaged copies are determined. Subsequently, different repair methods can be adopted to repair the damaged copies based on different damage types.

Further, as shown in FIG8 , the file detection method disclosed in the present invention includes the following steps:

In the embodiment of the present disclosure, the method for determining the damaged replica and the damage type of the present disclosure is executed by each GFS service node corresponding to the file to be detected.

Step 801, searching for a copy name that is identical to the file name.

In the disclosed embodiment, for periodically triggered file detection tasks, after the glusterfsd process is started, each GFS service node will also start a BMST thread. The BMST thread of the GFS service node where the primary copy of the file to be detected is located accesses the metadata server, reads the metadata information of the file to be detected and distributes it to the BMST threads of the GFS service nodes where the slave copies are located. As a result, each GFS service node corresponding to the file to be detected can search for the copy name that is the same as the file name of the file to be detected from the node itself.

Furthermore, the BMST thread of each GFS service node distributes the metadata information of the files to be detected to each BWST thread. Each BWST thread maintains a thread queue, which is used to carry the metadata information of the files to be detected, so that each BWST thread performs file detection on each file to be detected in the queue in sequence, where the detection order of the thread queue is first-in-first-out.

Furthermore, when the BMST thread distributes metadata information of the file to be detected to each BWST thread, it will preferentially distribute the metadata information of the file to be detected to the BWST thread with the smallest queue length based on the queue length of the thread queue of each BWST thread, so as to find the copy name that is the same as the file name of the file to be detected.

Alternatively, in the disclosed embodiment, for a responsively triggered file detection task, after the glusterfsd process is started, each GFS service node will also start a BMST thread. The BMST thread of the GFS service node where the primary copy of the file to be detected is located accesses the metadata server, reads the metadata information of the file to be detected and distributes it to the BMST threads of the GFS service nodes where the slave copies are located. The BMST thread preferentially distributes the metadata information of the file to be detected to the BWST thread with the smallest queue length to find a copy name that is the same as the file name of the file to be detected.

Step 802: Compare the file size of the to-be-detected file with the copy size of the file copy corresponding to the copy name.

In the disclosed embodiment, the metadata information of the file to be detected includes the file size of the file to be detected, and the BWST thread compares the file size of the file to be detected with the copy size of the file copy corresponding to the found copy name.

Step 803 , determining whether the file size of the file to be detected is the same as the copy size of the file copy corresponding to the copy name, if yes, go to step 804 ; if no, go to step 814 .

In the disclosed embodiment, for example, the file ABC to be detected has a file copy on GPS service node 1, GPS service node 2, and GPS service node 3, respectively, which are ABC-1, ABC-2, and ABC-3. Therefore, it is determined whether the file size of file ABC is the same as the copy size of ABC-1, the copy size of ABC-2, and the copy size of ABC-3.

Step 804: Use the first replica block of the file replica as the current replica block and the first file block of the file to be detected as the current file block.

In the disclosed embodiment, the metadata information of the file to be detected also includes the hash values of each file block after the file to be detected is divided. For example, the file ABC to be detected is divided into 6 blocks according to a preset offset, and the hash values of each file block are metaABC-1, metaABC-2, metaABC-3, metaABC-4, metaABC-5, and metaABC-6 respectively.

Furthermore, when the file size of the file to be detected is the same as the copy size of the file copy corresponding to the copy name, the hash value of each text block in the metadata information of the file to be detected and the hash value of each text block in the file copy are compared one by one, specifically:

The first replica block of the file replica is used as the current replica block, and the first file block of the file to be detected is used as the current file block. For example, the BWST thread of GPS service node 1 uses the first replica block of ABC-1 as the current replica block, the BWST thread of GPS service node 2 uses the first replica block of ABC-2 as the current replica block, and the BWST thread of GPS service node 3 uses the first replica block of ABC-3 as the current replica block.

Step 805, calculating the hash value of the current replica block.

In the disclosed embodiment, for example, the BWST thread of GPS service node 1 calculates the hash value hash(ABC-1-1) of the current replica block of ABC-1, the BWST thread of GPS service node 2 calculates the hash value hash(ABC-2-1) of the current replica block of ABC-2, and the BWST thread of GPS service node 3 calculates the hash value hash(ABC-3-1) of the current replica block of ABC-3.

Step 806, determine whether the hash value of the current replica block is equal to the hash value of the current file block, if yes, go to step 807; if not, go to step 808.

In the disclosed embodiment, for example, the BWST thread of GPS service node 1 compares the hash value hash(ABC-1-1) of the current replica block with the hash value (metaABC-1) of the current text block in the metadata information, the BWST thread of GPS service node 2 compares the hash value hash(ABC-2-1) of the current replica block with the hash value metaABC-1 of the current text block in the metadata information, and the BWST thread of GPS service node 3 compares the hash value hash(ABC-3-1) of the current replica block with the hash value metaABC-1 of the current text block in the metadata information, and so on.

Step 807 , update the next replica block of the current replica block to the current replica block, and update the next file block of the current file block to the current file block, and go to step 805 .

In the disclosed embodiment, when the hash value of the current replica block is equal to the hash value of the current file block, the next replica block of the current replica block is updated to the current replica block, and the next file block of the current file block is updated to the current file block, and the comparison is performed in order. For example, the BWST thread of GPS service node 1 updates the second replica block of ABC-1 to the current replica block, the BWST thread of GPS service node 2 updates the second replica block of ABC-2 to the current replica block, and the BWST thread of GPS service node 3 updates the second replica block of ABC-3 to the current replica block.

Step 808: Determine that the current copy block is a damaged block and the file copy is a damaged copy.

In the disclosed embodiment, when the hash value of the current replica block is not equal to the hash value of the current file block, the current replica block is determined to be a damaged block and the file replica corresponding to the damaged block is a damaged replica. For example, the BWST thread of GPS service node 1 detects and finds that the fourth replica block of ABC-1 is a damaged block, and the file replica ABC-1 is a damaged replica.

Furthermore, statistics are counted from the block identifier of the damaged block of the damaged copy until the block identifier of the last copy block of the damaged copy, to obtain a damaged block identifier group. For example, the BWST thread of GPS service node 1 starts counting from the block identifier ABC-1-4 of the damaged block until the block identifier ABC-1-6 of the last copy block of the damaged copy ABC-1, to obtain a damaged block identifier group (ABC-1-4, ABC-1-5, ABC-1-6), and the file copy ABC-1 is a damaged copy.

Step 809, counting the number of damaged copies of the file to be detected, and determining whether the number of damaged copies of the file to be detected is equal to the total number of corresponding text copies; if so, go to step 810; if not, go to step 811.

Step 810: Determine that the damage type of the damaged copy is the first type.

Step 811, determine whether the number of normal copies of the file except the damaged copy is greater than or equal to the concurrency threshold, if yes, go to step 812; if not, go to step 813.

In the disclosed embodiment, the concurrency threshold is greater than or equal to 2, so that multiple replicas can be synchronized concurrently during the subsequent repair process, thereby improving the repair efficiency.

Further, preferably, the concurrency threshold is 2.

Step 812: Determine that the damage type of the damaged copy is the second type.

Step 813: Determine that the damage type of the damaged copy is the third type.

Step 814 , determining that the file copy corresponding to the copy name is the damaged copy, and going to step 809 .

In the disclosed embodiment, when the file size of the to-be-detected file is different from the copy size of the file copy corresponding to the copy name, it is determined that the file copy corresponding to the copy name is a damaged copy.

Furthermore, the GFS server creates a temporary storage area to back up the damaged copy. The size of the temporary storage area can be selectively set as needed. For example, the temporary storage area is 100G.

In the disclosed embodiment, through the file detection method of the disclosed embodiment, since the BMST thread directly reads the metadata information for comparison to determine whether the file to be detected is damaged, rather than directly scanning the entire disk, the detection cost can be greatly reduced, thereby shortening the file detection time, reducing the occupancy of detection resources, and reducing the detection cost, thereby reducing the system load and improving the system performance, ensuring the user's normal data access and normal request response, and improving the user's access experience.

Step 104: Perform backup repair or synchronous repair on the damaged copy according to the damage type.

In the disclosed embodiment, different types of damage can be repaired using different repair methods, such as directly using replica synchronization when the damage is minor, and using a backup server for backup repair when the damage is major, which can maximize the accuracy of the file data content while reducing the cost of repair.

Further, as shown in FIG9 , the file repair method of the present disclosure includes:

Step 901, determine whether the damage type of the damaged copy is the first type, if yes, go to step 902; if no, go to step 903.

Step 902: Repair the damaged copy by using a backup repair method.

In an embodiment of the present disclosure, as shown in FIG10 , the backup and repair method of the present disclosure includes the following steps:

Step 1001: access the backup server according to the backup server address in the endpoint configuration.

In the disclosed embodiment, the BMST thread of the GFS service node where the damaged copy is located reads the backup server address from the endpoint object and accesses the backup server to read the backup copy of the damaged copy.

Step 1002: Read the backup copy of the damaged copy from the backup server according to the copy name of the damaged copy.

Step 1003: Use the backup copy to overwrite the damaged copy.

In the disclosed embodiment, the BMST thread of the GFS service node where the damaged copy is located overwrites the damaged copy with the backup copy read.

In the embodiment of the present disclosure, the backup and repair method of the present disclosure is used to perform backup and repair on the damaged copy of the first type, thereby realizing an extreme scenario, that is, file repair under a complete failure of the file copy, so as to improve the repair efficiency, reduce the repair cost, ensure the accuracy of the file data content, and improve the user access experience.

Furthermore, the repaired replica can be verified to determine whether the hash value of each replica block is consistent with that in the backup server to ensure the accuracy of the repair.

Step 903: Repair the damaged copy by synchronous repair.

In an embodiment of the present disclosure, as shown in FIG11 , the synchronous repair method of the present disclosure includes the following steps:

Step 1101, determine whether the damage type of the damaged copy is the second type, if yes, go to step 1102; if no, go to step 1103.

Step 1102, repair the damaged copy by concurrent synchronization.

In the embodiment of the present disclosure, when the damage type of the damaged copy is the second type, it indicates that the number of normal copies is greater than or equal to the concurrency threshold, so there are multiple normal copies, and the data content of the damaged block can be synchronized to the damaged copy concurrently, which greatly improves the repair speed and improves the repair efficiency.

Step 11021, assign each block identifier of the damaged block identifier group, and match at least one target block identifier for each normal copy corresponding to the damaged copy.

In the disclosed embodiment, the BMST thread of the GFS service node where the damaged replica is located distributes the block identifiers in the damaged block identifier group to the BMST thread of the GFS service node where the normal replica is located, so that the BMST thread of the GFS service node where each normal replica is located matches at least one target block identifier, and sends the data content corresponding to the damaged block to the BMST thread of the GFS service node where the damaged replica is located to synchronize, so as to repair the damaged replica. The distribution method may be average distribution.

Furthermore, for the remaining block identifiers that cannot be evenly distributed, the BMST threads corresponding to the normal replicas can be randomly selected, that is, according to the number of remaining block identifiers, the BMST threads where the normal replicas of the corresponding number are located are randomly selected for distribution, and the remaining block identifiers that are distributed also belong to the target block identifiers that match the selected normal replicas. For example, the BWST thread of GPS service node 1 evenly distributes the damaged block identifier group (ABC-1-4, ABC-1-5, ABC-1-6) to the service nodes where the normal replicas ABC-2 and ABC-3 are located. When evenly distributing to the normal replicas ABC-2 and ABC-3, there will be one block identifier left. The normal replica ABC-2 is randomly selected for distribution, so the target block identifiers of the normal replica ABC-2 include ABC-1-4 and ABC-1-5, and the target block identifier of the normal replica ABC-3 is ABC-1-6.

Step 11022: Based on the target block identifier, the replica blocks synchronized from the normal replicas are used to overwrite the replica blocks corresponding to the block identifier in the damaged replica.

In the disclosed embodiment, the BMST thread of the GFS service node where each normal replica is located sends the replica block corresponding to the matched target block identifier to the BMST thread of the GFS service node where the damaged replica is located; the BMST thread of the GFS service node where the damaged replica is located uses the replica blocks synchronized from the GFS service nodes where each normal replica is located to overwrite the replica block with the corresponding block identifier in the damaged replica based on the target block identifier. For example, the BMST thread of GFS service node 2 where the normal replica ABC-2 is located sends the replica blocks of ABC-2-4 and ABC-2-5 corresponding to the matched ABC-1-4 and ABC-1-5 to the BMST thread of GFS service node 1 where the damaged replica ABC-1 is located, and the BMST thread of GFS service node 3 where the normal replica ABC-3 is located sends the replica block of ABC-3-6 corresponding to the matched ABC-1-6 to the BMST thread of GFS service node 1 where the damaged replica is located; after the BMST thread of GFS service node 1 where the damaged replica ABC-1 is located receives the replica blocks corresponding to ABC-2-4, ABC-2-5, and ABC-3-6, it uses the replica blocks corresponding to ABC-2-4, ABC-2-5, and ABC-3-6 to overwrite the replica blocks of ABC-1-4, ABC-1-5, and ABC-1-6 in the damaged replica ABC-1 respectively.

Step 1103: Repair the damaged copy by using a separate synchronization method.

In the embodiment of the present disclosure, when the damage type of the damaged replica is the third type, it indicates that the number of normal replicas is less than the asynchronous threshold, and there is only one normal replica, so the normal replica is used to synchronize with each damaged replica.

Step 11031, receiving replica blocks synchronized with normal replicas that match the respective block identifiers of the damaged block identifier group.

In the embodiment of the present disclosure, when there is only one normal replica, the BMST thread of the GFS service node where the normal replica is located sends the replica blocks corresponding to each block identifier in the damaged block identifier group to the BMST thread of the GFS service node where the damaged replica is located.

Step 11032: Use the replica blocks synchronized from the normal replica to overwrite the replica blocks corresponding to the block identifiers of the damaged block identifier group in the damaged replica.

In the disclosed embodiment, the BMST thread of the GFS service node where the damaged replica is located uses the replica blocks synchronized from the GFS service node where the normal replica is located to overwrite the replica blocks with corresponding block identifiers in the damaged replica according to the block identifiers of the damaged block identifier group. For example, the damaged block identification group of the damaged replica ABC-1 is (ABC-1-4, ABC-1-5, ABC-1-6), and the damaged block identification group of the damaged replica ABC-2 is (ABC-2-1, ABC-2-2, ABC-2-3, ABC-2-4, ABC-2-5, ABC-2-6). The BMST thread of GFS service node 3 where the normal replica ABC-3 is located synchronizes the replica blocks of ABC-3-4, ABC-3-5, and ABC-3-6 to the BMST thread of GFS service node 1 where the damaged replica ABC-1 is located. The BMST thread of GFS service node 3 where the normal replica ABC-3 is located synchronizes the replica blocks of ABC-3-1, ABC-3-2, ABC-3-3, ABC-3-4, ABC-3-5, and ABC-3-6 to the BMST thread of GFS service node 2 where the damaged replica ABC-2 is located.

After the BMST thread of GFS service node 1 where the damaged replica ABC-1 is located receives the replica blocks corresponding to ABC-3-4, ABC-3-5, and ABC-3-6, it uses the replica blocks corresponding to ABC-3-4, ABC-3-5, and ABC-3-6 to overwrite the replica blocks of ABC-1-4, ABC-1-5, and ABC-1-6 in the damaged replica ABC-1 respectively; after the BMST thread of GFS service node 2 where the damaged replica ABC-2 is located receives the replica blocks corresponding to ABC-3-1, ABC-3-2, ABC-3-3, ABC-3-4, ABC-3-5, and ABC-3-6, it uses the replica blocks corresponding to ABC-3-1, ABC-3-2, ABC-3-3, ABC-3-4, ABC-3-5, and ABC-3-6 to overwrite the replica blocks of ABC-2-1, ABC-2-2, ABC-2-3, ABC-2-4, ABC-2-5, and ABC-2-6 in the damaged replica ABC-2 respectively.

In the embodiment of the present disclosure, through the synchronous repair method of the present disclosure, the damaged copy is repaired by using the normal copy concurrently or individually to synchronously overwrite the damaged copy, which can greatly improve the repair speed, improve the repair efficiency, and ensure the accuracy of the file data content.

In the disclosed embodiments, through the file detection and repair method disclosed in the present invention, a metadata server is introduced to store metadata information of all files, so that damaged files can be located based on the metadata information in the subsequent detection process, which greatly shortens the detection time and computing resource consumption, improves the detection efficiency, reduces the system load, and achieves the dual effects of reduced detection costs and improved system performance; and, different repair methods are determined according to the damage type of the damaged copy, and concurrent repair is used to increase the repair speed and improve the repair efficiency. At the same time, backup and repair cover the extreme full damage scenario, ensuring the accuracy and completeness of the data content, guaranteeing the user's normal data access and immediate response to requests, and improving the user access experience.

Furthermore, during the detection and repair process of the file, corresponding detection logs and repair logs may be generated to record various detection information and repair information for subsequent auditing.

Furthermore, during the detection and repair process of files, the node indicators of each GFS service node are monitored to perform speed limit control in a timely manner to avoid the obvious fluctuation of GFS load affecting online access users, so that users do not feel any daily access and improve the user access experience. As shown in FIG12 , the system indicator monitoring method disclosed in the present invention includes the following steps:

Step 1201, monitor the node indicators of each GFS service node.

In the disclosed embodiment, a monitoring agent may be set on each GFS service node to monitor node indicators in real time or periodically, including CPU idle percentage, average load, CPU occupancy of the server process, and/or the number of bytes read and written per second of the server process.

Furthermore, the monitoring frequency of the periodic monitoring can be selectively set as needed, for example, the monitoring frequency is once every 3 seconds, or the monitoring frequency can be the same as the periodic frequency of the file detection task.

Step 1202, determine whether each of the node indicators exceeds the corresponding indicator threshold, if yes, go to step 1203; if no, go to step 1201.

In the disclosed embodiment, the indicator thresholds of each node indicator can be selectively set as needed. For example, the CPU idle percentage threshold is 50%. If the CPU idle percentage exceeds 50%, it means that the GFS service node has a very low idle ratio and is too busy. The average load is determined by counting the average number of CPU cores used per unit time. The unit time can be 1min, 3min, 5min, etc. The average load threshold can be 48 cores/min. The CPU occupancy threshold of the server process is 60%, etc.

Step 1203, suspending response to the file detection task.

In the disclosed embodiment, when any node indicator exceeds the corresponding indicator threshold, the BMST thread of the GFS service node stops sending file detection tasks and sends a stop signal to each managed BMWT thread.

In the disclosed embodiment, each GFS service node is monitored through the system indicator monitoring method disclosed in the present invention to prevent obvious load fluctuations of GFS from affecting user access, thereby improving detection and repair efficiency while ensuring users' daily normal access and improving user access experience.

FIG. 13 is a schematic diagram of the main modules of the file detection and repair device according to an embodiment of the present disclosure. As shown in FIG. 13 , the file detection and repair device 1300 of the present disclosure is applied to a GFS service node of a GFS server, including:

The receiving module 1301 is used to receive one or more file detection tasks.

The reading module 1302 is used to access the metadata server in response to the file detection task according to the metadata server address in the endpoint configuration, and read the metadata information of the file to be detected corresponding to the file detection task from the metadata server.

The comparison module 1303 is used to compare the metadata information of the file to be detected with the file copy stored on the corresponding service node, and determine the damaged copy in each of the file copies and the damage type of each of the damaged copies.

The repair module 1304 is used to perform backup repair or synchronous repair on the damaged copy according to the damage type.

In the embodiment of the present disclosure, the interactive process of the file detection and repair system shown in FIG5 includes:

The glusterfsd process of the GFS service node periodically triggers the file detection task, or the glusterfsd process of the GFS service node receives the file detection task triggered by the GFS client;

The GFS service node sends the file detection task to the BMST thread;

The BMST thread responds to the file detection task and accesses the metadata server according to the metadata server address in the endpoint configuration;

The metadata server reads the metadata information of the file to be detected according to the file name of the file to be detected, and returns it to the BMST thread;

The BMST thread compares the metadata information of the file to be detected with the file copy to determine the damaged copy in the file copy and the damage type of the damaged copy;

The BMST thread performs backup repair or synchronous repair on the damaged replica according to the damage type of the damaged replica. Specifically:

The damage type of the damaged copy is the first type, accessing the backup server according to the backup server address in the endpoint configuration, reading the backup copy of the damaged copy from the backup server, and overwriting the damaged copy with the backup copy;

The damage type of the damaged replica is the second type, each block identifier of the damaged block identifier group is allocated, at least one target block identifier is matched for each normal replica corresponding to the damaged replica, and the replica blocks synchronized from each normal replica are used to overwrite the replica blocks corresponding to the block identifiers in the damaged replica;

The damage type of the damaged replica is the third type, and the replica blocks corresponding to the respective block identifiers of the damaged block identifier group in the damaged replica are overwritten with the replica blocks synchronized from the normal replica.

The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method according to the embodiments of the present disclosure.

The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

With reference to Figure 14, the structural block diagram of the electronic device 1400 that can be used as the server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and/or required herein.

As shown in FIG. 14 , the electronic device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input/output (I/O) interface 1405 is also connected to the bus 1404.

Multiple components in the electronic device 1400 are connected to the I/O interface 1405, including: an input unit 1406, an output unit 1407, a storage unit 1408, and a communication unit 1409. The input unit 1406 can be any type of device that can input information to the electronic device 1400, and the input unit 1406 can receive input digital or character information, and generate key signal input related to user settings and/or function control of the electronic device. The output unit 1407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 1408 can include but is not limited to a disk, an optical disk. The communication unit 1409 allows the electronic device 1400 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and/or the like.

The computing unit 1401 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above. For example, in some embodiments, the methods of Figures 1, 4, 6 to 12 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 1400 via ROM 1402 and/or communication unit 1409. In some embodiments, the computing unit 1401 may be configured to perform the methods of Figures 1, 4, 6 to 12 in any other appropriate manner (e.g., by means of firmware).

The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions/operations specified in the flow chart and/or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims (20)

1. A method for detecting and repairing a file, characterized in that the method is applied to a GFS service node of a GFS server, and the method comprises: Receive one or more file detection tasks; In response to the file detection task, access the metadata server according to the metadata server address in the endpoint configuration, and read metadata information of the file to be detected corresponding to the file detection task from the metadata server; Comparing the metadata information of the file to be detected with the file copies stored on the corresponding service node, determining the damaged copies among the file copies and the damage type of each damaged copy; According to the damage type, the damaged copy is backed up and repaired or synchronously repaired. 2. The file detection and repair method according to claim 1, wherein the metadata information includes a file name, a file size, and a node identifier; and the step of comparing the metadata information of the file to be detected with the file copies stored on the corresponding service node to determine the damaged copy in each of the file copies comprises: On each service node corresponding to the node identifier, searching for a replica name identical to the file name; Comparing the file size of the to-be-detected file with the copy size of the file copy corresponding to the copy name; In a case where the file size of the to-be-detected file is different from the copy size, it is determined that the file copy corresponding to the copy name is the damaged copy. 3. The file detection and repair method according to claim 2, characterized in that the metadata information also includes hash values of each file block corresponding to the file name; and when the file size of the file to be detected is the same as the size of the copy, it also includes: Using the first replica block of the file replica as the current replica block and the first file block of the file to be detected as the current file block; Calculate the hash value of the current replica block; Determine whether the hash value of the current replica block is equal to the hash value of the current file block; When the hash value of the current replica block is equal to the hash value of the current file block, the next replica block of the current replica block is updated to the current replica block, the next file block of the current file block is updated to the current file block, and the calculation and the hash value of the current replica block are re-executed. 4. The file detection and repair method as described in claim 3 is characterized in that when the hash value of the current replica block and the hash value of the current file block are not equal, the current replica block is determined to be a damaged block and the file copy is a damaged copy. 5. The file detection and repair method according to claim 4, further comprising: Counting starts from the block identifier of the damaged block of the damaged copy until the block identifier of the last copy block of the damaged copy, to obtain a damaged block identifier group. 6. The file detection and repair method according to claim 1, wherein the determining the damage type of each damaged copy comprises: Determining whether the number of copies of the damaged copy of the to-be-detected file is equal to the total number of copies of the corresponding text copy; When the number of the damaged copies is equal to the total number of copies, it is determined that the damage type of the damaged copy is the first type. 7. The file detection and repair method according to claim 6, characterized in that, when the number of copies of the damaged copy is not equal to the total number of copies, it further comprises: Determine whether the number of normal copies of the file other than the damaged copy is greater than or equal to a concurrency threshold; When the number of copies of the normal copies is greater than or equal to the concurrency threshold, the damage type of the damaged copy is determined to be the second type; or, when the number of copies of the normal copies is less than the concurrency threshold, the damage type of the damaged copy is determined to be the third type. 8. The file detection and repair method according to claim 1, wherein the damage type includes a first type, a second type and a third type, and the backing up and repairing the damaged copy according to the damage type comprises: When the damage type is the first type, the damaged copy is repaired by using a backup repair method; When the damage type is the second type or the third type, the damaged copy is repaired by synchronous repair. 9. The file detection and repair method according to claim 8, wherein the step of repairing the damaged copy by using a backup repair method comprises: Access the backup server according to the backup server address in the endpoint configuration; Reading a backup copy of the damaged copy from the backup server according to the copy name of the damaged copy; The damaged copy is overwritten with the backup copy. 10. The file detection and repair method according to claim 8, wherein the synchronous repair comprises concurrent synchronization and separate synchronization, and when the damage type is the second type or the third type, repairing the damaged copy by synchronous repair comprises: When the damage type is the second type, repairing the damaged copy in a concurrent synchronization manner; or, When the damage type is the third type, the damaged copy is repaired by a separate synchronization method. 11. The file detection and repair method according to claim 10, wherein the repairing the damaged copy in a concurrent synchronization manner comprises: Allocate each block identifier of the damaged block identifier group, and match at least one target block identifier for each normal copy corresponding to the damaged copy; According to the target block identifier, the replica blocks in the damaged replica are overwritten with the replica blocks synchronized from each of the normal replicas. 12. The file detection and repair method according to claim 10, wherein the repairing the damaged copy by using a separate synchronization method comprises: Receiving replica blocks synchronized by a normal replica that match each block identifier of the damaged block identifier group; The replica blocks corresponding to the respective block identifiers of the damaged block identifier group in the damaged replica are overwritten with the replica blocks synchronized from the normal replica. 13. The file detection and repair method according to claim 1, further comprising: Periodically trigger the file detection task; or, Receive file detection tasks triggered by GFS clients. 14. The file detection and repair method according to claim 1, further comprising: Monitor each node indicator; Determine whether each of the node indicators exceeds the corresponding indicator threshold; When any node indicator exceeds the corresponding indicator threshold, the response to the file detection task is suspended. 15. The file detection and repair method according to claim 14, wherein the node index comprises: CPU idle percentage, load average, CPU utilization of the server process, and/or the number of bytes read and written per second by the server process. 16. A file detection and repair device, characterized in that the device is applied to a GFS service node of a GFS server, and the device comprises: A receiving module, used for receiving one or more file detection tasks; A reading module, configured to access the metadata server in response to the file detection task according to the metadata server address in the endpoint configuration, and read metadata information of the file to be detected corresponding to the file detection task from the metadata server; A comparison module, used to compare the metadata information of the file to be detected with the file copies stored on the corresponding service node, to determine the damaged copies among the file copies and the damage type of each damaged copy; The repair module is used to perform backup repair or synchronous repair on the damaged copy according to the damage type. 17. A file detection and repair system, characterized in that it comprises: a GFS server composed of multiple GFS service nodes including the file detection and repair device as described in claim 16, a GFS client and a metadata server. 18. The file detection and repair system according to claim 17, further comprising: Back up the server. 19. An electronic device comprising: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to execute the file detection and repair method according to any one of claims 1 to 15. 20. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the file detection and repair method according to any one of claims 1 to 15.