CN106855790A - Document handling method, device and equipment - Google Patents

Abstract

The present disclosure relates to a file processing method, device and equipment. The method includes: obtaining screening conditions of files; screening files in a database according to the screening conditions to obtain target files; performing Raid processing on the target files. The file processing method, device and equipment provided by the present disclosure can improve the storage space utilization rate of a distributed multi-copy file system and reduce the time delay when accessing damaged files.

Description

File processing method, device and equipment

technical field

The present disclosure relates to the technical field of data processing, and in particular to a file processing method, device and equipment.

Background technique

In a distributed multi-copy file system, in order to ensure the availability of data, multiple copies of data will be kept on multiple nodes. Doing so will cause the occupancy rate of system storage space to increase geometrically.

In the prior art, in order to reduce the consumption of storage space, the files are generally subjected to Raid processing. The so-called Raid is to encode different data blocks of the same file to form some redundant blocks. When the data blocks of the file are lost, it can be The original data blocks are re-decoded through these redundant blocks, and the same data reliability as multiple copies can be achieved through Raid processing, but only a small amount of storage space is required. Taking the encoding of 6+3 as an example, raid can encode 6 data blocks of the file to form 3 redundant data blocks, and any 3 blocks in the subsequent 9 data blocks are damaged, and can be re-decoded through these data blocks However, to achieve the goal of being able to tolerate the damage of any 3 data blocks simply through multiple copies of data, it is necessary to save 4 copies for each data block, and a total of 24 data blocks need to be consumed. However, through the raid method, only Consumes 9 data blocks.

However, in the existing Raid scheme, Raid processing is performed on all files in the system or files in certain fixed directories. If the encoded file is really damaged, it will take a long time to wait for decoding when reading. The access delay is large.

Contents of the invention

The present disclosure provides a file processing method, device and equipment, which are used to improve the storage space utilization rate of a distributed multi-copy file system and reduce the time delay when accessing damaged files.

According to a first aspect of an embodiment of the present disclosure, a file processing method is provided, including:

Filter conditions for obtaining files;

Filtering the files in the database according to the filtering conditions to obtain the target file;

Raid processing is performed on the target file.

The technical solution may include the following beneficial effects: by filtering out files meeting specific screening conditions from the database as target files, and performing Raid processing on the target files, the number of files for Raid processing can be reduced, thereby shortening the consumption of Raid processing. , reducing the occupancy of system resources. Moreover, since the files that undergo Raid processing are only files that meet specific filtering conditions, rather than all files in the system, after Raid processing, the number of redundant data blocks generated is reduced, saving the storage space of the system. Moreover, when the user accesses the damaged file, the waiting time will also be reduced due to the reduction of redundant data blocks.

In a first possible implementation manner of the first aspect, the filtering conditions for obtaining files include:

Get pre-stored filter criteria from the database, or;

Obtain the filtering conditions entered by the user from the user interface.

The technical solution may include the following beneficial effects: by obtaining the pre-set filtering conditions from the database or directly obtaining the filtering conditions input by the user in real time from the user interaction interface, the personalization of file screening can be realized. At the same time, the problem of waste of storage space caused by performing Raid processing on all files can be avoided, and the utilization rate of storage space is improved.

According to the first aspect or the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the filtering conditions include at least one of the following conditions: file type, file attribute, file name, file content;

Said filtering the files in the database according to the filtering conditions to obtain the target files includes:

The files meeting the at least one condition are selected from the database as target files.

According to the second possible implementation manner of the first aspect, in the third possible implementation manner of the first aspect, the filtering out the files meeting the at least one condition from the database as the target file includes:

A standard depth-first traversal method or a breadth-first traversal method is used to periodically filter out files that meet the at least one condition from the database as target files.

The technical solution may include the following beneficial effects: by setting multiple filtering conditions, the accurate positioning of the target file can be realized, and the scope and quantity of the target file can be further reduced, so as to save storage space and improve storage space utilization . In addition, due to the reduction of files processed by the Raid, the waiting time for users to access damaged files can be effectively reduced, thereby improving user experience. And through periodic screening, files meeting the screening conditions can be avoided from being missed, which improves the reliability of the system.

According to a second aspect of an embodiment of the present disclosure, a file processing device is provided, including:

The acquisition module is configured to acquire the filter conditions of the file;

A screening module configured to screen files in the database according to the screening conditions to obtain target files;

The processing module is configured to perform Raid processing on the target file.

The technical solution may include the following beneficial effects: by filtering out files meeting specific screening conditions from the database as target files, and performing Raid processing on the target files, the number of files for Raid processing can be reduced, thereby shortening the consumption of Raid processing. , reducing the occupancy of system resources. Moreover, since the files that undergo Raid processing are only files that meet specific filtering conditions, rather than all files in the system, after Raid processing, the number of redundant data blocks generated is reduced, saving the storage space of the system. Moreover, when the user accesses the damaged file, the waiting time will also be reduced due to the reduction of redundant data blocks.

In a first possible implementation manner of the second aspect, the obtaining module includes:

The first obtaining submodule is configured to obtain pre-stored filter conditions from the database. or,

The second acquiring submodule is configured to acquire the filtering conditions input by the user from the user interaction interface.

The technical solution may include the following beneficial effects: by obtaining the pre-set filtering conditions from the database or directly obtaining the filtering conditions input by the user in real time from the user interaction interface, the personalization of file screening can be realized. At the same time, the problem of waste of storage space caused by performing Raid processing on all files can be avoided, and the utilization rate of storage space is improved.

According to the first aspect or the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the filtering conditions include at least one of the following conditions: file type, file attribute, file name, file content;

The screening modules include:

The first screening submodule is configured to filter out files meeting the at least one condition from the database as target files.

According to the second possible implementation of the first aspect, in the third possible implementation of the first aspect, the first screening submodule is further configured to adopt a standard depth-first traversal method or a breadth-first traversal method , periodically filter out files that meet the at least one condition from the database as target files.

The technical solution may include the following beneficial effects: by setting multiple filtering conditions, the accurate positioning of the target file can be realized, and the scope and quantity of the target file can be further reduced, so as to save storage space and improve storage space utilization . In addition, due to the reduction of files processed by the Raid, the waiting time for users to access damaged files can be effectively reduced, thereby improving user experience. And through periodic screening, files meeting the screening conditions can be avoided from being missed, which improves the reliability of the system.

According to a third aspect of an embodiment of the present disclosure, there is provided a file processing device, including:

processor;

memory configured to store processor-executable instructions;

Wherein, the processor is configured as:

Filter conditions for obtaining files;

Filtering the files in the database according to the filtering conditions to obtain the target file;

Raid processing is performed on the target file.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

Fig. 1 is a flow chart of Embodiment 1 of a file processing method shown according to an exemplary embodiment;

Fig. 2 is a flow chart of Embodiment 2 of a file processing method according to an exemplary embodiment;

Fig. 3 is a block diagram of Embodiment 1 of a file processing device according to an exemplary embodiment;

Fig. 4 is a block diagram of Embodiment 2 of a file processing device according to an exemplary embodiment;

Fig. 5 is a block diagram of a file processing device according to an exemplary embodiment.

By means of the above-mentioned drawings, certain embodiments of the present disclosure have been shown and will be described in more detail hereinafter. These drawings and written description are not intended to limit the scope of the disclosed concept in any way, but to illustrate the disclosed concept for those skilled in the art by referring to specific embodiments.

detailed description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

Fig. 1 is a flow chart of Embodiment 1 of a file processing method according to an exemplary embodiment, the method can be executed by a file processing device, and the file processing device can be integrated in a file processing device such as a server, as shown in Fig. 1 As shown, the method includes the following steps:

In step 101, filter conditions of files are obtained.

The files involved in the embodiments of the present disclosure may be specifically referred to as files in a distributed multi-copy file system (hereinafter referred to as the system).

Filtering conditions are set by users according to actual needs or understanding of files, and are used to filter out certain files that meet certain characteristics. In practical applications, the filter conditions may be preset and stored in the database. When filtering system files, it is called directly from the database. The filtering conditions may also be input by the user in real time through the human-computer interaction interface on the file processing device. When the user triggers the input, the file processing device acquires the filtering conditions input by the user from the man-machine interaction interface.

In this embodiment, the screening conditions include at least one of the following conditions: file type, such as compressed files, archived backup files, etc.; file attributes, such as the last modification time of the file, the size of the file, etc.; file name; file content , such as a file with a specific content at the head of the file, or a file with a specific content at the end of the file, etc. Of course, what is here is only an example rather than a sole limitation to the present disclosure.

In step 101, by obtaining the pre-set filtering conditions from the database or directly obtaining the filtering conditions input by the user in real time from the user interaction interface, the personalization of file screening can be realized. At the same time, the problem of waste of storage space caused by performing Raid processing on all files can be avoided, and the utilization rate of storage space is improved.

In step 102, the files in the database are screened according to the screening conditions to obtain the target file.

In this embodiment, the target file refers to a file to be subjected to Raid processing.

For example, assume that the filtering conditions set by the user are file type, file attribute, and file content, wherein the file type includes compressed files, and the file attribute includes a specified range of file size (25kb, 100kb). After obtaining the screening conditions, the file processing device selects from the database of the system the files that meet the above-mentioned file attribute conditions and file type conditions at the same time as the target file, for example, a compressed file whose size is 80kb, then this file is used as the target file , and so on, until all qualified files are filtered out from the system database.

In step 103, Raid processing is performed on the target file.

The purpose of Raid processing is to encode the data blocks of the target file to generate a preset number of redundant data blocks. When one or some data blocks in the target file are damaged, good data blocks can be obtained by re-encoding according to these data blocks and redundant data blocks. In the embodiment of the present disclosure, the method for performing Raid processing on the target file is similar to the prior art, and will not be repeated here.

In this embodiment, by screening out the files that meet the specific screening conditions from the database as the target files, and performing Raid processing on the target files, the number of files for Raid processing can be reduced, thereby shortening the time-consuming of Raid processing and reducing the occupancy of system resources. Moreover, since the files that undergo Raid processing are only files that meet specific filtering conditions, rather than all files in the system, after Raid processing, the number of redundant data blocks generated is reduced, saving the storage space of the system. Moreover, when the user accesses the damaged file, the waiting time will also be reduced due to the reduction of redundant data blocks.

Fig. 2 is a flow chart of Embodiment 2 of a file processing method according to an exemplary embodiment. As shown in Fig. 2, the method may include the following steps:

In step 201, filter conditions of files are acquired, wherein the filter conditions include at least one of the following conditions: file type, file attribute, file name, and file content.

The files involved in the embodiments of the present disclosure may be specifically referred to as files in a distributed multi-copy file system (hereinafter referred to as the system).

Filtering conditions are set by users according to actual needs or understanding of files, and are used to filter out certain files that meet certain characteristics. In practical applications, the filter conditions may be preset and stored in the database. When filtering system files, it is called directly from the database. The filtering conditions may also be input by the user in real time through the human-computer interaction interface on the file processing device. When the user triggers the input, the file processing device acquires the filtering conditions input by the user from the man-machine interaction interface.

In this embodiment, the screening conditions include at least one of the following conditions: file type, such as compressed files, archived backup files, etc.; file attributes, such as the last modification time of the file, the size of the file, etc.; file name; file content , such as a file with a specific content at the head of the file, or a file with a specific content at the end of the file, etc. Of course, what is here is only an example rather than a sole limitation to the present disclosure.

In step 201, by obtaining the preset filtering conditions from the database or directly obtaining the filtering conditions input by the user in real time from the user interaction interface, the personalization of file screening can be realized. At the same time, the problem of waste of storage space caused by performing Raid processing on all files can be avoided, and the utilization rate of storage space is improved.

In step 202, files meeting the at least one condition are screened out from the database as target files.

In this embodiment, the target file refers to a file to be subjected to Raid processing.

In practical applications, the standard depth-first traversal method or the breadth-first traversal method can be used to traverse the files in the database, and the files meeting all the filter conditions can be used as target files. Wherein, the above-mentioned traversal may be performed periodically or according to an artificially set strategy, which is not limited in this embodiment. For example, assume that the filtering conditions set by the user are file type, file attribute, and file content, wherein the file type includes compressed files, and the file attribute includes a specified range of file size (25kb, 100kb). After obtaining the screening conditions, the file processing device selects from the database of the system the files that meet the above-mentioned file attribute conditions and file type conditions at the same time as the target file, for example, a compressed file whose size is 80kb, then this file is used as the target file , and so on, until all qualified files are filtered out from the system database.

In step 203, Raid processing is performed on the target file.

The purpose of Raid processing is to encode the data blocks of the target file to generate a preset number of redundant data blocks. When one or some data blocks in the target file are damaged, good data blocks can be obtained by re-encoding according to these data blocks and redundant data blocks. In the embodiment of the present disclosure, the method for performing Raid processing on the target file is similar to the prior art, and will not be repeated here.

In this embodiment, by setting multiple screening conditions, accurate positioning of the target file can be realized, and the scope and quantity of the target file can be further reduced, so as to save storage space and improve storage space utilization. In addition, due to the reduction of files processed by the Raid, the waiting time for users to access damaged files can be effectively reduced, thereby improving user experience. And through periodic screening, files meeting the screening conditions can be avoided from being missed, which improves the reliability of the system.

Fig. 3 is a block diagram of Embodiment 1 of a file processing device according to an exemplary embodiment. As shown in Fig. 3 , the device includes:

The obtaining module 11 is configured to obtain the filter conditions of the file;

The screening module 12 is configured to screen the files in the database according to the screening conditions to obtain the target file;

The processing module 13 is configured to perform Raid processing on the target file.

Wherein, the acquisition module 11 includes:

The first obtaining submodule 111 is configured to obtain pre-stored filter conditions from a database.

The second obtaining sub-module 112 is configured to obtain the filter condition input by the user from the user interaction interface.

The file processing device provided in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG. 1 , and its execution mode and beneficial effects are similar, and details are not repeated here.

Fig. 4 is a block diagram of Embodiment 2 of a file processing device according to an exemplary embodiment. As shown in Fig. 4, on the basis of the embodiment shown in Fig. 3, the filtering conditions include the following conditions At least one of: file type, file attribute, file name, file content.

Described screening module 12 comprises:

The first screening submodule 121 is configured to filter out files meeting the at least one condition from the database as target files.

Wherein, the first screening sub-module 121 is further configured to use a standard depth-first traversal method or a breadth-first traversal method to periodically filter out files that meet the at least one condition from the database as target files .

The file processing device provided in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG. 2 , and its execution mode and beneficial effect are similar, and details are not repeated here.

With regard to the file processing apparatus in the above embodiments, the specific manners of performing operations by various modules and sub-modules have been described in detail in the embodiments related to the method, and will not be described in detail here.

The internal functions and structure of the file processing device have been described above, as shown in Figure 5, in practice, the file processing device can be implemented as a file processing device such as a server, including:

processor;

memory configured to store processor-executable instructions;

Wherein, the processor is configured as:

Filter conditions for obtaining files;

Filtering the files in the database according to the filtering conditions to obtain the target file;

Raid processing is performed on the target file.

In this embodiment, by screening out the files that meet the specific screening conditions from the database as the target files, and performing Raid processing on the target files, the number of files for Raid processing can be reduced, thereby shortening the time-consuming of Raid processing and reducing the occupancy of system resources. Moreover, since the files that undergo Raid processing are only files that meet specific filtering conditions, rather than all files in the system, after Raid processing, the number of redundant data blocks generated is reduced, saving the storage space of the system. Moreover, when the user accesses the damaged file, the waiting time will also be reduced due to the reduction of redundant data blocks.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory storing instructions, which can be executed by a processor of the file processing device to implement the above method. For example, the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of the file processing device, the file processing device can execute a file processing method, the method comprising:

Filter conditions for obtaining files;

Filtering the files in the database according to the filtering conditions to obtain the target file;

Raid processing is performed on the target file.

Other embodiments of the present disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (11)

1. A file processing method, characterized in that the method comprises: Filter conditions for obtaining files; Filtering the files in the database according to the filtering conditions to obtain the target file; Raid processing is performed on the target file. 2. The method according to claim 1, wherein the filter conditions for obtaining files include: Get pre-stored filter conditions from the database. 3. The method according to claim 1, wherein the filter conditions for obtaining files include: Obtain the filtering conditions entered by the user from the user interface. 4. The method according to any one of claims 1 to 3, wherein the screening conditions include at least one of the following conditions: file type, file attribute, file name, file content; Said filtering the files in the database according to the filtering conditions to obtain the target files includes: The files meeting the at least one condition are selected from the database as target files. 5. The method according to claim 4, wherein the filtering out the files meeting the at least one condition from the database as target files includes: A standard depth-first traversal method or a breadth-first traversal method is used to periodically filter out files meeting the at least one condition from the database as target files. 6. A file processing device, characterized in that the device comprises: The acquisition module is configured to acquire the filter conditions of the file; A screening module configured to screen files in the database according to the screening conditions to obtain target files; The processing module is configured to perform Raid processing on the target file. 7. The device according to claim 6, wherein the acquiring module comprises: The first obtaining submodule is configured to obtain pre-stored filter conditions from the database. 8. The device according to claim 6, wherein the acquiring module comprises: The second acquiring submodule is configured to acquire the filtering conditions input by the user from the user interaction interface. 9. The device according to any one of claims 6 to 8, wherein the filtering conditions include at least one of the following conditions: file type, file attribute, file name, file content; The screening modules include: The first screening submodule is configured to filter out files meeting the at least one condition from the database as target files. 10. The device according to claim 9, wherein the first screening submodule is further configured to periodically filter out simultaneous A file meeting the at least one condition is used as a target file. 11. A file processing device, characterized in that it comprises: processor; memory configured to store processor-executable instructions; Wherein, the processor is configured as: Filter conditions for obtaining files; Filtering the files in the database according to the filtering conditions to obtain the target file; Raid processing is performed on the target file.