CN104616680B - Repeating data deleting system based on optical disc storage as well as data operating method and device - Google Patents

Abstract

The present invention is applicable to the technical field of optical disk storage, and provides a data deduplication system based on optical disk storage, a data operation method, and a device. The system includes a high-speed storage device, a memory, a processor, and an optical disk cabinet. The data operation method includes a data In the backup method and the data restoration method, in the data backup method, the data block is buffered in the buffer zone, and when the buffered amount of the data block reaches the capacity of the optical disc, the data block is stored in batches in the optical disc; The file directory stored in the area, the entire disc is cached in the buffer zone, and the file data blocks are obtained and integrated. The present invention makes full use of the characteristics of optical disc storage, can greatly reduce the number of times of reading and writing of optical discs, thereby realizing a deduplication system based on optical disc storage.

Description

Deduplication system, data operation method and device based on optical disc storage

technical field

The invention belongs to the technical field of optical disc data management, and in particular relates to an optical disc storage-based deduplication system, a data operation method and a device.

Background technique

With the rapid growth of data in the era of big data, the demand for information storage of today's enterprises is also growing rapidly, and the improvement of storage devices can no longer meet the needs of enterprise data storage. In the process of data backup, enterprises will repeatedly store a large amount of the same data in the storage device, resulting in a serious waste of storage resources. Data de-duplication technology is a popular technology in current research. In the process of data storage, the same data is only stored once, which greatly reduces the waste of storage resources.

Some current data deduplication systems, such as P-Dedupe and Extreme Binning, all use disks as data storage devices, and use data blocks as basic storage units for data storage. When the system stores a data block, it detects the data block. If the same data block exists in the storage device, it does not need to store the data block in the storage device; if there is no identical data block, it is The data block is stored in the storage device. The data backed up at one time usually contains a large number of data blocks. The process of storing in the storage device requires frequent access to the storage device. In order to ensure the performance of the deduplication system, there are usually certain requirements for the performance of the storage device. A large number of high-performance storage devices greatly increases the cost of the deduplication system.

Compared with disk storage, optical disk storage is an inexpensive way of long-term data preservation. The one-time writing feature of the optical disc greatly reduces the access frequency of the optical disc when writing data. How to make reasonable use of the characteristics of the optical disc, cooperate with the data cache, organize and manage the data, and realize the data reading and writing on the deduplication system is an urgent need to solve The problem.

Contents of the invention

In view of the above problems, the purpose of the present invention is to provide a data deduplication system based on optical disk storage, a data operation method, and a device, aiming at solving the technical problems of high access frequency and low storage efficiency of existing disk storage systems.

On the one hand, the deduplication system based on optical disc storage includes a high-speed storage device, a memory, a processor, and an optical disc cabinet. The high-speed storage device is used as a buffer of the system and is divided into a data block buffer, a fingerprint storage area, and a file directory. A storage area and an optical disc buffer area; the internal memory is used as a buffer area of the system, and is divided into a fingerprint buffer area and a data block buffer area of a deduplication system based on optical disc storage; multiple optical discs are arranged in the optical disc cabinet;

Wherein the data block buffer is used to store a certain amount of data blocks during data backup, and a block file is composed of several data blocks at the same time. When the number of the block files reaches a certain number, the block files are written in batches into the optical disc in the optical disc cabinet;

The fingerprint storage area is used to store the fingerprint information of all non-repeated data blocks during data backup, and to judge the repeatability of the data blocks by querying the fingerprints in the fingerprint storage area during the process of repeated query of the data. The block fingerprints all point to the physical address of the data block, and the physical address contains the storage location information of the data block in the optical disk cabinet, which is used to fill the directory information of the file corresponding to the data block; similarly, a block is composed of several fingerprints files, when the number of the block files reaches a certain amount, these data blocks are written into the optical disc in the optical disc cabinet in batches;

The file directory storage area is used to store the file directory information of the file during data backup, records the disc information of the file storage and the storage information of each data block, and is used for data block positioning and organization in the data recovery process. At the same time, After the current file data backup is completed, if the size of the file directory does not exceed the capacity of a CD, then write the file directory into a CD in the CD cabinet, if the size of the file directory exceeds the capacity of a CD capacity, it is divided into several parts, which are sequentially stored in several consecutive optical discs in the optical disc cabinet;

The optical disc buffer area is used for buffering the optical disc data in units of the entire optical disc during data restoration, buffering the optical disc data containing the file data during the data recovery process, and at the same time, buffering some optical disc data with high access frequency;

The fingerprint cache area is used to cache some fingerprints with high access frequency during data backup;

The data block buffer area is used for buffering some data blocks with higher reading frequency during data recovery, and cooperates with the optical disc buffer area in the buffer area to form a two-level cache;

The optical discs in the optical disc cabinet are used to store data blocks, fingerprints and file directories, and are stored in units of a single optical disc, and the data in each area is stored in the optical disc in sequence.

Further, the data block fingerprints and file directories are stored and backed up in double discs in the optical disc cabinet.

On the other hand, the data operation method of the deduplication system based on optical disc storage includes a data backup method and a data recovery method;

Wherein said data backup method includes:

After receiving the file data backup request, divide the file data to be backed up into several data blocks, select the data blocks in order, and calculate the fingerprint of the current data block;

For the current data block, query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint cache area and the fingerprint storage area; if there is the same fingerprint, then obtain the physical address of the corresponding data block of the fingerprint, and store the The file directory information of the data block is filled in the storage area; if there is no identical fingerprint, the data block is stored in the data block buffer, and the data block fingerprint is stored in the fingerprint storage area at the same time, in the file Fill the file directory information of the data block in the directory storage area;

Whenever the size of the block file in the data buffer/fingerprint cache storage area reaches the capacity of one optical disc, write the data in the data buffer/fingerprint buffer storage area to the corresponding optical disc in the optical disc cabinet;

After the file data backup is completed, write the file directory in the file directory storage area into the corresponding CD in the CD cabinet;

The data recovery methods mentioned therein include:

When a file data recovery request is received, the file directory information corresponding to the file data is obtained, and the optical disc data storing the file data is pre-read into the optical disc buffer area in units of the entire disc;

First access the data block buffer area, and then access the optical disc buffer area to obtain the data of each data block of the file;

After all the data blocks are acquired, the data blocks are integrated to obtain the original file.

Further, in the data backup method, after the file data backup request is received, the file data to be backed up is divided into several data blocks, the data blocks are selected in order, and the fingerprint of the current data block is calculated, which specifically includes :

After receiving the file data backup request, divide the file data to be backed up into several data blocks of the same size;

Select the data blocks in order, and calculate the corresponding data block fingerprint through the hash algorithm for the current data block.

Further, in the data backup method, for the current data block, query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer area and the fingerprint storage area; if there is the same fingerprint, obtain the Fingerprint corresponds to the physical address of the data block, and fills the file directory information of the data block in the file directory storage area; if there is no identical fingerprint, the data block is stored in the data block buffer, and the data The block fingerprint is stored in the fingerprint storage area, and the step of filling the file directory information of the data block in the file directory storage area specifically includes:

For each data block, first inquire whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer;

If the same fingerprint does not exist in the fingerprint buffer area, then continue to inquire whether there is a fingerprint identical to the current data block fingerprint in the fingerprint storage area;

If the same fingerprint does not exist in the fingerprint storage area, the data block is stored in the data block buffer, and the data block fingerprint is stored in the fingerprint storage area;

If the same fingerprint exists in the fingerprint buffer area, then obtain the physical address of the data block corresponding to the fingerprint;

If the same fingerprint exists in the fingerprint storage area, then obtain the physical address of the data block corresponding to the fingerprint, and cache the fingerprint in the fingerprint buffer area;

The file directory information of the data block is filled in the file directory storage area.

Further, in the data recovery method, the step of first accessing the data block buffer area, then accessing the optical disc buffer area, and obtaining data of each data block of the file specifically includes:

For each data block in the optical disk cache, first inquire whether there is a current data block in the data block cache, and if so, obtain the data block;

If the current data block does not exist in the data block buffer area, the data block is acquired from the optical disk storage buffer area.

In the third aspect, the data operation device of the deduplication system based on optical disc storage includes a data backup device and a data recovery device,

Wherein said data backup device includes:

The block calculation unit is used to divide the file data to be backed up into several data blocks after receiving the file data backup request, select the data blocks in order, and calculate the fingerprint of the current data block;

The query filling unit is used to query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint cache area and the fingerprint storage area for the current data block; if there is the same fingerprint, obtain the physical data of the data block corresponding to the fingerprint. address, and fill in the file directory information of the data block in the file directory storage area; if there is no identical fingerprint, the data block is stored in the data block buffer, and the data block fingerprint is stored in the A fingerprint storage area, filling the file directory information of the data block in the file directory storage area;

The data block fingerprint writing unit is used to write the data in the data buffer/fingerprint buffer storage area into the optical disk cabinet whenever the size of the block file in the data buffer/fingerprint buffer storage area reaches the capacity of a disc Corresponding disc in

The directory writing unit is used to write the file directory in the file directory storage area into the corresponding optical disk in the optical disk cabinet after the file data backup is completed;

Wherein said data recovery device includes:

The data pre-storage unit is used to obtain the file directory information corresponding to the file data when receiving the file data restoration request, and pre-read the optical disc data storing the file data into the optical disc buffer area in units of the entire disc;

A data acquisition unit, configured to first access the data block buffer area, and then access the optical disc buffer area to acquire data of each data block of the file;

The data integration unit is configured to integrate the data blocks to obtain the original file after all the data blocks have been acquired.

Further, the block calculation unit includes:

The block module is used to divide the file data to be backed up into several data blocks of the same size after receiving the file data backup request;

The hash calculation module is used to select data blocks in order, and calculate the corresponding data block fingerprints for the current data block through hash algorithm.

Further, the query filling unit includes:

The first query module is configured to, for each data block, first query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer;

The second query module is used to continue to query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint storage area if the same fingerprint does not exist in the fingerprint buffer;

A data block fingerprint storage module, configured to store the data block in the data block buffer and store the data block fingerprint in the fingerprint storage area if the same fingerprint does not exist in the fingerprint storage area;

An address obtaining module, configured to obtain the physical address of the data block corresponding to the fingerprint if the same fingerprint does not exist in the fingerprint buffer area and the fingerprint storage area;

The directory filling module is used to fill the file directory information of the data block in the file directory storage area.

Further, the data acquisition unit includes:

The third query module is configured to, for each data block in the optical disc cache, first query whether the current data block exists in the data block buffer area, and obtain the data block if it exists;

A fourth query module, configured to obtain the data block from the optical disk storage buffer if the current data block does not exist in the data block buffer.

The beneficial effect of the present invention is: the technical scheme of the present invention comprises file data backup and file data recovery, in the process of file data backup, first carry out data block, after the data block is checked for duplication according to fingerprint, store in buffer zone first, Then batches are stored in optical discs. The present invention utilizes the performance of one-time writing of optical discs to reduce the access frequency of storage resources during data writing and improve the efficiency of data storage; Information, the disc data storing the file data is cached in parallel to the buffer zone, which greatly reduces the number of disk reads, and adds data block cache at the same time, which improves the efficiency of reading data; in addition, in the process of backup and recovery, the present invention provides CD The two-level cache mechanism of cache and data block cache fully adapts to the nature of optical disc data reading and writing, and improves the efficiency of reading data.

Description of drawings

Fig. 1 is the hardware structural diagram of deduplication system;

Fig. 2 is a schematic structural diagram of the data deduplication system;

Fig. 3 is a structural diagram of an optical disc cabinet;

Fig. 4 is a buffer structure diagram;

Fig. 5 is a schematic diagram of a data structure of a data block fingerprint;

Fig. 6 is a schematic diagram of the data structure of the file directory;

Fig. 7 is a flow chart of the data backup method provided by the first embodiment of the present invention;

Fig. 8 is a flow chart of the data backup method provided by the second embodiment of the present invention;

FIG. 9 is a flow chart of a data recovery method provided in a third embodiment of the present invention;

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The embodiment of the present invention provides a deduplication system based on optical disc storage. From the perspective of the system hardware structure, as shown in FIG. 1 , combined with the schematic diagram of the deduplication system shown in FIG. Including high-speed storage device (such as SSD, hard disk), internal memory, processor (ie CPU) and optical disc cabinet, described high-speed storage device is used as the buffer zone of the system, and the buffer zone is mainly divided into data block buffer zone, fingerprint storage area, file directory storage area and optical disc buffer area; a buffer area is set in the memory, and the buffer area is composed of a fingerprint buffer area and a data block buffer area; the optical disc cabinet is provided with multiple optical discs, specifically, the optical disc cabinet is composed of a plurality of optical discs Composition, each optical disc group includes multiple optical discs, the optical disc is the main storage device, and the optical disc cabinet mainly stores data blocks, fingerprints and file directories.

Each part of the above-mentioned system will be described in detail below.

The buffer includes a data block buffer, a fingerprint storage area, a file directory storage area and an optical disk buffer area. Wherein, the data block buffer stores a certain amount of data blocks, a block file is composed of several data blocks, and when the number of block files reaches a certain amount of data, these data blocks are written into the optical disk in batches.

The fingerprint storage area is used to store the fingerprint information of all non-repetitive data blocks. In the process of repeated query of data, the repeatability of the data blocks is judged by querying the fingerprints in the fingerprint storage area. Each data block fingerprint points to the data block Physical address, the physical address contains the storage location information of the data block in the CD cabinet, which is used to fill the directory information of the file corresponding to the data block; similarly, a block file is composed of several fingerprints, when the number of block files reaches a certain amount of data , and write these data blocks to the disc in batches.

The file directory storage area is used to store the file directory information of all files, records the disc information of file storage and the storage information of each data block, and is used for data block positioning and organization in the data recovery process. At the same time, the current file data backup After completion, if the size of a file directory does not exceed the capacity of a CD, the file directory will be written into a CD, if the size of the file directory exceeds the capacity of a CD, it will be divided into several parts and stored in successive CDs middle.

The disc buffer area caches the disc data in units of the entire disc, and caches the disc data containing the file data during the data recovery process. At the same time, it caches some disc data with high access frequency. , ensuring that the data in the optical disc will not change, and ensuring the reliability of the cached data.

The cache includes a fingerprint cache and a data block cache. The fingerprint cache area is used to cache some fingerprints with high access frequency, which improves the efficiency of fingerprint query.

The data block buffer area is used for buffering some data blocks with high reading frequency, cooperates with the optical disc buffer area in the buffer area, fully utilizes the characteristics of the optical disc, forms a two-level buffer, and improves the efficiency of file data recovery.

For ease of description, assume that the capacity of a CD is 40G, 15 CDs of 40G constitute a CD group, and several CD groups constitute a CD cabinet. The CD cabinet structure shown in Figure 3, the CD cabinet is divided into three parts, one part stores data blocks, one part stores file directories, and the other part is used to store fingerprints, and the data in each part is in the order of the whole disk. Therefore, the storage location of the data is predictable, which ensures that the data block storage information in the file directory is known when the data block is stored in the buffer zone and not yet stored in the optical disc.

Based on the above assumptions, refer to the buffer structure shown in Figure 4:

The size of the data block buffer is n*40GB, consisting of n parts of 40GB equal to the capacity of the optical disc, each data block is 16KB in size, and a 16MB chunkfile is formed by 1024 data blocks. When the number of block files stored in the data block buffer reaches 2560, that is, the data temporarily stored in the buffer reaches 40GB (that is, the size of a CD), the 40GB data is stored in a CD in the CD cabinet at one time middle. The data block buffer is n*40GB, which ensures that the data blocks entering the buffer can be stored in the data block buffer at the same time as the data is written to the disc, so that the data read and write in different storage areas can be read and written at the same time, improving efficiency and ensuring data reliability.

In the fingerprint storage area, the fingerprint of a data block is 64B, and a block file is composed of 256K fingerprints. Every time 2560 block files are stored, they are stored in a CD in the CD cabinet at one time. Referring to the data structure diagram of the data block fingerprint shown in Figure 5, a data block fingerprint is 32B, the fingerprint points to the physical address of a data block, the physical address is 32B, and the physical address contains the address information of the data block stored in the optical disk cabinet (disc group ID, disc ID, block file ID, block ID).

In the file directory storage area, once the file backup is completed, if the size of the directory corresponding to the file is less than 40GB of the size of a CD, the file directory is directly stored in a CD in the CD cabinet; otherwise, the file directory is stored according to the 40GB is the basic storage unit, which is sequentially stored in several CDs in the CD cabinet. Referring to the data structure schematic diagram of the file directory as shown in Figure 6, the file directory is a tree-shaped data structure, and the file ID points to each disc address (disc group ID, disc ID) that stores the file data block, and the disc points to store the The block file of the file data block, the ID and block address of the data block are stored in the block file, the data block ID is the storage address of the data block in the block file, and the block address is the storage address of the data block in the original file. In the process of data backup, if the fingerprint of the data block is the same as the fingerprint in the fingerprint storage area, that is, there is the same data block that has been stored in the optical disc, then the physical address information pointed to by the same data block fingerprint is directly sent to the The storage information of the data block can be written in the file directory corresponding to the data block. At the same time, in the process of data recovery, the file directory structure cooperates with the optical disc data prefetch function in the optical disc cache area.

The optical disc buffer area provides an optical disc data prefetch function during the data recovery process, and uses an optical disc as a unit to cache the data in the optical disc storing file data in the optical disc buffer area, and cache some frequently accessed optical discs at the same time data.

Based on the above system, an embodiment of the present invention also provides a data operation method, including a data backup method and a data recovery method. The technical solution of the present invention will be described below through specific embodiments.

Embodiment one:

FIG. 7 shows the flow of the data backup method provided by the first embodiment of the present invention, and only shows the part related to the embodiment of the present invention for convenience of description.

As shown in Figure 7, the data backup method provided in this embodiment includes:

Step S701. After receiving the file data backup request, divide the file data to be backed up into several data blocks, select the data blocks in order, and calculate the fingerprint of the current data block.

When data backup is required, the generation of a file data backup request is first triggered. And receive the file data to be backed up, install a predetermined format, divide the file data to be backed up into several data blocks, each data block is 16KB in size, and then process each data block in turn. Specifically, data blocks are selected one by one in sequence, and the fingerprint of the current data block is calculated. This embodiment does not limit the specific algorithm of the fingerprint, which is the identification information of the data.

Step S702. For the current data block, query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer area and the fingerprint storage area.

The fingerprint buffer area stores some fingerprints with high access frequency, and the fingerprint storage area stores complete fingerprint data.

Step S703, if the same fingerprint exists, obtain the physical address of the data block corresponding to the fingerprint, and fill in the file directory information of the data block in the file directory storage area;

Step S704, if the same fingerprint does not exist, store the data block in the data block buffer, store the data block fingerprint in the fingerprint storage area, and fill the data block in the file directory storage area File directory information.

First query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint cache area, if it exists, directly obtain the physical address of the data block corresponding to the fingerprint, and fill in the file directory information of the data block in the file directory storage area; otherwise continue Query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint storage area, if it exists, directly obtain the physical address of the data block corresponding to the fingerprint, and fill in the file directory information of the data block in the file directory storage area, and at the same time Cache the fingerprint in the fingerprint cache area; explain that this data block is repeated in the CD cabinet, so the data block does not need to be backed up, and directly obtains the physical address of the data block corresponding to the fingerprint, and stores it in the file directory storage area , just write the storage information, that is, write to the file directory corresponding to the current data block. If the same fingerprint as the current data block fingerprint is still not found, it means that the data block data is not saved in the CD cabinet, so the data block needs to be stored, specifically, the data block is stored in the data block buffer, and at the same time The fingerprint of the data block is stored in the fingerprint storage area, and the file directory information of the data block is filled in the file directory storage area.

Step S705 , whenever the size of the block files in the data buffer/fingerprint cache storage area reaches the capacity of one optical disc, write the data in the data buffer/fingerprint buffer storage area to the corresponding optical disc in the optical disc cabinet.

After completing the processing of a data block, continue to select the next data block for the same processing, and when the size of the block file in the data buffer/fingerprint buffer storage area reaches the capacity of an optical disc, that is, when the block file reaches 40G, the The data buffer/fingerprint cache storage area is written to a disc in the disc cabinet.

Step S706. After the file data backup is completed, write the file directory in the file directory storage area into the corresponding optical disk in the optical disk cabinet.

When the backup of the file data is completed, that is, the current data block is the last data block of the file data to be processed and the processing is completed, the file directory in the file directory storage area is finally written into the corresponding optical disk in the optical disk cabinet.

In this embodiment, the optical disk cabinet is composed of several optical disk groups, and each optical disk group has a certain number of optical disks. The optical disk cabinet is divided into several areas for storing data blocks, fingerprints and file directories respectively. In the optical disc cabinet, data is stored in units of a single optical disc, and data in each area is stored in the optical disc in sequence. In order to avoid unrecoverable errors on the optical disc, preferably, in the process of writing the optical disc once, the next adjacent optical disc stores the same data in parallel to ensure the reliability of the data. That is, the data block fingerprints and file directories are stored and backed up in double disks in the CD cabinet.

Embodiment two:

FIG. 8 shows the flow of the data backup method provided by the second embodiment of the present invention, and only shows the part related to the embodiment of the present invention for convenience of description.

As shown in Figure 8, the data backup method provided in this embodiment includes:

Step S801, after receiving the file data backup request, divide the file data to be backed up into several data blocks of the same size;

Step S802, selecting data blocks in sequence, and calculating the corresponding data block fingerprints for the current data block through hash algorithm;

Step S803, for each data block, first inquire whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer;

Step S804, if the same fingerprint does not exist in the fingerprint buffer area, continue to inquire whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint storage area;

Step S805, if the same fingerprint does not exist in the fingerprint storage area, store the data block in the data block buffer, and store the data block fingerprint in the fingerprint storage area;

Step S806, if the same fingerprint exists in the fingerprint cache area or the fingerprint storage area, obtain the physical address of the data block corresponding to the fingerprint, and cache the fingerprint in the fingerprint cache area;

Step S807, filling the file directory information of the data block in the file directory storage area;

Step S808, whenever the size of the block file in the data buffer/fingerprint buffer storage area reaches the capacity of one optical disc, write the data in the data buffer/fingerprint buffer storage area to the corresponding optical disc in the optical disc cabinet;

Step S809, when the file data backup is completed, write the file directory in the file directory storage area into the corresponding optical disk in the optical disk cabinet.

The method of this embodiment discloses some specific steps on the basis of Embodiment 1. When dividing blocks, a hash algorithm is used to query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer area and the fingerprint storage area. First, query the fingerprint buffer area , and then query the fingerprint storage area if it does not exist. If there is no fingerprint consistent with the fingerprint of the current data block in the fingerprint cache area and the fingerprint storage area, the current data block is determined to be a new data block, and the data block needs to be stored; otherwise, the current data block is a duplicate data block, and only Just record the file directory information.

Embodiment three:

FIG. 9 shows the flow of the data recovery method provided by the third embodiment of the present invention, and only shows the part related to the embodiment of the present invention for convenience of description.

As shown in Figure 9, the data recovery method provided in this embodiment includes:

Step S901. When a file data recovery request is received, obtain the file directory information corresponding to the file data, and pre-read the optical disc data storing the file data into the optical disc buffer area in units of the entire disc;

Step S902, first accessing the data block buffer area, and then accessing the optical disc buffer area to obtain data of each data block of the file;

Step S903, when all the data blocks are obtained, integrate the data blocks to obtain the original file.

When data recovery is required, first trigger the generation of a file data recovery request, the file data recovery request includes the directory path of the file data to be recovered, based on which the CD that stores the file directory information in the CD cabinet can be found, and the corresponding file directory can be found After the information is analyzed, the optical disc where the file data to be recovered can be obtained, and then the data in the optical disc is pre-read into the optical disc buffer area as a unit, and then the data block buffer area and the optical disc buffer area are accessed to obtain the data block. After all the data blocks are acquired, the data blocks are integrated to obtain the original file.

When accessing the data block buffer area and the optical disc buffer area, for each data block in the optical disc buffer area, first inquire whether there is a current data block in the data block buffer area, and if so, obtain the data block; If the current data block does not exist in the data block buffer area, the data block is acquired from the optical disk storage buffer area. Since some data blocks with higher reading frequency are cached in the data block buffer area, if the data block to be restored is in the data block buffer area, it can be read directly, otherwise it is in the optical disk buffer area Read, thus forming a two-level cache, which improves the efficiency of file data recovery.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (10)

1. A de-duplication system based on optical disc storage, characterized in that, the de-duplication system includes a high-speed storage device, internal memory, a processor and an optical disc cabinet, and the high-speed storage device is divided into data buffers as a buffer of the system. Block buffer area, fingerprint storage area, file directory storage area and optical disc buffer area; the memory as the system buffer area is divided into fingerprint buffer area and data block buffer area; multiple optical discs are arranged in the optical disc cabinet; Wherein the data block buffer is used to store a certain amount of data blocks during data backup, and a block file is composed of several data blocks at the same time. When the number of the block files reaches a certain number, the block files are written in batches into the optical disc in the optical disc cabinet; The fingerprint storage area is used to store the fingerprint information of all non-repeated data blocks during data backup, and to judge the repeatability of the data blocks by querying the fingerprints in the fingerprint storage area during the process of repeated query of the data. The block fingerprints all point to the physical address of the data block, and the physical address contains the storage location information of the data block in the optical disk cabinet, which is used to fill the directory information of the file corresponding to the data block; similarly, a block is composed of several fingerprints files, when the number of the block files reaches a certain amount, these data blocks are written into the optical disc in the optical disc cabinet in batches; The file directory storage area is used to store the file directory information of the file during data backup, records the disc information of the file storage and the storage information of each data block, and is used for data block positioning and organization in the data recovery process. At the same time, After the current file data backup is completed, if the size of the file directory does not exceed the capacity of a CD, then write the file directory into a CD in the CD cabinet, if the size of the file directory exceeds the capacity of a CD capacity, it is divided into several parts, which are sequentially stored in several consecutive optical discs in the optical disc cabinet; The optical disc buffer area is used for buffering the optical disc data in units of the entire optical disc during data restoration, buffering the optical disc data containing the file data during the data recovery process, and at the same time, buffering some optical disc data with high access frequency; The fingerprint cache area is used to cache some fingerprints with high access frequency during data backup; The data block buffer area is used for buffering some data blocks with higher reading frequency during data recovery, and cooperates with the optical disc buffer area in the buffer area to form a two-level cache; The optical discs in the optical disc cabinet are used to store data blocks, fingerprints and file directories, and are stored in units of a single optical disc, and the data in each area is stored in the optical disc in sequence. 2. The data deduplication system based on optical disk storage according to claim 1, wherein the data block fingerprint and file directory are stored and backed up in double disks in the optical disk cabinet. 3. A data operation method based on the deduplication system as claimed in claim 1 or 2, wherein the data operation method comprises a data backup method and a data recovery method; The data backup methods include: After receiving the file data backup request, divide the file data to be backed up into several data blocks, select the data blocks in order, and calculate the fingerprint of the current data block; For the current data block, query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint cache area and the fingerprint storage area; if there is the same fingerprint, then obtain the physical address of the corresponding data block of the fingerprint, and store the The file directory information of the data block is filled in the storage area; if there is no identical fingerprint, the data block is stored in the data block buffer, and the data block fingerprint is stored in the fingerprint storage area at the same time, in the file Fill the file directory information of the data block in the directory storage area; Whenever the size of the block file in the data buffer/fingerprint cache storage area reaches the capacity of one optical disc, write the data in the data buffer/fingerprint buffer storage area to the corresponding optical disc in the optical disc cabinet; After the file data backup is completed, write the file directory in the file directory storage area into the corresponding CD in the CD cabinet; The data recovery methods include: When a file data recovery request is received, the file directory information corresponding to the file data is obtained, and the optical disc data storing the file data is pre-read into the optical disc buffer area in units of the entire disc; First access the data block buffer area, and then access the optical disc buffer area to obtain the data of each data block of the file; After all the data blocks are acquired, the data blocks are integrated to obtain the original file. 4. The data operation method according to claim 3, wherein in the data backup method, after the file data backup request is received, the file data to be backed up is divided into several data blocks, and the data is selected in order block, and calculate the fingerprint steps of the current data block, including: After receiving the file data backup request, divide the file data to be backed up into several data blocks of the same size; Select the data blocks in order, and calculate the corresponding data block fingerprint through the hash algorithm for the current data block. 5. the data operation method as claimed in claim 4, is characterized in that, in the data backup method, after receiving the file data backup request, the file data to be backed up is divided into several data blocks, and the data blocks are selected in order; For the current data block, the fingerprint of the corresponding data block is calculated by the hash algorithm, and whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer area and the fingerprint storage area; if there is the same fingerprint, the fingerprint is obtained Corresponding to the physical address of the data block, and filling the file directory information of the data block in the file directory storage area, and buffering the fingerprint into the fingerprint buffer area at the same time; if there is no identical fingerprint, then storing the data block In the data block buffer, store the fingerprint of the data block into the fingerprint storage area at the same time, and fill the file directory information step of the data block in the file directory storage area, specifically including: For each data block, first inquire whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer; If the same fingerprint does not exist in the fingerprint buffer area, then continue to inquire whether there is a fingerprint identical to the current data block fingerprint in the fingerprint storage area; If the same fingerprint does not exist in the fingerprint storage area, the data block is stored in the data block buffer, and the data block fingerprint is stored in the fingerprint storage area; If the same fingerprint exists in the fingerprint buffer area, then obtain the physical address of the data block corresponding to the fingerprint; If the same fingerprint exists in the fingerprint storage area, then obtain the physical address of the data block corresponding to the fingerprint, and cache the fingerprint in the fingerprint buffer area; The file directory information of the data block is filled in the file directory storage area. 6. as claimed in claim 3, based on the data operation method in the deduplication system stored in the optical disc, it is characterized in that, in the data recovery method, first access the data block buffer area, then access the optical disc buffer area, and obtain The data steps of each data block in the file include: For each data block in the optical disk cache, first inquire whether there is a current data block in the data block cache, and if so, obtain the data block; If the current data block does not exist in the data block buffer area, the data block is acquired from the optical disk storage buffer area, and the data block is cached in the data block buffer area. 7. A data operation device based on the deduplication system as claimed in claim 1 or 2, wherein the data operation device comprises a data backup device and a data recovery device, Wherein said data backup device includes: The block calculation unit is used to divide the file data to be backed up into several data blocks after receiving the file data backup request, select the data blocks in order, and calculate the fingerprint of the current data block; The query filling unit is used to query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint cache area and the fingerprint storage area for the current data block; if there is the same fingerprint, obtain the physical data of the data block corresponding to the fingerprint. address, and fill in the file directory information of the data block in the file directory storage area; if there is no identical fingerprint, the data block is stored in the data block buffer, and the data block fingerprint is stored in the A fingerprint storage area, filling the file directory information of the data block in the file directory storage area; The data block fingerprint writing unit is used to write the data in the data buffer/fingerprint buffer storage area into the optical disk cabinet whenever the size of the block file in the data buffer/fingerprint buffer storage area reaches the capacity of a disc Corresponding disc in The directory writing unit is used to write the file directory in the file directory storage area into the corresponding optical disk in the optical disk cabinet after the file data backup is completed; Wherein said data recovery device includes: The data pre-storage unit is used to obtain the file directory information corresponding to the file data when receiving the file data restoration request, and pre-read the optical disc data storing the file data into the optical disc buffer area in units of the entire disc; A data acquisition unit, configured to first access the data block buffer area, and then access the optical disc buffer area to acquire data of each data block of the file; The data integration unit is configured to integrate the data blocks to obtain the original file after all the data blocks have been acquired. 8. The data operation device in the deduplication system based on optical disc storage according to claim 7, wherein the block calculation unit comprises: The block module is used to divide the file data to be backed up into several data blocks of the same size after receiving the file data backup request; The hash calculation module is used to select data blocks in order, and calculate the corresponding data block fingerprints for the current data block through hash algorithm. 9. The data operation device in the deduplication system based on optical disc storage according to claim 8, wherein the query filling unit comprises: The first query module is configured to, for each data block, first query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint buffer; The second query module is used to continue to query whether there is a fingerprint identical to the fingerprint of the current data block in the fingerprint storage area if the same fingerprint does not exist in the fingerprint buffer; A data block fingerprint storage module, configured to store the data block in the data block buffer and store the data block fingerprint in the fingerprint storage area if the same fingerprint does not exist in the fingerprint storage area; An address obtaining module, configured to obtain the physical address of the data block corresponding to the fingerprint if the same fingerprint does not exist in the fingerprint buffer area and the fingerprint storage area; The directory filling module is used to fill the file directory information of the data block in the file directory storage area. 10. The data operation device in the deduplication system based on optical disc storage according to claim 7, wherein the data acquisition unit comprises: The third query module is configured to, for each data block in the optical disc cache, first query whether the current data block exists in the data block buffer area, and obtain the data block if it exists; A fourth query module, configured to obtain the data block from the optical disk storage buffer if the current data block does not exist in the data block buffer.