Disclosure of Invention
In order to solve the above problems, one or more embodiments of the present disclosure describe a method and apparatus for performing multi-control storage expansion based on SPDK architecture for memory sharing.
According to a first aspect, there is provided a SPDK architecture multi-control storage expansion method based on memory sharing, the method comprising:
constructing interconnection links between the controllers and other controllers based on a high-speed interconnection protocol supporting memory sharing, wherein the interconnection links are used for performing memory mapping sharing on a plurality of controllers;
Creating an original annular queue in SPDK processes, mapping the mapping annular queue corresponding to the original annular queue in SPDK processes of the rest controllers based on the interconnection links, wherein the original annular queue matched with the mapping annular queue is shared with queue data of the mapping annular queue in real time;
traversing and processing the queue data in the original annular queue.
Preferably, the high-speed interconnection protocol is a non-transparent bridge protocol or a computing fast link protocol.
Preferably, the original ring queue is only provided with message dequeuing authority, and the mapping ring queue is only provided with message enqueuing authority.
Preferably, after the original ring queue is created in the SPDK process, the method further includes:
Initializing the original ring queue.
Preferably, the traversing processes queue data in the original ring queue, including:
Traversing the original annular queue in a continuous polling mode, taking out the queue data from the original annular queue when the queue data exist in the original annular queue, and processing the queue data.
Preferably, the method further comprises:
When a data writing instruction is detected, determining a target mapping annular queue and a target writing line corresponding to data to be written, and writing the data to be written into the target writing line of the target mapping annular queue.
Preferably, the creating an original ring queue in the SPDK process, mapping, in SPDK processes of the rest of the controllers, a mapping ring queue corresponding to the original ring queue based on the interconnection link, includes:
When more than two other controllers needing memory sharing exist, the same number of original annular queues are created in SPDK processes based on the number of the other controllers needing memory sharing, and mapping annular queues corresponding to different original annular queues are mapped in SPDK processes of each other based on the interconnection links.
According to a second aspect, there is provided a SPDK-architecture multi-control storage expansion device based on memory sharing, the device comprising:
The construction module is used for constructing interconnection links between the controller and other controllers based on a high-speed interconnection protocol supporting memory sharing, and the interconnection links are used for carrying out memory mapping sharing on a plurality of controllers;
The mapping module is used for creating an original annular queue in SPDK processes, mapping the mapping annular queue corresponding to the original annular queue in SPDK processes of the rest controllers based on the interconnection link, and sharing the queue data of the original annular queue and the mapping annular queue which are matched with each other in real time;
And the traversing module is used for traversing and processing the queue data in the original annular queue.
According to a third aspect, there is provided an electronic device comprising a processor and a memory;
the processor is connected with the memory;
The memory is used for storing executable program codes;
The processor runs a program corresponding to executable program code stored in the memory by reading the executable program code for performing the steps of the method as provided in the first aspect or any one of the possible implementations of the first aspect.
According to a fourth aspect, there is provided a computer readable storage medium having stored thereon a computer program having instructions stored therein which, when run on a computer or processor, cause the computer or processor to perform a method as provided by any one of the possible implementations of the first aspect or the first aspect.
The method and the device provided by the embodiment oF the specification do not need to support upper communication protocols such as FC, ISCSI, NVMe-oF and the like or need to be converted by block equipment, directly operate the opposite end through SPDK process, namely operate the annular queue mapped by the other controllers, treat the SPDK process oF the opposite end as a self thread, thereby converting communication between processes into communication between threads, reducing resource consumption, improving communication speed, expanding the operation oF SPDK from single controller operation to multi-controller operation, and realizing multi-control storage expansion under SPDK architecture.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.
In the following description, the terms "first," "second," and "first," are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The following description provides various embodiments of the application that may be substituted or combined between different embodiments, and thus the application is also to be considered as embracing all possible combinations of the same and/or different embodiments described. Thus, if one embodiment includes feature A, B, C and another embodiment includes feature B, D, then the application should also be seen as embracing one or more of all other possible combinations of one or more of A, B, C, D, although such an embodiment may not be explicitly recited in the following.
The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made in the function and arrangement of elements described without departing from the scope of the application. Various examples may omit, replace, or add various procedures or components as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.
Referring to fig. 1, fig. 1 is a flow chart of a SPDK architecture multi-control storage expansion method based on memory sharing according to an embodiment of the present application. In an embodiment of the present application, the method includes:
s101, constructing an interconnection link between the controller and the rest of controllers based on a high-speed interconnection protocol supporting memory sharing.
The interconnection link is used for performing memory mapping sharing on a plurality of controllers.
The execution subject of the present application may be a controller.
In the embodiment of the present disclosure, there may be multiple controllers in the multi-control system, and in order to implement multi-control storage expansion under SPDK architecture, the controllers first need to directly construct an interconnection link with each other through a high-speed interconnection protocol supporting memory sharing, so as to implement memory mapping sharing between the multiple controllers through the interconnection link. The high-speed protocol is commonly used in the storage field, and has the advantages oF low cost, high speed and the like compared with upper-layer communication protocols such as FC, ISCSI, NVMe-oF and the like, and the communication connection constructed in a memory mapping sharing mode can enable controllers to be directly connected through an interconnection link, does not need support oF the upper-layer communication protocol, does not need conversion oF block equipment and is higher in efficiency. Meanwhile, in the memory sharing mapping mode, the controller can directly operate the data mapped by other controllers through SPDK processes, so that SPDK processes of the other controllers are treated as threads of the other controllers, and communication between the processes is converted into communication between the threads, so that the operation of SPDK is expanded from a single controller to multiple controllers. The high-speed interconnection protocol supporting memory sharing can be selected from Non-transparent bridges (Non-TRANSPARENT BRIDGE, NTB), computing fast links (Compute Express Link, CXL), unified port interfaces (Unified Port Interface, UPI) and the like. Wherein the rest controllers refer to controllers except the controller in the multi-control system.
In one embodiment, the high speed interconnect protocol is a non-transparent bridge protocol or a computational fast link protocol.
In the embodiment of the present specification, NTB (non-transparent bridge) technology is derived on standard PCIe rules, and two hosts may forward PCIe transactions from one side of the NTB to the other through an address translation (Address Translation) mechanism of the NTB, which may enable different hosts to communicate. The memory of other machines can be mapped to the local end by means of the NTB characteristic, so that the memory sharing is realized. CXL is a high-speed data transmission protocol which allows fast and reliable data transmission between different components inside a computer system, and aims to solve bottleneck problems in high-performance calculation, including problems of memory capacity, memory bandwidth, I/O delay and the like. The CXL technology not only provides high-speed transmission, but also supports memory sharing and virtualization, so that the cooperation between controllers is more compact and efficient. An interconnection link architecture constructed according to a non-transparent bridge protocol or a computational fast link protocol is shown in fig. 2, and a plurality of controllers may be interconnected by pcie ntb links or CXL links. If pcie ntb links are used, other controller memory can be accessed between controls through ntb. If the CXL link is used, a plurality of controllers can access the shared memory hung under the CXL link at the same time.
S102, an original annular queue is created in SPDK processes, and a mapping annular queue corresponding to the original annular queue is mapped in SPDK processes of the rest controllers based on the interconnection link.
And the original annular queues matched with each other are shared with the queue data of the mapping annular queues in real time.
In this embodiment of the present disclosure, the controller creates an original ring queue in the SPDK process, and then shares the original ring queue mapping created by the controller to other controllers through the memory sharing mechanism of the interconnection link, so that a mapping ring queue corresponding to the original ring queue of the controller will also exist in the other controllers. The queue data of the original annular queue and the mapped annular queue which are matched with each other, namely the original annular queue and the mapped annular queue which are intrinsically attributed to one annular queue are shared in real time through an interconnection link, so that other controllers can perform operations such as reading and writing on the mapped annular queue in own SPDK processes, a process of operating the original annular queue of the controller by the other controllers is realized, and a specific principle schematic diagram of the process is shown in fig. 3.
In one embodiment, the original ring queue is provided with only message dequeuing rights and the mapped ring queue is provided with only message enqueuing rights.
In the embodiment of the present disclosure, in order to prevent the present controller and the other controllers mapped with the mapped ring queues from simultaneously reading and writing to the same ring queue, that is, simultaneously reading and writing to conflicts generated by the same memory, only the message dequeuing authority will be set in the original ring queue, and only the message enqueuing authority will be set in the mapped ring queue, so that any controller can only read data from the original ring queue in itself SPDK and write data from the mapped ring queue in itself SPDK.
In one implementation manner, the creating an original ring queue in the SPDK process, mapping a mapping ring queue corresponding to the original ring queue in SPDK processes of the rest of the controllers based on the interconnection link, includes:
When more than two other controllers needing memory sharing exist, the same number of original annular queues are created in SPDK processes based on the number of the other controllers needing memory sharing, and mapping annular queues corresponding to different original annular queues are mapped in SPDK processes of each other based on the interconnection links.
In the embodiment of the present disclosure, in normal cases, an original ring queue only matches with a mapped ring queue, that is, an original ring queue only forms a mapped ring queue in one remaining controller for memory sharing. If a certain annular queue of the controller needs to be shared with a plurality of controllers, the controllers can generate a plurality of original annular queues of the same kind according to the number of the other controllers needing to be shared, and the mapping annular queues are mapped in each other controller through different original annular queues respectively so as to ensure the independence of the same mapping annular queue in the different other controllers and avoid mutual interference.
In one embodiment, after the original ring queue is created in the SPDK process, the method further includes:
Initializing the original ring queue.
In the embodiment of the present specification, after the original ring queue is created, the original ring queue needs to be initialized first. Specific procedures for initialization may include allocating memory space, setting queue size, initializing control variables, marking queue status, setting boundary conditions, initializing auxiliary variables, and the like. Where allocating memory space refers to allocating sufficient memory to store the data structure of the ring queue, it generally includes an array of elements in the queue and control variables (e.g., head pointer, tail pointer, etc.) that manage the state of the queue. Setting the queue size refers to determining the maximum capacity of the queue, i.e., how many elements the queue can store at most, and in general the size of the queue should be a power of 2, thereby simplifying the management logic of the ring queue with bit manipulation. Initializing the control variable refers to setting initial values of a head pointer (head) and a tail pointer (tail) of the queue. The tag queue status refers to a status flag of the initialization queue, such as whether it is empty, full, etc. The boundary conditions are set to construct the wrap-around logic after the head and tail pointers of the queue reach the end of the queue. The initialization auxiliary variable refers to a counter that initializes the number of elements currently in the record queue.
S103, traversing and processing the queue data in the original annular queue.
In the embodiment of the present specification, through the foregoing arrangement, each controller may operate the ring queues of the remaining controllers, so as to implement mutual communication between the controllers. In the communication process, corresponding data are generated in the annular queue, and the controller traverses the original annular queue to determine the queue data therein and uniformly processes the queue data. The process of traversing may choose periodic traversing, which may lead to delay in data processing, or aperiodic continuous real-time traversing, which is generally preferred because of higher latency.
In one embodiment, the traversing processes queue data in the original ring queue, comprising:
Traversing the original annular queue in a continuous polling mode, taking out the queue data from the original annular queue when the queue data exist in the original annular queue, and processing the queue data.
In the present embodiment, the controller will continue to traverse the original ring queue in a constantly polling manner. In the process of traversing the original annular queue, if the queue data is found in the queue, dequeuing operation is firstly carried out on the queue data, namely the queue data is taken out, the queue data can be processed after the queue data is taken out, the data is not processed directly in the queue, and meanwhile, the processed data is prevented from remaining in the queue.
In one embodiment, the method further comprises:
When a data writing instruction is detected, determining a target mapping annular queue and a target writing line corresponding to data to be written, and writing the data to be written into the target writing line of the target mapping annular queue.
In the embodiment of the present disclosure, in the case where there are more than two controllers in the storage system, there may be different mapped ring queues mapped by different controllers in the SPDK process of one controller. When a user writes data through the controller, the controller determines which mapping ring queue the data to be written currently is to be written into according to the data writing instruction, namely, determines the target mapping ring queue, and determines a target writing line, namely, writes a specific position in the target mapping ring queue.
Referring to fig. 4, a multi-control storage expansion device with SPDK architecture based on memory sharing according to an embodiment of the present application will be described in detail. It should be noted that, in the method for executing the embodiment of the present application shown in fig. 1, only the portion relevant to the embodiment of the present application is shown for convenience of description, and specific technical details are not disclosed, please refer to the embodiment of the present application shown in fig. 1.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a SPDK architecture multi-control storage expansion device based on memory sharing according to an embodiment of the present application. As shown in fig. 4, the apparatus includes:
A construction module 401, configured to construct an interconnection link between the controller and the rest of the controllers based on a high-speed interconnection protocol supporting memory sharing, where the interconnection link is used to perform memory mapping sharing on a plurality of controllers;
A mapping module 402, configured to create an original ring queue in a SPDK process, map, based on the interconnect link, a mapping ring queue corresponding to the original ring queue in SPDK processes of the remaining controllers, where the original ring queue matched with the mapping ring queue shares queue data of the mapping ring queue in real time;
a traversing module 403, configured to perform traversing processing on queue data in the original ring queue.
In one embodiment, the high speed interconnect protocol is a non-transparent bridge protocol or a computational fast link protocol.
In one embodiment, the original ring queue is provided with only message dequeuing rights and the mapped ring queue is provided with only message enqueuing rights.
In one embodiment, the mapping module 402 is specifically further configured to:
Initializing the original ring queue.
In one embodiment, the traversal module 403 is specifically configured to:
Traversing the original annular queue in a continuous polling mode, taking out the queue data from the original annular queue when the queue data exist in the original annular queue, and processing the queue data.
In one embodiment, the apparatus further comprises:
and the writing module is used for determining a target mapping annular queue and a target writing line corresponding to the data to be written when the data writing instruction is detected, and writing the data to be written into the target writing line of the target mapping annular queue.
In one embodiment, the mapping module 402 is specifically configured to:
When more than two other controllers needing memory sharing exist, the same number of original annular queues are created in SPDK processes based on the number of the other controllers needing memory sharing, and mapping annular queues corresponding to different original annular queues are mapped in SPDK processes of each other based on the interconnection links.
It will be clear to those skilled in the art that the technical solutions of the embodiments of the present application may be implemented by means of software and/or hardware. "unit" and "module" in this specification refer to software and/or hardware capable of performing a particular function, either alone or in combination with other components, such as Field-Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA), integrated circuits (INTEGRATED CIRCUIT, ICs), and the like.
The processing units and/or modules of the embodiments of the present application may be implemented by an analog circuit that implements the functions described in the embodiments of the present application, or may be implemented by software that executes the functions described in the embodiments of the present application.
Referring to fig. 5, a schematic structural diagram of an electronic device according to an embodiment of the present application is shown, where the electronic device may be used to implement the method in the embodiment shown in fig. 1. As shown in fig. 5, the electronic device 500 may include at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.
Wherein a communication bus 502 is used to enable connected communications between these components.
The user interface 503 may include a Display screen (Display) and a Camera (Camera), and the optional user interface 503 may further include a standard wired interface and a standard wireless interface.
The network interface 504 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), among others.
Wherein the processor 501 may include one or more processing cores. The processor 501 utilizes various interfaces and lines to connect various portions of the overall electronic device 500, perform various functions of the electronic device 500, and process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and invoking data stored in the memory 505. Alternatively, the processor 501 may be implemented in at least one hardware form of digital signal Processing (DIGITAL SIGNAL Processing, DSP), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 501 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image central processing unit (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like, the GPU is used for rendering and drawing contents required to be displayed by the display screen, and the modem is used for processing wireless communication. It will be appreciated that the modem may not be integrated into the processor 501 and may be implemented by a single chip.
The memory 505 may include a random access memory (Random Access Memory, RAM) or a Read-only memory (rom). Optionally, the memory 505 comprises a non-transitory computer readable medium (non-transitory computer-readable storage medium). Memory 505 may be used to store instructions, programs, code sets, or instruction sets. The memory 505 may include a stored program area that may store instructions for implementing an operating system, instructions for at least one function (e.g., a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, etc., and a stored data area that may store data related to the various method embodiments described above, etc. The memory 505 may also optionally be at least one storage device located remotely from the processor 501. As shown in fig. 5, an operating system, a network communication module, a user interface module, and program instructions may be included in the memory 505, which is a type of computer storage medium.
In the electronic device 500 shown in fig. 5, the user interface 503 is mainly used for providing an input interface for a user to obtain data input by the user, and the processor 501 may be used for calling a SPDK architecture multi-control storage expansion application program based on memory sharing stored in the memory 505, and specifically performing the following operations:
constructing interconnection links between the controllers and other controllers based on a high-speed interconnection protocol supporting memory sharing, wherein the interconnection links are used for performing memory mapping sharing on a plurality of controllers;
Creating an original annular queue in SPDK processes, mapping the mapping annular queue corresponding to the original annular queue in SPDK processes of the rest controllers based on the interconnection links, wherein the original annular queue matched with the mapping annular queue is shared with queue data of the mapping annular queue in real time;
traversing and processing the queue data in the original annular queue.
The present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above method. The computer-readable storage medium may include, among other things, any type of disk including floppy disks, optical disks, DVDs, CD-ROMs, micro-drives, and magneto-optical disks, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, such as the division of the units, merely a logical function division, and there may be additional manners of dividing the actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some service interface, device or unit indirect coupling or communication connection, electrical or otherwise.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable memory. Based on this understanding, the technical solution of the present application may be embodied essentially or partly in the form of a software product, or all or part of the technical solution, which is stored in a memory, and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or part of the steps of the method according to the embodiments of the present application. The Memory includes a U disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, etc. which can store the program codes.
Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be implemented by hardware associated with a program of instructions, which may be stored in a computer readable Memory, which may include a flash disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, etc.
The foregoing is merely exemplary embodiments of the present disclosure and is not intended to limit the scope of the present disclosure. That is, equivalent changes and modifications are contemplated by the teachings of this disclosure, which fall within the scope of the present disclosure. Embodiments of the present disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a scope and spirit of the disclosure being indicated by the claims.