WO2025054911A1 - Photoacoustic-imaging-based method and apparatus for real-time naked-eye visualization of blood vessels on body surface - Google Patents

Abstract

Disclosed in the present invention are a photoacoustic-imaging-based method and apparatus for real-time naked-eye visualization of blood vessels on a body surface. The method comprises: generating a pulsed laser, and focusing same on a tissue surface to form a linear light spot; collecting a generated photoacoustic signal, and performing computer processing and reconstruction on same to obtain a photoacoustic blood vessel image; projecting the reconstructed photoacoustic blood vessel image to the tissue surface by means of a projector combined with an approximate ellipse fitting curved surface projection algorithm; and collecting a projection image of the tissue surface in real time by means of an RGBD camera, transmitting the projection image to a computer, using a learning-based target tracking algorithm to calculate the pose of the tissue surface in real time, and when the tissue surface undergoes an involuntary movement, performing blood vessel image tracking and projection. In the present invention, real-time naked-eye visualization of blood vessels on a body surface is implemented by means of photoacoustic imaging, such that the imaging depth and resolution are high, and the scanning range is large and flexible; and by using a target tracking algorithm, a blood vessel image can still be projected in-situ onto a tissue surface in real time when tissue moves, such that real-time visualization is more accurate.

Description

Real-time naked-eye visualization method and device for blood vessel surface based on photoacoustic imaging Technical Field

The present invention belongs to the technical field of photoacoustic imaging technology and computer vision, and specifically relates to a method and device for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging.

Background Art

Accurate positioning of blood vessels is crucial in vascular-related surgeries. Currently, there are many ways to image blood vessels, such as Doppler ultrasound, computed tomography angiography, magnetic resonance angiography, and transmission near-infrared imaging. Among these vascular imaging methods, Doppler ultrasound has low sensitivity to microvessels and cannot image high-resolution microvessel images; computer tomography angiography requires intravenous injection of contrast agents, and patients may be allergic to contrast agents, causing complications, and this imaging method has ionizing radiation, which can cause harm to the human body; nuclear magnetic resonance angiography also requires intravenous injection of contrast agents, which can cause complications; and although transmission near-infrared imaging can image blood vessels non-destructively, the imaging depth of transmission near-infrared imaging is shallow and is mostly used for imaging of superficial veins of the skin; and multi-element photoacoustic computed tomography can not only image blood vessels non-destructively, but also blood vessels have the characteristic of specific absorption of light, which makes photoacoustic imaging have an inherent advantage in vascular imaging, and photoacoustic imaging can be used to perform high-resolution vascular imaging of blood vessels. At the same time, photoacoustic computed tomography uses linear light spots for photoacoustic signal excitation, has a deep penetration depth, and adopts a multi-element signal receiving method, with a large imaging range. Therefore, photoacoustic computed tomography has the advantages of large depth and high resolution imaging.

In traditional image navigation, images are presented on a 2D screen, separated from the real tissue surface. The combination of vascular images and the real tissue surface depends entirely on the experience of medical staff. With the development of technology, many devices have emerged that can directly combine vascular images with the tissue surface, making blood vessels visible on the body surface. However, the current vascular surface visualization devices have the problems of shallow imaging depth and low resolution. At the same time, even though many devices use cameras for positioning, they do not process the curved surface projection, making the projection of blood vessels on the curved surface inaccurate.

Patent CN 104665766 A discloses a portable venous vessel imager. The patent uses an infrared imaging device to image the venous vessels, and uses a projector to project the vessels onto the skin surface for visualization. Although this venous vessel imager is easy to use, the vascular imaging method adopted in the patent is infrared imaging. This imaging method has a shallow imaging depth and low imaging resolution due to its projection irradiation and diffuse reflection reception. Therefore, the patent can only be used for skin surface vein imaging. In addition, the patent does not have a target positioning device, and the accuracy of vascular projection may be low.

Patent CN 112773333 A discloses a portable vascular imaging device. The patent uses near-infrared imaging to image the surface veins of the skin, and uses a projector to project the vascular image onto the skin surface, while using a camera module for positioning. The near-infrared imaging used in the patent also has the characteristics of shallow imaging depth and poor imaging resolution. In addition, even if a camera is used for positioning, the patent does not use target tracking and curved surface projection strategies to improve the accuracy of vascular projection.

Patent CN 104116496 A discloses a medical three-dimensional venous blood vessel augmented reality device and method. The patent designs a pair of glasses to fuse the venous blood vessel image with the tissue surface. By wearing the glasses, the venous blood vessel image can be visualized on the tissue surface. One disadvantage of this method is that the augmented reality image cannot be visualized with the naked eye, and the effect of the blood vessels displayed on the body surface is only presented to the wearer of the glasses, and cannot be shared by other members of the surgical team. In addition, the patent only focuses on the visualization of venous blood vessels and does not mention the deep microvascular network.

Patent CN 210842997 U discloses a clinical blood collection pad, and patent CN 214632123 U discloses a near-infrared blood vessel projector. These two patents also use near-infrared imaging, which has the disadvantages of shallow imaging depth and low resolution.

Summary of the invention

The main purpose of the present invention is to provide a method and device for real-time naked-eye visualization of vascular surface based on photoacoustic imaging to address the problems of shallow imaging depth, poor imaging resolution, and inaccurate naked-eye visualization of blood vessels in current vascular surface visualization technology.

In order to achieve the above object, the present invention adopts the following technical solutions:

One aspect of the present invention provides a method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging, comprising the following steps:

Generate pulsed laser and focus it on the tissue surface through the photoacoustic imaging probe to form a light spot;

The photoacoustic signal is collected by a photoacoustic imaging probe and processed and reconstructed by a computer to obtain a photoacoustic vascular image;

The reconstructed photoacoustic vascular image is projected onto the tissue surface by a projector combined with an approximate ellipse fitting surface projection algorithm;

The projection image of the tissue surface is collected in real time by an RGBD camera and transmitted to a computer. The posture of the tissue surface is solved in real time using a learning-based target tracking algorithm. When the tissue surface moves involuntarily, vascular image tracking and projection are performed.

As a preferred technical solution, the pulsed laser is generated and focused on the tissue surface to form a light spot through a photoacoustic imaging probe, specifically:

The pulse controller is controlled by a computer to send out a pulse signal, which in turn drives the pulse laser to generate a pulse laser. The generated pulse laser is coupled into the optical fiber bundle through the optical fiber coupler. The optical fiber bundle is connected to the photoacoustic imaging probe. After the pulse laser passes through the photoacoustic imaging probe, a focal line spot is formed on the tissue surface.

As a preferred technical solution, the approximate ellipse fitting surface projection algorithm is specifically as follows:

Using the RGBD camera to perform three-dimensional surface reconstruction on the tissue surface;

Mathematical modeling of three-dimensional surface models;

The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model, and the distances x, y, and z between two points in the three-dimensional space are solved based on these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted by an approximate ellipse fitting method, and this fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in a corresponding proportion with the ratio of the length of the ellipse curved edge to the major axis of the ellipse, thereby realizing the projection of the two-dimensional vascular image on the curved surface tissue.

As a preferred technical solution, the learning-based target tracking algorithm specifically includes:

Input layer for inputting images;

The object recognition network uses the MobileNet network for feature extraction, specifically: adding three additional convolutional layers of 1x1, 3x3, and 5x5 before the original 1x1 convolutional layer of the MobileNet network, and adding a pooling layer after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map;

A feature extraction network for extracting feature points through a CNN network, specifically: adding a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers in parallel on the basis of the original 3x3 convolution layer of the CNN network, that is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and the pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and the pooling layer;

The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps.

As a preferred technical solution, the device calibration method of the projector and the RGBD camera is specifically as follows:

The customized standard photoacoustic imaging sample and the image reconstructed after the sample photoacoustic imaging are used to extract the characteristic points of the sample image collected by the RGBD camera and the photoacoustic image reconstructed after imaging, and the calibration is completed by solving the transformation relationship between the corresponding characteristic points on the sample image and the photoacoustic image; the customized standard The quasi-photoacoustic imaging sample was made by placing carbon rods in agar in a checkerboard pattern.

As a preferred technical solution, no marker is used for registration after photoacoustic imaging. The tissue surface is captured by the RGBD camera in an arbitrary posture and transmitted to a computer. Feature points of the tissue surface in an arbitrary posture are extracted in the computer, the registration relationship between the vascular image and the tissue surface is estimated, and the projected vascular image is affine transformed, and finally projected by a projector.

Another aspect of the present invention also provides a real-time naked-eye visualization device for blood vessel surface based on photoacoustic imaging, comprising a pulsed laser generator, a photoacoustic imaging probe, an acquisition and amplification circuit, a computer, a projector, and an RGBD camera;

The pulse laser generating device is connected to the photoacoustic imaging probe and is used to generate pulse laser and focus the pulse laser on the tissue surface through the photoacoustic imaging probe to form a light spot;

The photoacoustic imaging probe is also used to collect the generated photoacoustic signals and transmit them to the computer via the collection and amplification circuit;

The computer is used for processing and reconstructing photoacoustic blood vessel images, and is loaded with an approximate ellipse fitting surface projection algorithm and a learning-based target tracking algorithm;

The projector is used to project the reconstructed photoacoustic blood vessel image onto the tissue surface in combination with an approximate ellipse fitting surface projection algorithm;

The RGBD camera is used to align the projected photoacoustic vascular image with the tissue surface in combination with a learning-based target tracking algorithm, and to reproject when the tissue surface moves involuntarily.

As a preferred technical solution, the pulse laser generating device includes a pulse controller, a pulse laser, a fiber coupler, and a fiber bundle which are connected in sequence; the pulse controller is controlled by a computer to send out a pulse signal, thereby driving the pulse laser to generate a pulse laser, and the generated pulse laser is coupled into the fiber bundle through the fiber coupler, and the fiber bundle is connected to a photoacoustic imaging probe. After passing through the photoacoustic imaging probe, the pulse laser is focused on the tissue surface to form a line spot.

As a preferred technical solution, the photoacoustic imaging probe includes a cylindrical lens, a reflector and a multi-element ultrasonic transducer array. The optical fiber bundle is focused into a line spot by the cylindrical lens, and then vertically enters the tissue surface after being reflected 45 degrees by two reflectors. The multi-element ultrasonic transducer array is located next to the cylindrical lens and directly above the focused line spot, and is used to receive the generated photoacoustic signal; the cylindrical lens is placed in a precision slide bracket, and the spot size of the focused line spot is adjusted by adjusting the distance between the precision slide and the optical fiber bundle, thereby adjusting the imaging depth and imaging resolution; the multi-element ultrasonic transducer array is composed of 128 ultrasonic transducers with a main frequency of 10MHz.

As a preferred technical solution, the approximate ellipse fitting surface projection algorithm is specifically as follows:

Using the RGBD camera to perform three-dimensional surface reconstruction on the tissue surface;

Mathematical modeling of three-dimensional surface models;

The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model, and the distances x, y, and z between two points in the three-dimensional space are solved with these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted by an approximate ellipse fitting method, and this fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in a corresponding proportion with the ratio of the length of the ellipse curved edge to the major axis of the ellipse, so as to realize the projection of the two-dimensional vascular image on the curved surface tissue.

The learning-based target tracking algorithm includes:

Input layer for inputting images;

The object recognition network uses the MobileNet network for feature extraction, specifically: adding three additional convolutional layers of 1x1, 3x3, and 5x5 before the original 1x1 convolutional layer of the MobileNet network, and adding a pooling layer after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map;

The feature extraction network for feature point extraction through the CNN network is as follows: on the basis of the original 3x3 convolution layer of the CNN network, a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers are added in parallel, that is, the original One path of the output of the 3x3 convolutional layer is connected to the fully connected layer through the 3x3 convolutional layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolutional layer and pooling layer; the output of the newly added 3x3 convolutional layer is also connected to the fully connected layer through the newly added 5x5 convolutional layer and pooling layer;

The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

(1) The vascular imaging of the present invention is deeper and has a higher imaging resolution: Photoacoustic computed tomography uses a large spot to irradiate, has a deep penetration depth, and utilizes the specific absorption of light by blood vessels, and has a high sensitivity for blood vessel and microvessel detection;

(2) The scanning range of the present invention is larger and more flexible: the present invention can use a mechanical scanning structure to drive the probe for scanning, and can also scan and image the tissue by hand, with flexible imaging methods and unlimited imaging range;

(3) The real-time visualization of blood vessels on the body surface of the present invention is more accurate: the tissue surface is identified by a target tracking algorithm based on deep learning, and features are extracted from the tissue surface by a customized feature extraction network, so that the posture solution is more accurate when the tissue surface moves involuntarily, and the blood vessel image can still be projected in real time on the tissue surface when the tissue moves. In addition, combined with the approximate elliptical surface algorithm, the blood vessel image can be projected on the curved tissue surface more accurately.

(4) The blood vessel positioning of the present invention has three-dimensional depth information: the three-dimensional model of the surgical surface and the three-dimensional photoacoustic blood vessel are integrated to provide the position and structural relationship between the surgical surface and the arm blood vessels, and at the same time provide the depth information of the subcutaneous blood vessels.

(5) The installation and deployment of the present invention is simpler: real-time blood vessel visualization on the body surface can be achieved using only a commercial projector and a commercial RGBD camera.

(6) The present invention has a more flexible blood vessel visualization method: only one photoacoustic imaging is required. The vascular image can be used all the time without the need for additional fixtures for real-time imaging. In addition, the target tracking algorithm can recognize any patient posture, and the vascular image can be accurately visualized on the body surface in real time with the naked eye without the need for additional fixtures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic structural diagram of a device for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to an embodiment of the present invention;

FIG2 is a schematic diagram of the structure of a photoacoustic imaging probe according to an embodiment of the present invention;

FIG3 is a schematic diagram of a flow chart of a target tracking algorithm according to an embodiment of the present invention;

FIG4 is a schematic diagram of a flow chart of an approximate elliptical surface projection algorithm according to an embodiment of the present invention;

FIG5 is a schematic diagram of the structure of an ellipse model according to an embodiment of the present invention;

FIG6 is a schematic diagram of a neural network structure of a target tracking algorithm according to an embodiment of the present invention.

Explanation of the accompanying drawings: 1. Pulse controller; 2. Pulse laser; 3. Pulse laser beam; 4. Fiber coupler; 5. Fiber bundle; 6. Photoacoustic imaging probe; 6-1. Cylindrical lens; 6-2. Reflector; 6-3 Laser beam, 6-4 Multi-element ultrasonic transducer; 6-5. Precision slide rail; 7. Focus line spot; 8. Tissue surface; 9. Amplifier circuit; 10. Data acquisition system; 11. Computer; 12. Projector; 13. RGBD camera.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the present application, the following will be combined with the drawings in the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are The embodiments all belong to the protection scope of this application.

Example 1

As shown in FIG1 , the present embodiment provides a real-time naked-eye visualization device for vascular body surface based on photoacoustic imaging, comprising a pulse controller 1, a pulse laser 2, a pulse laser beam 3, a fiber coupler 4, a fiber bundle 5, a photoacoustic imaging probe 6, a focal line spot 7, a tissue surface 8, an amplifier circuit 9, a data acquisition system 10, a computer 11, a projector 12, and an RGBD camera 13; the computer 11 communicates with the pulse controller 1, controls the pulse controller 1 to generate a pulse signal to drive the pulse laser 2 to emit a pulse laser beam 3, and the laser beam is coupled into the fiber bundle 5 through the fiber coupler 4, and the fiber bundle is connected to a photoacoustic imaging probe 6 to form a focal line spot 7 on the tissue surface 8 for exciting a photoacoustic signal, and the excited photoacoustic signal is received by the photoacoustic imaging probe, amplified by the amplifier circuit 9, and then collected and stored by the data acquisition system 10, and finally a photoacoustic vascular image is reconstructed on the computer 11. After a complete imaging, the projector 12 can be used in combination with an approximate elliptical surface fitting algorithm to project the photoacoustic vascular image onto the tissue surface 8 of any posture, and the RGBD camera 13 can be used in combination with a target tracking algorithm based on deep learning to perform vascular image registration and target tracking, so that when the tissue surface 8 moves involuntarily, the vascular image can still be accurately projected onto the tissue surface 8, achieving real-time naked eye visualization effect.

Further, as shown in FIG2 , the photoacoustic imaging probe 6 includes a cylindrical lens 6-1, two reflectors 6-2, and a 128-element multi-element ultrasonic transducer 6-4. The laser beam 6-3 from the optical fiber bundle is focused by the cylindrical lens 6-1, and after focusing, it is reflected by the reflector 6-2, so that the focus line spot 7 is just below the multi-element ultrasonic transducer 6-4. The cylindrical lens 6-1 is connected to the precision slide rail 6-5, and the distance between the cylindrical lens 6-1 and the optical fiber bundle can be adjusted by adjusting the slide rail 6-5 to adjust the spot size of the focus line spot 7, thereby adjusting different imaging depths and imaging resolutions.

Furthermore, the multi-element ultrasonic transducer array is composed of 128 ultrasonic transducers with a main frequency of 10 MHz.

Furthermore, when the photoacoustic imaging probe 6 performs photoacoustic imaging on tissues, different imaging methods can be selected according to different scenarios and different needs. The tissues can be scanned and imaged in a handheld manner, or the photoacoustic imaging probe 6 can be placed on a two-dimensional motor structure platform for mechanical scanning imaging.

Furthermore, the target tracking algorithm based on deep learning includes two networks, a target recognition network and a feature extraction network. The target tracking algorithm process is shown in FIG3. First, the RGBD camera 13 collects images of the tissue surface 8 at a frame rate of 30fps, and then inputs the collected images into the target recognition network. The network is the product of training the tissue surface 8 and the non-tissue surface using an improved lightweight MobileNet network. The target recognition network can detect and identify the tissue surface 8, and then focus on the tissue surface area, while eliminating the interference of the non-tissue surface. Then the camera image is input into the feature extraction network to extract features from the area of the tissue surface 8. The network is the product of customized training of the feature points of the tissue surface 8 using an improved convolutional neural network CNN, with the purpose of solving the problem of weak texture of skin tissue. After the feature extraction is completed, the posture is solved and the moment judgment is performed. If the posture changes, the projected image needs to be transformed so that the vascular image can still be accurately aligned when reprojected to the tissue surface 8.

The improved lightweight MobileNet network is shown in the dotted box on the left side of FIG6 , specifically: three additional convolutional layers of 1x1, 3x3 and 5x5 are added before the original 1x1 convolutional layer of the MobileNet network (as shown in the dark gray box on the left side of FIG6 ), and a pooling layer is added after the original 1x1 convolutional layer (as shown in the dark gray box on the left side of FIG6 ) to adjust the size and number of channels of the feature map; the newly added three convolutional layers are used to enhance the performance of feature extraction and recognition of the target, and the one pooling layer is used to prevent overfitting;

The improved convolutional neural network CNN is shown in the dotted box on the right side of FIG6 , specifically: on the basis of the original 3x3 convolution layer of the CNN network, a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers are added in parallel (as shown in the dark gray box on the right side of FIG6 ), that is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and the pooling layer; the newly added The output of the 3x3 convolutional layer is also connected to the fully connected layer through the newly added 5x5 convolutional layer and pooling layer; the newly added two convolutional layers can better extract features, and the newly added two pooling layers can improve the scale invariance and rotation invariance when extracting features from the target.

Furthermore, the approximate elliptical surface projection algorithm includes three-dimensional surface imaging of the tissue surface 8, accurate mathematical modeling and approximate ellipse fitting. The algorithm flow chart of the approximate elliptical surface projection algorithm is shown in Figure 4. First, the RGBD camera 13 collects RGB images and depth, and then uses the collected images to perform dense three-dimensional surface reconstruction on the tissue surface 8. The reconstructed model can easily obtain the three-dimensional coordinates of the edge vertex A of the imaging area and the highest point B of the model. The coordinates can be used to solve the x, y, and z distances between A and B in three-dimensional space. Use x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse to establish an ellipse model, and the specific geometric model is shown in Figure 5. The circumference of the ellipse is solved using the method of approximate ellipse fitting. The solved 1/4 ellipse circumference is the curve distance between A and B in the two-dimensional vertical plane. By enlarging the projected image using the ratio of the 1/4 ellipse circumference and the major semi-axis, accurate projection on the curved tissue can be achieved.

Furthermore, the real-time naked-eye visualization device of the vascular surface based on photoacoustic imaging can also perform three-dimensional surface reconstruction of the surgical surface through the RGBD camera 13. The reconstructed three-dimensional surface model can be fused with the three-dimensional photoacoustic vascular image to view the vascular image, the position of the subcutaneous blood vessels and the position of the arm surface, the structural relationship and the vascular depth information in the three-dimensional point cloud space, so as to provide assistance for preoperative planning.

Furthermore, the projector 12 and the RGBD camera 13 form a visual projection tracking device, which can be freely set at any height directly above the tissue surface 8 and connected to the computer 11. By adjusting the height from the tissue surface 8, the range of vascular surface visualization can be adjusted. After each height adjustment, only a simple device calibration needs to be completed on the computer 11 to complete the configuration of the visual projection tracking device. The device calibration method is specifically as follows:

The customized standard photoacoustic imaging samples and the images reconstructed after the sample photoacoustic imaging were used to extract The characteristic points in the sample image captured by the RGBD camera 13 and the photoacoustic image reconstructed after imaging are calibrated by solving the transformation relationship between the sample image and the corresponding characteristic points on the photoacoustic image; the customized standard photoacoustic imaging sample is made of 20 carbon rods with a length of 30 mm and a diameter of 0.5 mm arranged in a checkerboard shape in agar.

Furthermore, after photoacoustic imaging, no markers are used for registration, and the tissue surface 8 can be captured by the RGBD camera 13 in any posture and transmitted to the computer 11. In the computer 11, the feature points of the tissue surface 8 in any posture can be extracted, the registration relationship between the vascular image and the tissue surface 8 can be accurately estimated, and the projected vascular image can be affine transformed. Finally, after projection by a projector, it can be accurately visualized in situ on the skin surface with the naked eye.

It should be noted here that the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. The device is a real-time naked-eye visualization method of the vascular surface based on photoacoustic deep and high-resolution imaging that can be applied to the following embodiments.

Example 2

In this embodiment, a method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging is provided. The method can apply a real-time naked-eye visualization device for blood vessel surface based on photoacoustic imaging of the above embodiment, and comprises the following steps:

S1, generate a pulsed laser beam 3 and focus it on the tissue surface 8 through the photoacoustic imaging probe 6 to form a line spot, specifically:

The computer 11 controls the pulse controller 1 to send a pulse signal, which in turn drives the pulse laser 2 to generate a pulse laser beam 3. The generated pulse laser beam 3 is coupled into the optical fiber bundle 5 through the optical fiber coupler 4. The optical fiber bundle 5 is connected to the photoacoustic imaging probe 6. After passing through the photoacoustic imaging probe 6, the pulse laser beam 3 is formed in the group The fabric surface 8 forms a focused line spot 7;

S2, collecting the photoacoustic signal generated by the photoacoustic imaging probe 6 and processing and reconstructing it by the computer 11 to obtain a photoacoustic blood vessel image;

S3, projecting the reconstructed photoacoustic blood vessel image onto the tissue surface 8 by means of the projector 12 in combination with an approximate ellipse fitting surface projection algorithm;

Furthermore, the approximate ellipse fitting surface projection algorithm is specifically as follows:

Performing three-dimensional surface reconstruction of the tissue surface 8 using the RGBD camera 13;

Mathematical modeling of three-dimensional surface models;

The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model, and the distances x, y, and z between two points in the three-dimensional space are solved with these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted by an approximate ellipse fitting method, and this fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in a corresponding proportion with the ratio of the length of the ellipse curved edge to the major axis of the ellipse, so as to realize the projection of the two-dimensional vascular image on the curved surface tissue.

S4. The projection image of the tissue surface 8 is collected in real time by the RGBD camera 13 and transmitted to the computer 11. The posture of the tissue surface 8 is solved in real time by the learning-based target tracking algorithm. When the tissue surface 8 moves involuntarily, the vascular image tracking projection is performed.

Furthermore, the learning-based target tracking algorithm specifically includes:

(1) Input layer for inputting images;

(2) The target recognition network for feature extraction through the MobileNet network, as shown in the dashed box on the left side of Figure 6, specifically: three additional convolutional layers of 1x1, 3x3 and 5x5 are added before the original 1x1 convolutional layer of the MobileNet network (as shown in the dark gray box on the left side of Figure 6), and a pooling layer is added after the original 1x1 convolutional layer (as shown in the dark gray box on the left side of Figure 6) to adjust the size and number of channels of the feature map; The three newly added convolutional layers are used to enhance the performance of target feature extraction and recognition, and the one pooling layer is used to prevent overfitting;

(3) The feature extraction network for feature point extraction through the CNN network is shown in the dotted box on the right side of Figure 6. Specifically, a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers are added in parallel on the basis of the original 3x3 convolution layer of the CNN network (as shown in the dark gray box on the right side of Figure 6). That is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and pooling layer; the newly added two convolution layers can better extract features, and the newly added two pooling layers can improve the scale invariance and rotation invariance when extracting features from the target.

(4) Attention mechanism, which inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps

Furthermore, the projector 12 and the RGBD camera 13 form a visual projection tracking device, which can be freely set at any height directly above the tissue surface 8 and connected to the computer 11. The range of vascular surface visualization can be adjusted by adjusting the height from the tissue surface 8. After each height adjustment, only a simple device calibration needs to be completed on the computer 11 to complete the configuration of the visual projection tracking device. The device calibration method is specifically as follows:

A customized standard photoacoustic imaging sample and an image reconstructed after sample photoacoustic imaging are used to extract feature points in the sample image captured by the RGBD camera 13 and the photoacoustic image reconstructed after imaging, and calibration is completed by solving the transformation relationship between the corresponding feature points on the sample image and the photoacoustic image; the customized standard photoacoustic imaging sample is made of 20 carbon rods with a length of 30 mm and a diameter of 0.5 mm arranged in a checkerboard shape in agar.

Furthermore, after photoacoustic imaging, no markers are used for registration, and the tissue surface 8 is randomly The posture is captured by the RGBD camera 13 and transmitted to the computer 11, in which the characteristic points of the tissue surface 8 of any posture are extracted, the registration relationship between the vascular image and the tissue surface 8 is estimated, and the projected vascular image is affine transformed, and finally projected by the projector.

Furthermore, only one complete photoacoustic imaging is required to locate and track the tissue surface 8 through the RGBD camera 13 in combination with the target tracking algorithm, so that the imaging results can be used for naked-eye visualization of surface blood vessels.

It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims (10)

A method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging, characterized in that it comprises the following steps: Generate pulsed laser and focus it on the tissue surface through the photoacoustic imaging probe to form a light spot; The photoacoustic signal is collected by a photoacoustic imaging probe and processed and reconstructed by a computer to obtain a photoacoustic vascular image; The reconstructed photoacoustic vascular image is projected onto the tissue surface by a projector combined with an approximate ellipse fitting surface projection algorithm; The projection image of the tissue surface is collected in real time by an RGBD camera and transmitted to a computer. The posture of the tissue surface is solved in real time using a learning-based target tracking algorithm. When the tissue surface moves involuntarily, vascular image tracking and projection are performed. The method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 1 is characterized in that the pulsed laser is generated and focused on the tissue surface to form a light spot through a photoacoustic imaging probe, specifically: The pulse controller is controlled by a computer to send out a pulse signal, which in turn drives the pulse laser to generate a pulse laser. The generated pulse laser is coupled into the optical fiber bundle through the optical fiber coupler. The optical fiber bundle is connected to the photoacoustic imaging probe. After the pulse laser passes through the photoacoustic imaging probe, a focal line spot is formed on the tissue surface. The real-time naked-eye visualization method of blood vessel surface based on photoacoustic imaging according to claim 1 is characterized in that the approximate ellipse fitting surface projection algorithm is specifically: Using the RGBD camera to perform three-dimensional surface reconstruction on the tissue surface; Mathematical modeling of three-dimensional surface models; The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model. The distances x, y, and z between two points in the three-dimensional space are solved with these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted by the approximate ellipse fitting method. The curved edge of this fitting That is, the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in proportion to the ratio of the length of the ellipse curved edge to the major axis of the ellipse to realize the projection of the two-dimensional vascular image on the curved surface tissue. The method for real-time naked-eye visualization of vascular surface based on photoacoustic imaging according to claim 1 is characterized in that the learning-based target tracking algorithm specifically includes: Input layer for inputting images; The object recognition network uses the MobileNet network for feature extraction, specifically: adding three additional convolutional layers of 1x1, 3x3, and 5x5 before the original 1x1 convolutional layer of the MobileNet network, and adding a pooling layer after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map; A feature extraction network for extracting feature points through a CNN network, specifically: adding a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers in parallel on the basis of the original 3x3 convolution layer of the CNN network, that is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and the pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and the pooling layer; The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps. The method for real-time naked-eye visualization of vascular body surface based on photoacoustic imaging according to claim 1 is characterized in that the device calibration method of the projector and the RGBD camera is specifically as follows: A customized standard photoacoustic imaging sample and an image reconstructed after sample photoacoustic imaging are used to extract feature points in the sample image acquired by the RGBD camera and the photoacoustic image reconstructed after imaging, and calibration is completed by solving the transformation relationship between the corresponding feature points on the sample image and the photoacoustic image; the customized standard photoacoustic imaging sample is made by arranging carbon rods in agar in a checkerboard shape. According to the method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 1, The method is characterized in that no marker is used for registration after photoacoustic imaging, the tissue surface is captured by the RGBD camera in an arbitrary posture and transmitted to a computer, feature points of the tissue surface in an arbitrary posture are extracted in the computer, the registration relationship between the vascular image and the tissue surface is estimated, and the projected vascular image is affine transformed, and finally projected by a projector. A real-time naked-eye visualization device for blood vessel surface based on photoacoustic imaging, characterized in that it includes a pulsed laser generating device, a photoacoustic imaging probe, an acquisition and amplification circuit, a computer, a projector, and an RGBD camera; The pulse laser generating device is connected to the photoacoustic imaging probe and is used to generate pulse laser and focus the pulse laser on the tissue surface through the photoacoustic imaging probe to form a light spot; The photoacoustic imaging probe is also used to collect the generated photoacoustic signals and transmit them to the computer via the collection and amplification circuit; The computer is used for processing and reconstructing photoacoustic blood vessel images, and is loaded with an approximate ellipse fitting surface projection algorithm and a learning-based target tracking algorithm; The projector is used to project the reconstructed photoacoustic blood vessel image onto the tissue surface in combination with an approximate ellipse fitting surface projection algorithm; The RGBD camera is used to align the projected photoacoustic vascular image with the tissue surface in combination with a learning-based target tracking algorithm, and to reproject when involuntary movement occurs on the tissue surface. According to the real-time naked-eye visualization device of blood vessel surface based on photoacoustic imaging as described in claim 7, it is characterized in that the pulse laser generating device includes a pulse controller, a pulse laser, a fiber coupler, and a fiber bundle which are connected in sequence; the pulse controller is controlled by a computer to send a pulse signal, thereby driving the pulse laser to generate a pulse laser, and the generated pulse laser is coupled into the fiber bundle through the fiber coupler, and the fiber bundle is connected to the photoacoustic imaging probe, and the pulse laser is focused on the tissue surface to form a line spot after passing through the photoacoustic imaging probe. According to claim 7, the real-time naked-eye visualization device of vascular body surface based on photoacoustic imaging is characterized in that the photoacoustic imaging probe includes a cylindrical lens, a reflector and a multi-element ultrasonic transducer array, the optical fiber bundle is focused into a line spot by the cylindrical lens, and enters the tissue surface vertically after being reflected 45 degrees by two reflectors respectively, and the multi-element ultrasonic transducer array is located next to the cylindrical lens and directly above the focused line spot, and is used to receive the generated photoacoustic signal; the cylindrical lens is placed in a precision slide bracket, and the spot size of the focused line spot is adjusted by adjusting the distance of the precision slide from the optical fiber bundle, thereby adjusting the imaging depth and imaging resolution; the multi-element ultrasonic transducer array is composed of 128 ultrasonic transducers with a main frequency of 10 MHz. The real-time naked-eye visualization device for vascular body surface based on photoacoustic imaging according to claim 7 is characterized in that the approximate ellipse fitting surface projection algorithm is specifically: Using the RGBD camera to perform three-dimensional surface reconstruction on the tissue surface; Mathematical modeling of three-dimensional surface models; The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model, and the distances x, y, and z between two points in the three-dimensional space are solved with these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted by an approximate ellipse fitting method, and this fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in a corresponding proportion with the ratio of the length of the ellipse curved edge to the major axis of the ellipse, so as to realize the projection of the two-dimensional vascular image on the curved surface tissue. The learning-based target tracking algorithm includes: Input layer for inputting images; The object recognition network uses the MobileNet network for feature extraction, specifically: adding three additional convolutional layers of 1x1, 3x3, and 5x5 before the original 1x1 convolutional layer of the MobileNet network, and adding a pooling layer after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map; A feature extraction network for extracting feature points through a CNN network, specifically: adding a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers in parallel on the basis of the original 3x3 convolution layer of the CNN network, that is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and the pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and the pooling layer; The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps.