CN115327541B - Array scanning holographic penetration imaging method and handheld holographic penetration imaging radar system - Google Patents
Array scanning holographic penetration imaging method and handheld holographic penetration imaging radar system Download PDFInfo
- Publication number
- CN115327541B CN115327541B CN202211246412.4A CN202211246412A CN115327541B CN 115327541 B CN115327541 B CN 115327541B CN 202211246412 A CN202211246412 A CN 202211246412A CN 115327541 B CN115327541 B CN 115327541B
- Authority
- CN
- China
- Prior art keywords
- scanning
- radar probe
- detected
- imaging
- radar
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 120
- 230000035515 penetration Effects 0.000 title claims description 43
- 239000000523 sample Substances 0.000 claims abstract description 126
- 238000001514 detection method Methods 0.000 claims abstract description 74
- 238000012545 processing Methods 0.000 claims abstract description 35
- 238000000034 method Methods 0.000 claims abstract description 27
- 230000009466 transformation Effects 0.000 claims abstract description 10
- 230000008859 change Effects 0.000 claims description 15
- 230000000007 visual effect Effects 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 13
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 5
- 230000004807 localization Effects 0.000 claims description 4
- 230000003287 optical effect Effects 0.000 claims description 4
- 238000009825 accumulation Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 2
- 238000004148 unit process Methods 0.000 claims description 2
- 230000000149 penetrating effect Effects 0.000 abstract description 4
- 238000003491 array Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 abstract description 2
- 238000010304 firing Methods 0.000 abstract 1
- 238000004590 computer program Methods 0.000 description 10
- 239000002344 surface layer Substances 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 230000001427 coherent effect Effects 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 230000033001 locomotion Effects 0.000 description 4
- 238000007689 inspection Methods 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000002592 echocardiography Methods 0.000 description 2
- 239000010410 layer Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 230000017105 transposition Effects 0.000 description 2
- 239000002023 wood Substances 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000010985 leather Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000007769 metal material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000009659 non-destructive testing Methods 0.000 description 1
- 229910052755 nonmetal Inorganic materials 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/88—Radar or analogous systems specially adapted for specific applications
- G01S13/89—Radar or analogous systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
- G01S7/04—Display arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Physics & Mathematics (AREA)
- Electromagnetism (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
本申请涉及一种阵列扫描全息穿透成像方法及手持全息穿透成像雷达系统。所述方法包括：通过利用多发多收阵列雷达探头在介质表面的待探测区域内移动的同时进行连续多次快速扫描得到回波数据，在雷达探头在进行扫描的同时，利用图像记录雷达探头在移动时的位置变换，后续通过对位置图像进行处理得到雷达探头扫描是的位置坐标，再根据位置坐标以及回波数据进行计算最后得到待探测区域表层下目标的成像结果，在该方法中，将多发多收阵列雷达探测得到的回波数据结合利用图像记录其扫描轨迹的方式，相比于直接采用大型阵列的方案，既保证了探测效果，又显著减小了系统规模，降低成本。
The present application relates to an array scanning holographic penetrating imaging method and a handheld holographic penetrating imaging radar system. The method includes: obtaining echo data by performing multiple consecutive rapid scans while the radar probe is moving in the area to be detected on the surface of the medium by using the multi-firing and multi-receiving array radar probe; The position transformation when moving, and then the position coordinates of the radar probe scanning are obtained by processing the position images, and then calculated according to the position coordinates and echo data, and finally the imaging results of the target under the surface of the area to be detected are obtained. In this method, the Compared with the direct use of large-scale arrays, the echo data obtained by the multi-transmission and multi-reception array radar combined with the image recording method not only ensures the detection effect, but also significantly reduces the system scale and reduces the cost.
Description
Technical Field
The application relates to the technical field of microwave nondestructive detection, in particular to an array scanning holographic penetration imaging method and a handheld holographic penetration imaging radar system.
Background
Electromagnetic waves in the microwave frequency range can penetrate through a non-metal medium to propagate inside the medium, such as a wall, a wood board, leather, plastic and the like, and electromagnetic scattering can be caused at discontinuous parts of electromagnetic properties in the medium (such as buried objects, internal defects and the like). The holographic penetration imaging radar technology just utilizes the characteristic, each grid point transmits coherent electromagnetic waves and receives echoes by scanning on a two-dimensional plane grid, and a high-resolution holographic image of the electromagnetic scattering characteristic under the surface layer of the medium is obtained by coherent synthesis of two-dimensional space of the amplitude and the phase of the echoes, so that the condition of the internal structure of the medium or the buried target is visually reflected. Under the condition of not damaging the surface of the medium, the nondestructive detection of the interior of the medium is realized, and the method has great application value in the fields of security inspection, material detection, building flaw detection and the like.
The holographic penetration imaging radar probe needs to scan on a two-dimensional plane grid to complete spatial coherent synthesis, so that a mechanical or electromechanical device is needed to accurately control the position of the radar probe. A Holographic penetration imaging Radar system adopting a positioning wheel and a scale to scan is reported in the literature, namely, hologrphic surface Radar of SCAN types and Applications (S.I. Ivashov, and the like, published in the Journal of Selected topocs in Applied Earth observation objectives and Remote Sensing Journal, no. 4 of 2011, pages 763-778), wherein the Holographic penetration imaging Radar system adopts handheld operation, one direction is triggered by the positioning wheel to roll at equal intervals to realize uniform grid scanning, and the other direction moves linearly along the scale to control scanning intervals. Document "High Resolution Imaging with a volumetric Radar Mounted on a rotating Scanner" (l. Camperi et al, published In 2013 In the development In electronics Research Symposium Proceedings, page 1583) reports a system mode for loading a Radar probe with a four-wheel cart, which moves In one direction by a linear sliding table, and moves In the other direction by the cart, and operates In a stop-go-stop mode, so that automation is achieved, but the speed is still slow, and factors such as wheel slip and start-stop jitter affect a scanning Imaging result, and frequent start-stop also greatly limits a scanning speed.
Real-time or quasi-real-time holographic penetration imaging can be achieved by using an array antenna and replacing mechanical scanning with electric scanning, and airport personnel Security inspection system based on holographic penetration imaging Technology is reported in the document Walk Through Screening with multistative W Technology (Frank Gummmann et al, SPIE Security + Defence, proceedings Volume 9993, millimetrewave and Terahertz Sensors and Technology IX, p.999306), which realizes real-time scanning microwave holographic imaging. However, such systems are large in size and weight, complex, and costly, and are used primarily for stationary inspections in large locations.
In data processing, the current holographic penetration imaging algorithm is based on two-dimensional Fourier transform, and requires that sampling points are uniformly distributed on a two-dimensional plane grid, so that a radar scanning track is required to be controlled on a uniform grid line. For non-uniform spatial scanning, such as sparse antenna arrays, uniform grid interpolation is required, which limits the flexibility of the system.
The usability, the applicability and the usability of the holographic penetration imaging system are limited by the problems, and a new technology needs to be innovated and developed in multiple aspects of radar systems, scanning modes, algorithm processing and the like, so that novel available equipment is provided for nondestructive testing requirements.
Disclosure of Invention
In view of the above, it is necessary to provide an array scanning holographic penetration imaging method and a handheld holographic penetration imaging radar system capable of solving the above problems.
A method of array scanning holographic transfixing imaging, the method comprising:
acquiring detection data, wherein the detection data are echo data obtained by carrying out continuous and multiple rapid scanning while a radar probe with a multi-transmitting and multi-receiving array antenna moves in a region to be detected on the surface of a medium;
acquiring positioning data, wherein the positioning data are a plurality of position images for recording position changes of a radar probe when the radar probe performs multiple rapid scans in the area to be detected, and each position image corresponds to the position change of the radar probe during one rapid scan;
processing the positioning data to obtain the position coordinates of the radar probe in the area to be detected during each rapid scanning;
and processing according to the position coordinates of the radar probe during each quick scanning and the corresponding detection data to obtain the imaging result of the target under the surface layer of the area to be detected.
In one embodiment, when the radar probe of the multiple-input multiple-output array antenna performs fast scanning, one antenna is a transmitting antenna, the other antennas are receiving antennas, the other antenna is switched to be a transmitting antenna, the other antennas are receiving antennas, and the fast scanning is performed after all the antennas are traversed by analogy.
In one embodiment, the handheld radar probe reciprocates on the surface of the medium when performing quick scanning until the moving track covers the whole area to be detected.
In one embodiment, the processing the positioning data to obtain the position coordinates of the radar probe in the region to be detected during each fast scan includes:
calibration code elements are arranged on the boundary of a region to be detected on the surface of the medium, the number of the calibration code elements is three, the calibration code elements are respectively arranged at the upper left corner, the lower left corner and the upper right corner of the boundary of the region to be detected, the actual position coordinates of each calibration code element are obtained, and a positioning code element is also arranged on one side, back to the surface of the medium, of the radar probe;
carrying out contour detection and feature extraction on each position image, and identifying each calibration code element and the pixel coordinate of the center point of the positioning code element;
constructing a coordinate transformation matrix according to the pixel coordinates and the actual position coordinates of each calibration code element in each position image;
and converting the pixel coordinates of the central point of the positioning code element in each position image into position coordinates according to the coordinate transformation matrix, wherein the position coordinates are actual position coordinates in the area to be detected when the radar probe performs quick scanning each time.
In one embodiment, the processing according to the position coordinate of the radar probe during each fast scanning and the corresponding detection data to obtain the imaging result of the subsurface target of the region to be detected includes:
constructing an imaging space according to the region to be detected and a preset imaging depth;
calculating according to the imaging space and the position coordinates of the radar probe during each quick scanning to obtain a compensation phase;
and performing point-by-point compensation phase accumulation summation according to the detection data to obtain the imaging result.
A handheld holographic penetration imaging radar system, the system comprising: the system comprises a handheld radar probe, a visual positioning unit, an imaging processing unit and a scanning control unit;
the scanning control unit respectively sends detection instructions to the handheld radar probe and the visual positioning unit;
the handheld radar probe is a multi-transmitting multi-receiving antenna array radar, and is used for carrying out continuous and multi-time rapid scanning while moving in a to-be-detected area on the surface of a medium according to the detection instruction to obtain echo data and sending the echo data serving as detection data to the scanning control unit;
the visual positioning unit records a plurality of position images of the radar probe during multiple rapid scanning in the area to be detected according to the detection instruction, each position image corresponds to the position change of the radar probe during one rapid scanning, and the position images are used as positioning data to be sent to the scanning control unit;
the scanning control unit sends the received detection data and the positioning data to the imaging processing unit;
the imaging processing unit processes the detection data and the positioning data according to the array scanning holographic penetration imaging method to obtain an imaging result of the target under the surface layer of the area to be detected, and sends the imaging result to the scanning control unit;
the scanning control unit receives and displays the imaging result.
In one embodiment, the handheld radar probe is an integrated radar, and comprises a microcontroller, a multi-channel radio frequency transceiver chip, a radio frequency switch, a communication element and a plurality of antennas, wherein the plurality of antennas are integrated on a planar board to form an antenna array.
In one embodiment, the visual positioning unit comprises a camera device, three scaling code elements and a positioning code element;
the camera lens of the camera device is over against the area to be detected, and the camera shooting range of the camera device comprises all the area to be detected;
the three calibration code elements are respectively arranged at the upper left corner, the lower left corner and the upper right corner of the boundary of the area to be detected;
the positioning code element is arranged on one side, back to the medium surface, of the radar probe.
In one embodiment, the calibration code element and the positioning code element adopt optical mark patterns including annular code marks, cross code marks and two-dimensional code marks.
In one embodiment, the scan control unit is an upper computer.
A holographic transfixion imaging apparatus, said apparatus comprising:
the detection data acquisition module is used for acquiring detection data, wherein the detection data are echo data obtained by performing continuous and multiple rapid scanning while a radar probe with a multi-transmitting and multi-receiving array antenna moves in a region to be detected on the surface of a medium;
the positioning data acquisition module is used for acquiring positioning data, the positioning data are a plurality of position images for recording position changes of the radar probe when the radar probe performs multiple rapid scans in the area to be detected, and each position image corresponds to the position change of the radar probe during one rapid scan;
the position coordinate obtaining module is used for processing the positioning data to obtain the position coordinate of the radar probe in the area to be detected during each quick scanning;
and the imaging result obtaining module is used for processing according to the position coordinates of the radar probe in each quick scanning and the corresponding detection data to obtain the imaging result of the target under the surface layer of the area to be detected.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring detection data, wherein the detection data are echo data obtained by carrying out continuous and multiple rapid scanning while a radar probe with a multi-transmitting and multi-receiving array antenna moves in a region to be detected on the surface of a medium;
acquiring positioning data, wherein the positioning data are a plurality of position images for recording position changes of a radar probe when the radar probe performs multiple rapid scans in the area to be detected, and each position image corresponds to the position change of the radar probe during one rapid scan;
processing the positioning data to obtain the position coordinates of the radar probe in the area to be detected during each rapid scanning;
and processing according to the position coordinates of the radar probe during each quick scanning and the corresponding detection data to obtain the imaging result of the target under the surface layer of the area to be detected.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring detection data, wherein the detection data are echo data obtained by carrying out continuous and multiple rapid scanning while a radar probe with a multi-transmitting and multi-receiving array antenna moves in a region to be detected on the surface of a medium;
acquiring positioning data, wherein the positioning data are a plurality of position images for recording position changes of a radar probe when the radar probe performs multiple rapid scans in the area to be detected, and each position image corresponds to the position change of the radar probe during one rapid scan;
processing the positioning data to obtain a position coordinate of the radar probe in the area to be detected during each quick scanning;
and processing according to the position coordinates of the radar probe during each quick scanning and the corresponding detection data to obtain the imaging result of the target under the surface layer of the area to be detected.
According to the array scanning holographic penetration imaging method and the handheld holographic penetration imaging radar system, echo data are obtained by continuously and rapidly scanning multiple times while the multiple-shot array radar probe moves in a to-be-detected area on the surface of a medium, position transformation of the radar probe during movement is recorded by using an image while the radar probe scans, position coordinates of the radar probe during scanning are obtained by subsequently processing a position image, and an imaging result of a target under the surface layer of the to-be-detected area is finally obtained by calculating according to the position coordinates and the echo data.
Drawings
FIG. 1 is a schematic flow chart of a method for array scanning holographic transmission imaging in one embodiment;
FIG. 2 is a block diagram of a hand-held holographic penetration imaging radar system in one embodiment;
FIG. 3 is a schematic diagram of an antenna array layout of a multiple-transmit multiple-receive radar probe in one embodiment;
FIG. 4 is a simplified schematic diagram of a scanning scenario during an experiment, further showing the preset target shape, size and dimensions within the medium;
FIG. 5 is a schematic diagram of a movement track of a radar probe calculated from positioning data in an experiment;
FIG. 6 is a schematic diagram of imaging results obtained from an array scanning holographic transmission imaging method in an experiment;
FIG. 7 is a block diagram of a holographic transmission imaging device according to an embodiment;
FIG. 8 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
As shown in fig. 1, there is provided an array scanning holographic penetration imaging method, comprising the steps of:
step S100, acquiring detection data, wherein the detection data are echo data obtained by continuously and rapidly scanning multiple times while a multi-transmitting and multi-receiving array radar probe moves in a region to be detected on the surface of a medium;
step S110, acquiring positioning data, wherein the positioning data are a plurality of position images for recording position changes of the radar probe during a plurality of times of fast scanning in the area to be detected, and each position image corresponds to the position change of the radar probe during one time of fast scanning;
step S120, processing the positioning data to obtain the position coordinates of the radar probe in the area to be detected when the radar probe performs each quick scanning;
and step S130, processing according to the position coordinates of the radar probe during each quick scanning and the corresponding detection data to obtain the imaging result of the subsurface target of the area to be detected.
In step S100, the radar probe repeatedly moves in the region to be detected on the surface of the medium until the moving route covers the whole detection region with the radar probe, and the radar probe adopts a multi-transmission multi-reception array. When scanning, one antenna in the antenna array is used as a transmitting antenna, the other antennas are used as receiving antennas, then the other antenna is switched to be used as a transmitting antenna, the other antennas are used as receiving antennas, and the like, and when all the antennas are traversed, one-time quick scanning is finished. During the process that the radar probe finishes scanning the whole area to be detected, multiple times of rapid scanning are carried out.
In the embodiment, when the rapid scanning is performed, the radar probe reciprocates on the surface of the medium until the moving track covers the whole area to be detected.
Specifically, after the radar probe scans, the echo data, i.e. the detection data, received by each transceiving channel is corresponding to
Upper label ofnThe number of fast scans is indicated and,
、
respectively expressed in one fast scanmThe transmitting and receiving antenna positions of the transceiving combination.
In step S110, when the radar probe moves and scans the region to be detected, an image capturing device records the whole process, and a plurality of images record the moving track of the radar probe in the region to be detected. That is to say, each image records a different position of the radar probe in the area to be detected. And each position image corresponds to the position change of the radar probe during one-time quick scanning respectively.
In step S120, calibration code elements are arranged on the boundary of the to-be-detected region on the surface of the medium, the number of the calibration code elements is three, the calibration code elements are respectively arranged at the upper left corner, the lower left corner and the upper right corner of the boundary of the to-be-detected region, the actual position coordinates of each calibration code element are obtained, and a positioning code element is further arranged on one side of the radar probe, which faces away from the surface of the medium.
Further, a coordinate transformation relation is established according to the conversion from the shooting space of the position image to the physical space, and the actual position of each rapid scanning radar probe can be obtained according to the relation; and constructing a coordinate transformation matrix according to the pixel coordinates and the actual position coordinates of each calibration code element in each position image, and converting the pixel coordinates of the central point of the positioning code element in each position image into position coordinates according to the coordinate transformation matrix, wherein the position coordinates are the actual position coordinates in the region to be detected when the radar probe performs quick scanning each time.
Specifically, the pixel coordinates of the centers of three calibration code elements can be obtained by identifying each position image
Wherein 0, 1, 2 respectively represent the scaling code elements arranged at the upper left corner, the lower left corner and the upper right corner of the boundary of the region to be detected, and the subscript pix represents the pixel coordinates in the image. The actual position of the known scaled symbol is
The superscript T represents the matrix transposition operation, and the following relationship is given:
wherein,
（2）
solving from the pixel coordinates and the actual position coordinates of the three calibration symbols
And
further, a coordinate transformation matrix can be obtained
：
And the first place isnAt sub-fast scanLocating pixel coordinates of a symbol
The physical coordinates are transformed as follows:
in the formula (4), let us notenThe radar position at the time of the sub-snapshot is
。
In the embodiment, the scanning process of the radar probe is continuous, and a stop-go-stop mode is not needed, so that the detection efficiency is greatly improved compared with the point-by-point measurement in the prior art.
In the embodiment, the calibration code element and the positioning code element adopt optical mark patterns including annular code marks, cross-shaped code marks and two-dimensional code marks.
In this embodiment, the two-dimensional code identifier is used as the scaling symbol and the positioning symbol, and the coordinates of the center pixel of the scaling symbol and the positioning symbol can be detected from the image by the two-dimensional code detection technique.
In one embodiment, the scaling symbol and the positioning symbol are identified by a two-dimensional code, and the coordinates of the center pixel of the scaling symbol and the positioning symbol can be detected from the image by a two-dimensional code detection technology.
In step S130, an algorithm of point-by-point phase compensation coherent superposition imaging processing is adopted, and target imaging is performed by using the actual position coordinates of the radar and the detection data obtained in step S120 for each fast scanning, including: and finally, according to the detection data, carrying out accumulated summation by adopting point-by-point compensation phases to obtain the imaging result.
Specifically, when an imaging space is constructed, a region to be imaged is formedThe region to be detected is divided into grids with the grid interval of
，
And
respectively representxDirection andythe imaging range of the direction is set,
，
. The preset distance of the depth layer to be imaged is set as
And the coordinates of any point in the imaging space are recorded as
Thus, for a certain imaging point
Calculating the distance from the imaging point to the transmitting antenna and the receiving antenna of each channel
And
：
in the formula (5), the first and second groups,
representing an imaging point
To the firstnSecond in sub-fast scanmThe distance of the transmit antennas of the group transmit-receive combination,
representing an image point
To the firstnSecond in sub-fast scanmThe distance of the combined receive antenna is grouped.
Spatial wavenumber of known medium
Wherein
Is the frequency of the electromagnetic wave and,
is the propagation velocity of the electromagnetic wave in the medium, and the compensation phase is calculated by the following formula:
next, the phase accumulation summation is compensated point by point:
in the formula (7), the measurement data corresponding to each transceiving channel is
Upper label ofnThe number of fast scans is indicated and,
、
respectively expressed in a snapshotmThe transmitting and receiving antenna positions of the transceiving combination.
And finally, after imaging is finished, displaying to obtain a final imaging result in the medium so as to achieve the aim of detection.
As shown in fig. 2, the present application also provides a hand-held holographic penetration imaging radar system matched with the above method, comprising: the system comprises a handheld radar probe 1, a visual positioning unit, an imaging processing unit 2 and a scanning control unit 3;
firstly, a scanning control unit 3 respectively sends detection instructions to the handheld radar probe 1 and the visual positioning unit;
the handheld radar probe 1 moves in a region to be detected on the surface of a medium and simultaneously carries out continuous and multiple times of quick scanning to obtain echo data according to a detection instruction, and sends the echo data serving as detection data to the scanning control unit 3;
when the radar probe 1 is held by hand for scanning, the visual positioning unit records a plurality of position images of the radar probe, which change in position when the radar probe performs multiple times of quick scanning in the area to be detected, according to the detection instruction, wherein each position image corresponds to the position change of the radar probe during one time of quick scanning, and the position images are sent to the scanning control unit 3 as positioning data;
the scanning control unit 3 sends the received detection data and the positioning data to the imaging processing unit 2;
the imaging processing unit 2 processes the detection data and the positioning data according to the array scanning holographic penetration imaging method to obtain an imaging result of the target under the surface layer of the region to be detected, which has been described above, and is not described herein again, and sends the imaging result to the scanning control unit 3;
finally, the scan control unit 3 receives and displays the imaging result.
In this embodiment, the handheld radar probe 1 is a multiple-input multiple-output array integrated radar, and includes a microcontroller, a multi-channel rf transceiver chip, an rf switch, a communication element, and a plurality of antennas, where the plurality of antennas are integrated on a planar board to form an antenna array. And each antenna can be switched to a transmit or receive mode by the microcontroller through the radio frequency switch. When one antenna works in a transmitting mode, the other antennas are in a receiving state, and multiple-transmitting and multiple-receiving detection is realized by time-sharing switching of transmitting and receiving.
In the present embodiment, the antenna array layout of the multiple-transmit multiple-receive handheld radar probe 1 is shown in fig. 3, and includes 18 transmit-receive antennas, which operate in the C-band. The distance between the antennas is 2cm, and the array range is 6cm × 8cm. The local coordinates of each antenna within the array using the center of the array as the origin are noted
Middle and upper label of the formula
The numbers of the antennas are shown, and the superscript T indicates the matrix transposition operation.
After the radar probe receives a scanning instruction of the scanning control module, the microcontroller firstly transmits the No. 1 antenna and receives the rest antennas to form a receiving-transmitting combination of 1 transmitting and 17 receiving. Then, the No. 2 antenna is switched to be transmitting and the other antennas are used for receiving, and so on, and all the antennas are traversed. And removing repeated receiving and transmitting combinations to obtain 153 receiving and transmitting channel data, recording the channel data as one snapshot, and transmitting the snapshot to the scanning control module through the communication interface. The measured data corresponding to each transceiving channel is
Upper label ofnThe number of times of the snap shots is indicated,
、
respectively expressed in a snapshotmTransmitting and receiving of a transmitting and receiving combinationAnd receiving the antenna position.
In the present embodiment, the visual localization unit includes an image pickup device 41, three scaling symbols 42, and one localization symbol 43. The imaging lens of the imaging device 41 is treating the region to be detected, and the imaging range thereof includes the entire region to be detected. The three calibration code elements 42 are respectively arranged at the upper left corner, the lower left corner and the upper right corner of the boundary of the area to be detected. And the positioning code element 43 is arranged on the side of the handheld radar probe 1, which is opposite to the surface of the medium, so that the positioning code element 43 moves along with the handheld radar probe 1, and the position coordinates of the handheld radar probe 1 are finally determined by calculating the position coordinates of the positioning code element 43.
In the present embodiment, the calibration symbols 42 and the positioning symbols 43 adopt optical mark patterns, including annular coded marks, cross-shaped coded marks and two-dimensional code marks.
In one embodiment, the scaling symbols 42 and the positioning symbols 43 are identified by two-dimensional codes, and the coordinates of the center pixels of the scaling symbols 42 and the positioning symbols 43 can be detected from the image by a two-dimensional code detection technique.
In the present embodiment, the scan control unit 3 is an upper computer. The imaging processing unit 2 may be a calling program in an upper computer, may also be other computer devices, and may also use a storage medium as a carrier.
When the handheld holographic penetration imaging radar system is used for detecting a region to be detected:
firstly, arranging a calibration code element at the upper left corner, the upper right corner and the lower left corner of the boundary of a region to be detected, as shown in fig. 4, and also showing a target shape prearranged in the board in fig. 4, then arranging a positioning code element on a handheld radar probe, arranging a camera at a position right facing the region to be detected, enabling the camera shooting range of the camera to include the whole region to be detected, and then starting the initialization function of a scanning control unit.
And then, placing the handheld radar probe on the surface of the measured medium, starting a scanning process through a scanning control program, moving the handheld radar probe on the surface of the area to be detected in a motion mode similar to a blackboard erasing mode until the coverage of the scanning area is completed, and monitoring the current scanning process through the scanning control unit and the visual positioning unit in the scanning process.
And finally, after the handheld radar probe finishes scanning the whole detection area, sending detected echo data to a scanning control unit, sending positioning data recorded with the moving process of the radar probe to the scanning control unit by a visual positioning unit, calling an imaging processing unit through the scanning control unit, and processing the received echo data and the positioning data according to the array scanning holographic penetration imaging method to obtain final panoramic data of the target.
In this embodiment, an imaging experiment of a preset target inside a wood board according to the array scanning holographic penetration imaging method and the handheld holographic penetration imaging radar system is also provided. As shown in fig. 5, the moving track of the radar probe during scanning at the global position displayed by the coordinate axis is obtained after the positioning data is processed. As shown in fig. 6, a fused imaging result obtained after the positioning data and the detection data are processed according to the array scanning holographic penetration imaging method, compared with fig. 4, it can be proved that the size, the position, the shape, etc. of the target located inside the medium can be clearly detected according to the array scanning holographic penetration imaging method and the handheld holographic penetration imaging radar system proposed herein, and all the sizes, the positions, the shapes, etc. are consistent with the preset target, which illustrates the effectiveness of the method and the system.
In the array scanning holographic penetration imaging method and the handheld holographic penetration imaging radar system, the detection scanning speed can be greatly improved by the multi-transmitting and multi-receiving radar system, a complex electromechanical scanning device is omitted by a visual positioning scheme, the volume weight of the system is greatly reduced, the limitation of penetration imaging radar scanning on a regular scanning track is eliminated by the array scanning holographic penetration imaging method, and handheld free movement and rapid scanning are realized. Compared with the scheme of directly adopting a large array, the scheme of the system combining the multiple-transmitting and multiple-receiving array radar with the space scanning ensures the detection effect, obviously reduces the system scale and reduces the cost. The method can be applied to wall penetration imaging, detection of hidden objects, nondestructive detection of non-metallic materials and the like.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least some of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 7, there is provided a holographic transmission imaging apparatus comprising: a detection data obtaining module 200, a positioning data obtaining module 210, a position coordinate obtaining module 220, and an imaging result obtaining module 230, wherein:
a detection data acquisition module 200, configured to acquire detection data, where the detection data is echo data obtained by performing continuous multiple fast scanning while a radar probe with a multiple-transmit multiple-receive array antenna moves in a region to be detected on a medium surface;
a positioning data obtaining module 210, configured to obtain positioning data, where the positioning data are multiple position images that record position changes of the radar probe during multiple fast scans in the area to be detected, and each position image corresponds to a position change of the radar probe during one fast scan;
a position coordinate obtaining module 220, configured to process the positioning data to obtain a position coordinate of the radar probe in the region to be detected during each fast scanning;
and an imaging result obtaining module 230, configured to process the position coordinates of the radar probe during each fast scanning and the corresponding detection data to obtain an imaging result of the subsurface target of the region to be detected.
The specific definition of the holographic penetrating imaging device can be referred to the definition of the array scanning holographic penetrating imaging method in the above, and the detailed description is omitted here. The various modules in the holographic transmission imaging device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 8. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of array scanning holographic transfixion imaging. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the configuration shown in fig. 8 is a block diagram of only a portion of the configuration associated with the present application, and is not intended to limit the computing device to which the present application may be applied, and that a particular computing device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory having a computer program stored therein and a processor that when executing the computer program performs the steps of:
acquiring detection data, wherein the detection data are echo data obtained by carrying out continuous and multiple rapid scanning while a radar probe with a multi-transmitting and multi-receiving array antenna moves in a region to be detected on the surface of a medium;
acquiring positioning data, wherein the positioning data are a plurality of position images for recording position changes of a radar probe when the radar probe performs multiple rapid scans in the area to be detected, and each position image corresponds to the position change of the radar probe during one rapid scan;
processing the positioning data to obtain a position coordinate of the radar probe in the area to be detected during each quick scanning;
and processing according to the position coordinates of the radar probe during each quick scanning and the corresponding detection data to obtain the imaging result of the target under the surface layer of the area to be detected.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring detection data, wherein the detection data are echo data obtained by carrying out continuous and multiple rapid scanning while a radar probe with a multi-transmitting and multi-receiving array antenna moves in a region to be detected on the surface of a medium;
acquiring positioning data, wherein the positioning data are a plurality of position images for recording position changes of a radar probe when the radar probe performs multiple rapid scans in the area to be detected, and each position image corresponds to the position change of the radar probe during one rapid scan;
processing the positioning data to obtain a position coordinate of the radar probe in the area to be detected during each quick scanning;
and processing according to the position coordinates of the radar probe during each quick scanning and the corresponding detection data to obtain the imaging result of the target under the surface layer of the area to be detected.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by hardware instructions of a computer program, which may be stored in a non-volatile computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct Rambus Dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (8)
1. An array scanning holographic transfixion imaging method, comprising:
acquiring detection data, wherein the detection data are echo data obtained by carrying out continuous and multiple rapid scanning while a radar probe with multiple transmitting and multiple receiving array antennas moves in a to-be-detected area on the surface of a medium by hand, the radar probe is an integrated radar and comprises a microcontroller, a multi-channel radio frequency transceiver chip, a radio frequency switch, a communication element and multiple antennas, and the multiple antennas are integrated on a plane board to form an antenna array;
acquiring positioning data, wherein the positioning data are a plurality of position images for recording position changes of a radar probe when the radar probe performs multiple rapid scans in the area to be detected, and each position image corresponds to the position change of the radar probe during one rapid scan respectively;
the positioning data is processed to obtain the position coordinates of the radar probe in the area to be detected when the radar probe is rapidly scanned each time, and the method comprises the following steps: calibration code elements are arranged on the boundary of a region to be detected on the surface of the medium, the number of the calibration code elements is three, the calibration code elements are respectively arranged at the upper left corner, the lower left corner and the upper right corner of the boundary of the region to be detected, the actual position coordinates of each calibration code element are obtained, and a positioning code element is also arranged on one side, back to the surface of the medium, of the radar probe; carrying out contour detection and feature extraction on each position image, and identifying each calibration code element and the pixel coordinate of the center point of the positioning code element; constructing a coordinate transformation matrix according to the pixel coordinates and the actual position coordinates of each calibration code element in each position image; converting pixel coordinates of the center point of the positioning code element in each position image into position coordinates according to the coordinate transformation matrix, wherein the position coordinates are actual position coordinates in a region to be detected when the radar probe performs rapid scanning each time;
processing according to the position coordinates of the radar probe during each quick scanning and the corresponding detection data to obtain the imaging result of the subsurface target of the area to be detected, wherein the processing process comprises the following steps:
constructing an imaging space according to the region to be detected and a preset imaging depth;
calculating according to the imaging space and the position coordinates of the radar probe during each quick scanning to obtain a compensation phase;
and performing point-by-point compensation phase accumulation summation according to the detection data to obtain the imaging result.
2. The array scanning holographic penetration imaging method of claim 1, wherein when the radar probe of the multiple-input multiple-output array antenna performs fast scanning, one antenna is a transmitting antenna, the other antennas are receiving antennas, the other antenna is switched to be a transmitting antenna, the other antenna is a receiving antenna, and the fast scanning is recorded after repeating all the antennas.
3. The array scanning holographic penetration imaging method according to claim 2, wherein the radar probe reciprocates on the surface of the medium when performing fast scanning until the moving track covers the whole area to be detected.
4. A hand-held holographic transfixion imaging radar system, said system comprising: the system comprises a handheld radar probe, a visual positioning unit, an imaging processing unit and a scanning control unit;
the scanning control unit respectively sends detection instructions to the handheld radar probe and the visual positioning unit;
the handheld radar probe is a multi-transmitting multi-receiving antenna array radar, and is used for carrying out continuous and multi-time rapid scanning while moving in a to-be-detected area on the surface of a medium according to the detection instruction to obtain echo data and sending the echo data serving as detection data to the scanning control unit;
the visual positioning unit records a plurality of position images of position change of the radar probe during multiple rapid scanning in the area to be detected according to the detection instruction, each position image corresponds to the position change of the radar probe during one rapid scanning, and the position images are used as positioning data and sent to the scanning control unit;
the scanning control unit sends the received detection data and the positioning data to the imaging processing unit;
the imaging processing unit processes the detection data and the positioning data according to the array scanning holographic penetration imaging method of any one of claims 1 to 3 to obtain an imaging result of the subsurface target of the region to be detected, and sends the imaging result to the scanning control unit;
the scanning control unit receives and displays the imaging result.
5. The hand-held holographic penetration imaging radar system according to claim 4, wherein the hand-held radar probe is an integrated radar comprising a microcontroller, a multi-channel radio frequency transceiver chip, a radio frequency switch, a communication element, and a plurality of antennas integrated on a single planar board to form an antenna array.
6. The handheld holographic penetration imaging radar system of claim 4, wherein the visual localization unit comprises a camera, three scaling symbols, and one localization symbol;
the camera lens of the camera device is over against the area to be detected, and the camera shooting range of the camera device comprises all the area to be detected;
the three calibration code elements are respectively arranged at the upper left corner, the lower left corner and the upper right corner of the boundary of the area to be detected;
the positioning code element is arranged on one side, back to the medium surface, of the radar probe.
7. The handheld holographic penetration imaging radar system of claim 6, wherein the calibration symbols and the positioning symbols employ a pattern of optical markers including annular coded markers, cross-shaped coded markers, two-dimensional code markers.
8. The handheld holographic penetration imaging radar system of claim 4, wherein the scan control unit is a host computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211246412.4A CN115327541B (en) | 2022-10-12 | 2022-10-12 | Array scanning holographic penetration imaging method and handheld holographic penetration imaging radar system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211246412.4A CN115327541B (en) | 2022-10-12 | 2022-10-12 | Array scanning holographic penetration imaging method and handheld holographic penetration imaging radar system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115327541A CN115327541A (en) | 2022-11-11 |
| CN115327541B true CN115327541B (en) | 2023-03-14 |
Family
ID=83913954
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211246412.4A Active CN115327541B (en) | 2022-10-12 | 2022-10-12 | Array scanning holographic penetration imaging method and handheld holographic penetration imaging radar system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115327541B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101828092A (en) * | 2007-08-17 | 2010-09-08 | 瑞尼斯豪公司 | Non-contact probe |
| CN102879763A (en) * | 2012-09-11 | 2013-01-16 | 上海交通大学 | System and method for quickly positioning noise source through image identification and sound intensity scanning |
| CN109188431A (en) * | 2018-09-11 | 2019-01-11 | 合肥工业大学 | A kind of compressed sensing based holographic microwave fast imaging method |
| CN110632593A (en) * | 2015-12-25 | 2019-12-31 | 华讯方舟科技有限公司 | Human body security inspection system and method based on millimeter wave holographic three-dimensional imaging |
| CN110870792A (en) * | 2018-08-31 | 2020-03-10 | 通用电气公司 | System and method for ultrasound navigation |
| CN114002160A (en) * | 2021-12-30 | 2022-02-01 | 北京理工大学 | Terahertz frequency modulation continuous wave nondestructive testing imaging system and method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10726741B2 (en) * | 2004-11-30 | 2020-07-28 | The Regents Of The University Of California | System and method for converting handheld diagnostic ultrasound systems into ultrasound training systems |
| CN112155596B (en) * | 2020-10-10 | 2023-04-07 | 达闼机器人股份有限公司 | Ultrasonic diagnostic apparatus, method of generating ultrasonic image, and storage medium |
-
2022
- 2022-10-12 CN CN202211246412.4A patent/CN115327541B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101828092A (en) * | 2007-08-17 | 2010-09-08 | 瑞尼斯豪公司 | Non-contact probe |
| CN102879763A (en) * | 2012-09-11 | 2013-01-16 | 上海交通大学 | System and method for quickly positioning noise source through image identification and sound intensity scanning |
| CN110632593A (en) * | 2015-12-25 | 2019-12-31 | 华讯方舟科技有限公司 | Human body security inspection system and method based on millimeter wave holographic three-dimensional imaging |
| CN110870792A (en) * | 2018-08-31 | 2020-03-10 | 通用电气公司 | System and method for ultrasound navigation |
| CN109188431A (en) * | 2018-09-11 | 2019-01-11 | 合肥工业大学 | A kind of compressed sensing based holographic microwave fast imaging method |
| CN114002160A (en) * | 2021-12-30 | 2022-02-01 | 北京理工大学 | Terahertz frequency modulation continuous wave nondestructive testing imaging system and method |
Non-Patent Citations (2)
| Title |
|---|
| "全息穿透雷达非平整表面杂波抑制算法研究";陈诚等;《系统工程与电子技术》;20211215;正文第2-5页 * |
| "基于Kinect的机器人辅助超声扫描系统研究";孟勃等;《计算机工程与科学》;20160331;正文第2-4页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115327541A (en) | 2022-11-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11715228B2 (en) | Imaging systems and related methods including radar imaging with moving arrays or moving targets | |
| Oliveri et al. | Compressive sensing imaging of non-sparse 2D scatterers by a total-variation approach within the Born approximation | |
| Salucci et al. | GPR prospecting through an inverse-scattering frequency-hopping multifocusing approach | |
| Alvarez-Narciandi et al. | Towards turning smartphones into mmWave scanners | |
| CN103399315B (en) | High-resolution detecting and imaging method for real-aperture phased array radar | |
| Depatla et al. | Robotic through-wall imaging: Radio-frequency imaging possibilities with unmanned vehicles | |
| Li et al. | A high-resolution handheld millimeter-wave imaging system with phase error estimation and compensation | |
| CN113534140B (en) | Ground-penetrating radar three-dimensional imaging method based on wave field cross-correlation | |
| EP4273579B1 (en) | Systems and methods for non-intrusive material quality inspection using threedimensional monostatic radar based imaging | |
| Franceschini et al. | Inversion of phaseless total field data using a two-step strategy based on the iterative multiscaling approach | |
| CN113945914B (en) | Ship monitoring method, equipment, storage medium and water surface monitoring system | |
| Barzegar et al. | 3-D through-the-wall radar imaging using compressed sensing | |
| Zhang et al. | Unveiling three-dimensional sea surface signatures caused by internal solitary waves: insights from the surface water ocean topography mission | |
| CN115327541B (en) | Array scanning holographic penetration imaging method and handheld holographic penetration imaging radar system | |
| CN102798858B (en) | Holographic active microwave imaging method | |
| Fischer et al. | Detection of antipersonnel mines by using the factorization method on multistatic ground-penetrating radar measurements | |
| US11382604B2 (en) | Ultrasonic image system with synthesis of images of different ultrasonic waves received at respective positions of a probe | |
| Wu et al. | A simple strategy for moving target imaging via an experimental UWB through-wall radar | |
| CN108387895A (en) | A kind of hologram radar imaging method and system based on relatively prime receiving array | |
| Meschino et al. | A practical guide on using SPOT-GPR, a freeware tool implementing a SAP-DoA technique | |
| JP2010236872A (en) | Image processing apparatus, wireless communication apparatus, image processing method, and image processing program | |
| Feng et al. | WiFi-based imaging for GPR applications: Fundamental study and experimental results | |
| CN119535449B (en) | Target translational compensation and three-dimensional imaging method and system for close-range millimeter wave MIMO radar | |
| RU2483323C1 (en) | Method of forming high brightness and contrast location image and apparatus for realising said method | |
| Hafiz et al. | A Deep Approach for Reliable Subsurface Imaging in Software-Defined Ground Penetrating Radar |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |