TW202305405A - Method and system for detecting and analyzing objects - Google Patents

Abstract

A method for detecting objects and labeling the objects with distances in an image includes steps of: obtaining a thermal image from a thermal camera, an RGB image from an RGB camera, and radar information from an mmWave radar; adjusting the thermal image based on the RGB image to generate an adjusted thermal image, and generating a fused image based on the RGB image and the adjusted thermal image; generating a second fused image based on the fused image and the radar information; detecting objects in the images, and generating, based on the fused image, another fused image including bounding boxes marking the objects; and determining motion parameters of the objects.

Description

Method and system for detecting objects and marking distance

The present invention relates to a method for object distance detection, in particular to a method for detecting objects and marking distances by integrating visible light cameras, thermal cameras, millimeter-wave radars and artificial intelligence object recognition models.

When the visible light camera (RGB Camera) has good weather and bright light, the imaging effect of the objects within its shooting range is good, but when the light is dim, such as at night without a light source, the imaging effect is inversely proportional to the light intensity. In rainy, snowy, foggy and other bad weather or in environments with smoke and dust, it is easy to be covered and impenetrable, and the imaging effect is not good, which affects the recognition rate of objects in the image.

Thermal cameras (or infrared cameras, Thermal Cameras) have better imaging effects than visible light cameras in poor weather or dim light environments, but thermal cameras can only describe the appearance of objects, and cannot display the details of objects, such as The detailed features of the human face cannot be displayed, and when the temperature of the adjacent objects captured is similar, the thermal camera is likely to confuse the adjacent objects and affect the recognition rate of the objects in the image.

Millimeter wave radar (mmWaveRadar) has the ability to detect the distance, relative speed and angle of objects, and can penetrate many substances such as plastic, dry walls, cloth, etc., and is not affected by bad weather and light, and has all-weather working ability . However, due to its low resolution, it cannot display the facade of the object, so it can only detect whether the object exists, but cannot identify the object.

Therefore, if the advantages of the aforementioned three sensors can be integrated, an artificial intelligence (AI) model can be established through deep learning through a large amount of visible light images and thermal image data, so that the surrounding objects can be identified and marked. Detecting the distance of objects will be able to provide information about surrounding objects to people or computers around the clock to make judgments and take appropriate and necessary actions in a timely manner.

Therefore, the purpose of the present invention is to provide a method and system for detecting objects and marking the distance, which integrates three sensors of visible light camera, thermal camera and millimeter wave radar, and uses a large amount of visible light image and thermal image data The image object recognition model and millimeter-wave radar trained by deep learning can automatically and all-weather detect the surrounding objects and mark the distance of the objects.

Therefore, a method for detecting an object and marking a distance in the present invention is realized by a system for detecting an object and marking a distance. a visible light camera, a millimeter-wave radar fixed below the photographing position, and an image processing device, which is electrically connected to the thermal camera, the visible light camera and the millimeter-wave radar and controls the synchronous actions of the three; the method includes Follow the steps below.

(A) making the thermal camera, the visible light camera and the millimeter-wave radar act synchronously, causing the thermal camera to shoot a scene to obtain a thermal image, and causing the visible light camera to shoot the scene to obtain a visible light image, and A radar spatial information obtained by detecting the scene with the millimeter-wave radar, the radar spatial information includes a plurality of radar point coordinates of each object detected by the millimeter-wave radar in the scene; wherein the thermal camera and The visible light camera is adjacently fixed at a shooting position, and the millimeter-wave radar is fixed below the shooting position.

(B) A dual image fusion module of the image processing device corrects the thermal image according to the visible light image to generate a corrected thermal image, and superimposes the corrected thermal image and the visible light image to form a fusion image.

(C) The coordinate conversion module of the image processing device draws a radar point coordinate map showing the coordinates of the radar points according to the radar spatial information, and the coordinate conversion module draws the radar point coordinates according to a radar-to-camera projection matrix The radar point coordinates in the point coordinate map are projected onto the fused image to generate a fused image with radar information.

(D) An image object recognition module of the image processing device recognizes the object in the image according to the corrected thermal image and the visible light image, and selects the recognized object in the fused image to generate a fusion of presented object information image.

(E) An image integration module of the image processing device, based on the fused image with radar information and the fused image with the presented object information, excludes the fused image with radar information that does not correspond to the fused image with the presented object information The coordinates of the radar points in the framed object, and corresponding to the coordinates of the radar points in the framed object of the fused image that presents the object information, the radar that is closest to the millimeter-wave radar point coordinates, calculate an actual distance between the framed object of the fused image presenting object information and the millimeter-wave radar, and mark the actual distance at the framed object of the fused image presenting object information to generate And output a final image, the final image presents the framed object and the actual distance.

In some embodiments of the present invention, the coordinate conversion module needs to first obtain the projection matrix from the radar to the camera, and the method for obtaining the projection matrix from the radar to the camera includes: (C1) placing a calibration plate in a space , and let the visible light camera photograph the calibration plate to obtain a visible light image for calibration, and let the millimeter-wave radar detect the space to obtain radar spatial information for calibration, the radar spatial information for calibration includes multiple information of the calibration plate radar point coordinates; and (C2) a computer device draws a calibration radar point coordinate map showing the radar point coordinates of the calibration plate based on the calibration radar spatial information, and uses the principle of perspective transformation, according to the calibration with visible light The correlation between the calibration plate image in the image and the radar point coordinates of the calibration plate in the calibration radar point coordinate map is calculated to obtain the calibration plate image by projecting the radar point coordinates into the calibration visible light image A homography matrix on , and use the homography matrix as the projection matrix from the radar to the camera.

In some implementation aspects of the present invention, when the visible light camera uses a non-wide-angle lens, the dual image fusion module also obtains an internal parameter matrix for correcting the visible light camera, and the dual image fusion module corrects the internal parameter matrix according to the internal parameter matrix. visible light image to generate a corrected visible light image, and then superimpose the corrected thermal image and the corrected visible light image to form the fusion image; and the computer device also corrects the corrected visible light image according to the internal parameter matrix to generate a The corrected visible light image, and the homography matrix is calculated according to the correlation between the calibration plate image in the corrected visible light image and the radar point coordinates of the calibration plate in the calibration radar point coordinate map.

In some implementation aspects of the present invention, when the visible light camera uses a wide-angle lens, the dual-image fusion module also obtains a fisheye correction matrix for correcting the visible light camera, and the dual-image fusion module corrects according to the fisheye correction matrix The visible light image generates a corrected visible light image, and then the corrected thermal image is superimposed with the corrected visible light image to form the fusion image; and the computer device also corrects the corrected visible light image according to the fisheye correction matrix A corrected visible light image is generated, and the homography matrix is calculated according to the correlation between the calibration plate image in the corrected visible light image and the radar point coordinates of the calibration plate in the calibration radar point coordinate map.

The effect of the present invention is to generate the fused image by combining the advantages of the thermal camera, the visible light camera and the millimeter-wave radar, and identify objects based on the thermal image and the visible light image at the same time , and generate the fused image of the presenting object information, and use the radar space information generated by the millimeter-wave radar detection to make a radar point coordinate map, and project the radar point coordinate map to the fused image to generate the fused image with radar information , and calculate the actual distance between the object and the millimeter-wave radar by integrating the fused image presenting object information and the fused image with radar information, and generate the identified object and its relationship with the millimeter-wave radar The final image of the actual distance can be minimized, and the impact of the environment or weather can be minimized, and the detection information of surrounding objects can be provided to humans or computers for judgment, so that humans or computers can take appropriate and necessary actions in a timely manner. as.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are denoted by the same numerals.

Referring to FIG. 1, it is a main flowchart of an embodiment of the method for detecting objects and marking distances according to the present invention, and as shown in FIG. 2, this embodiment is mainly applied (but not limited) to a vehicle 100, and It is realized by a system that detects objects and marks the distance. The system mainly includes a thermal camera 1, a visible light camera 2, a millimeter wave radar 3 and an image processing device 4. The thermal camera 1 and the visible light camera 2 Adjacently fixed on a shooting position, such as the roof of the vehicle 100 and facing the front of the vehicle 100, the millimeter wave radar 3 is fixed below the shooting position, such as the front middle of the vehicle 100 near the license plate and facing In front of the vehicle 100 , the image processing device 4 is installed inside the vehicle 100 , and is electrically connected with the thermal camera 1 , the visible light camera 2 and the millimeter wave radar 3 to control the synchronous actions of the three. And as shown in Figure 3, this image processing device 4 can be but not limited to a microcomputer (or car computer) and includes a pair of image fusion modules 41, a coordinate conversion module 42, an image object recognition module 43 and An image integration module 44, and the above-mentioned module can be program software executed by such as the central processing unit and/or image processor in the image processing device 4, but it is not limited thereto, and the above-mentioned module can also be It is implemented by combining firmware or software with hardware.

Thus, when the vehicle 100 is driven on the road and the system for detecting objects and marking distances is activated, as shown in step S1 of FIG. 1 , the image processing device 4 makes the thermal camera 1, the visible light camera 2 and the mm The wave radar 3 operates synchronously, so that the thermal camera 1 shoots a scene in front of the vehicle 100 to obtain a thermal image 11 as shown in FIG. A visible light image 21, and at the same time make the millimeter-wave radar 3 detect the scene to obtain a radar space information, the radar space information includes a plurality of each object detected by the millimeter-wave radar 3 in the scene Radar point coordinates.

Next, as shown in step S2 of FIG. 1 , the image processing device 4 applies the dual image fusion module 41 therein to correct the thermal image 11 according to the visible light image 21 to generate a correction that can be well fused with the visible light image 21 and superimposing the corrected thermal image 11 ′ with the visible light image 21 to form a fused image 20 as shown in FIG. 4 . And the method for the dual-image fusion module 41 to generate the fused image 20 can refer to the dual-image fusion technology proposed by the same applicant in Taiwan Patent Application No. 110104936, so it will not be described in detail here. However, this application is not limited to only using the method of the above application to generate the fused image 20 . Moreover, the dual image fusion module 41 provides the fusion image 20 to the coordinate conversion module 42, and provides the corrected thermal image 11', the visible light image 21 and the fusion image 20 to the image object recognition module. 43.

In addition, as shown in step S3 of FIG. 1 and FIG. 3 , the image processing device 4 uses the coordinate conversion module 42 to acquire (extract) the millimeter-wave radar 3 from the radar spatial information to detect The radar point coordinates of each object obtained, and use a visual software package, such as but not limited to Rviz (Ros visualization) software, to generate two-dimensional radar point coordinates as shown in Figure 5 according to the radar point coordinates Fig. 51 (each point shown therein represents that there is an object there, and each point is actually a collection of coordinates of multiple radar points). Then, the coordinate conversion module 42 projects the radar point coordinates in the radar point coordinate map 51 to the fused image 20 according to a projection matrix from a radar (the millimeter wave radar 3 ) to a camera (the visible light camera 2 ). , and generate a fused image 61 with radar information as shown in FIG. 6 .

Specifically, the coordinate transformation module 42 needs to first obtain the projection matrix from the radar to the camera, and the method of obtaining the projection matrix from the radar to the camera is similar to the method of obtaining the homography matrix in the Taiwan No. 110104936 patent application, such as In step S71 of FIG. 7 , firstly, a calibration plate is placed in a space, and the visible light camera 2 is made to photograph the calibration plate to obtain a visible light image for calibration, and at the same time, the millimeter-wave radar 3 is made to detect the space to obtain a The radar spatial information for calibration, the radar spatial information for calibration includes a plurality of radar point coordinates corresponding to the calibration plate; then, as shown in step S72 of Figure 7, using a computer device to obtain the radars from the radar spatial information for calibration Point coordinates, and use such as the above-mentioned visualization software suite, Rviz (Ros visualization) software, according to the coordinates of the correction radar points obtained from the correction radar space information to generate a correction radar point similar to that shown in Figure 5 Coordinate map (all or most of the radar points in this map will be concentrated in the position corresponding to the correction plate image in the correction visible light image), and then, the computer device uses the principle of perspective transformation, according to the correction with the visible light image The correlation between the calibration plate image and the radar point coordinates in the calibration radar point coordinate map is calculated, and the radar point coordinates in the calibration radar point coordinate map are projected into the calibration visible light image A homography matrix on the correction plate image, and the homography matrix is used as the above-mentioned projection matrix from the radar to the camera.

In addition, if it is considered that the visible light camera 2 adopts a non-wide-angle lens (such as a general standard lens), the visible light camera 2 may have image distortion caused by factors such as the internal mechanism of the camera, the photosensitive element, and the lens. In this embodiment, It is also possible to find an internal parameter matrix of the visible light camera 2 to use the internal parameter matrix to correct the visible light image captured by the visible light camera 2 to correct the visible light camera 2, and because the internal parameter matrix of the visible light camera 2 is found It is already a known technology and is not the focus of the present invention, so it will not be described in detail here; therefore, in the above step S72, the computer device can also obtain in advance the internal parameter matrix for correcting the visible light camera 2, and then, according to the The internal parameter matrix calibrates the corrected visible light image to generate a re-corrected visible light image, and then calculates the homography matrix based on the re-corrected visible light image and the corrected radar point coordinate map.

Therefore, in the above-mentioned step S2, the dual-image fusion module 41 will firstly correct the visible light image 21 according to the internal parameter matrix to generate a corrected visible light image 21, and then combine the corrected thermal image 11' with the corrected visible light image The images 21 are superimposed to form the fused image 20 , and then the fused image 20 is provided to the coordinate conversion module 42 .

Or, when the visible light camera 2 adopts a wide-angle lens (such as a fisheye lens), the visible light camera 2 needs to perform fisheye correction through a fisheye correction matrix, so that the photographed images with fisheye effect (distortion) The visible light image is converted into a visible light image without the fisheye effect. At this time, in this embodiment, it is necessary to find a fisheye correction matrix of the visible light camera 2, so as to use the fisheye correction matrix to correct the visible light image of the visible light camera 2. The visible light image 21 taken, and because the technical means of obtaining the fisheye correction matrix is a known technology and not the focus of the present invention, so it will not be described in detail here; therefore, in the above step S72, when the computer device After the fisheye correction matrix is obtained, the fisheye correction matrix can be applied to correct the visible light image for correction. Similarly, in this embodiment, in the above step S2, the dual-image fusion module 41 will first correct the visible light image 21 according to the fisheye correction matrix to generate a corrected visible light image 21, and then the corrected visible light image 21 will be generated. The thermal image 11 ′ is superimposed with the corrected visible light image 21 to form the fusion image 20 , and then the fusion image 20 is provided to the coordinate transformation module 42 .

Furthermore, as shown in step S4 of FIG. 1 and FIG. 3 , the image processing device 4 uses an image object recognition module 43 to recognize objects in the image according to the corrected thermal image 11 ′ and the visible light image 21 , and Frame the identified object in the fused image 20 and mark its category, such as shown in FIG. , car (car, truck, locomotive, bus, etc.) , locomotives or people, objects of different colors will be framed and marked as such as bus, motor, people, and confidence index, etc.); thereby, a fusion image 81 presenting object information is generated, and the presenting object information is provided The fused image 81 is given to the image integration module 44. And the image object recognition module 43 can use the image generated by the training method of the image object recognition model proposed by the same applicant's Taiwan No. 110110527 patent application An object recognition model, but not limited thereto.

Then, as shown in step S5 of FIG. 1 and FIG. 3 , the image processing device 4 uses the image integration module 44 therein to exclude the fused image 61 having radar information and the fused image 81 presenting object information. The coordinates of the radar points in the fused image 61 with radar information do not correspond to the framed objects (i.e., the object frame 810) in the fused image 81 of the presenting object information, and from the coordinates corresponding to the objects falling in the presenting object information Among the radar point coordinates in the framed object (i.e. object frame 810) of the fused image 81, a radar point coordinate closest to the millimeter-wave radar 4 is found to represent an object coordinate of the framed object, and Calculate the actual distance between the framed object of the fused image 81 presenting object information and the millimeter wave radar 4 according to the object coordinates, more specifically, from the perspective of the image, any object is bounded by the object frame 810 After the frame is selected, the image integration module 44 can obtain the coordinates of the four corners of the object frame 810 through calculation, through the coordinates of the four corners and the radar points projected into the image in the previously obtained fusion image 61 with radar information coordinates, the image integration module 44 can determine which radar point coordinates fall in the object frame 810, and according to the characteristics of the millimeter-wave radar, all radar points will return relative distances, so the image integration module 44 The nearest radar point can then be selected as the distance between the millimeter-wave radar 4 and the object.

Then, the image integration module 44 marks the actual distance at the object framed by the object frame 810 of the fused image 81 presenting object information to generate and output a final image 91 as shown in FIG. 9 , the final image 91 will present the object framed by the object frame 810 and the actual distance to the millimeter wave radar 4 (or vehicle 100). For example, as shown in FIG. 9, the distance between the millimeter wave radar 4 and the vehicle directly ahead is 5.83 kilometers. ruler. In addition, the image integration module 44 can also calculate a relative velocity and angle between the framed object in the fused image 81 presenting object information and the millimeter-wave radar 4 according to the object coordinates. Specifically, millimeter-wave radar uses many technologies in the production process, the most well-known of which is the Doppler effect, which can calculate the distance of an object through the wavelength change caused by motion, that is, the relative speed and other information, and this is The characteristics of existing millimeter-wave radar products are not the technology developed by the present invention. In addition, regarding the acquisition of angles, the image integration module 44 can perform Kalman filter calculations through the past coordinates and current coordinate information of millimeter-wave radar points. And through this calculation, a relatively stable predicted coordinate can be predicted, and then the angle can be calculated through the coordinate difference between the predicted coordinate and the current coordinate.

It is worth mentioning that there is no priority in the execution of the above steps S3 and S4, that is, step S4 may be executed first, and then step S3 may be executed, or steps S3 and S4 may be executed simultaneously.

To sum up, the above embodiment generates the fused image 20 that fuses the thermal image and the visible light image by combining the advantages of the thermal camera 1, the visible light camera 2 and the millimeter-wave radar 3 sensors. The established image object recognition module (model) recognizes the object in the image based on the thermal image and the visible light image at the same time, and generates the fusion image 81 presenting the object information, and uses the radar space information generated by the millimeter wave radar 4 detection to make radar point coordinate map, and project the radar point coordinates in the radar point coordinate map to the fused image 20 to generate the fused image 61 with radar information, and according to the framed object in the fused image 81 presenting object information and the The corresponding relationship of the coordinates of the radar points in the fused image 61 with radar information is used to find out the coordinates of the object representing the framed object, and calculate the actual distance between the object and the millimeter wave radar 4 according to the coordinates of the object , so that the output final image 91 presents the object framed by the object frame 810 and the actual distance from the millimeter wave radar 4 (or vehicle 100 ), and this embodiment combines the thermal camera 1, the visible light camera 2 and the Millimeter wave radar 3 Three kinds of sensors detect objects and their distance, which can minimize the influence of the environment or weather, and can provide all-weather detection information of surrounding objects for people or computers (such as car computers) to judge , so that people or computers can take appropriate and necessary actions in a timely manner, and indeed achieve the effect and purpose of the present invention.

But what is described above is only an embodiment of the present invention, and should not limit the scope of the present invention. All simple equivalent changes and modifications made according to the patent scope of the present invention and the content of the patent specification are still within the scope of the present invention. Within the scope covered by the patent of the present invention.

100:vehicle 1: thermal camera 11: thermal image 11': Corrected thermal image 2: Visible light camera 20: Fusion Image 21:Visible light image 3: Millimeter wave radar 4: Image processing device 41: Dual image fusion module 42: Coordinate conversion module 43: Image object recognition module 44: Image integration module 51: Radar point coordinate map 61: Fused Imagery with Radar Information 81: Fusion image presenting object information 810: object frame 91: Final image S1~S5: steps S71, S72: steps

Other features and effects of the present invention will be clearly shown in the implementation manner with reference to the drawings, wherein: Fig. 1 is the main flowchart of an embodiment of the method for detecting objects and marking distances according to the present invention; Fig. 2 shows the application example of an embodiment of the system for detecting objects and marking the distance of the present invention and the hardware equipment mainly included in the system; FIG. 3 shows a schematic block diagram of modules mainly included in the image processing device of this embodiment; FIG. 4 shows a schematic diagram of the process of generating a fusion image in this embodiment; Fig. 5 shows the radar point coordinate diagram of the present embodiment; FIG. 6 shows the fused image with radar information of this embodiment; FIG. 7 shows the method steps of obtaining the projection matrix from the radar to the camera in this embodiment; FIG. 8 shows a fused image showing object information in this embodiment; and Figure 9 shows the final image of this example.

S1~S5: steps

Claims (8)

A method of detecting objects and marking distances, comprising: (A) making a thermal camera, a visible light camera and a millimeter-wave radar act synchronously, causing the thermal camera to shoot a scene to obtain a thermal image, and causing the visible light camera to shoot the scene to obtain a visible light image, and A radar spatial information obtained by detecting the scene with the millimeter-wave radar, the radar spatial information includes a plurality of radar point coordinates of each object detected by the millimeter-wave radar in the scene; wherein the thermal camera and The visible light camera is fixed adjacent to a shooting position, and the millimeter-wave radar is fixed below the shooting position; (B) A dual image fusion module of an image processing device corrects the thermal image according to the visible light image to generate a corrected thermal image, and superimposes the corrected thermal image and the visible light image to form a fusion image ; (C) The coordinate conversion module of the image processing device draws a radar point coordinate map showing the coordinates of the radar points according to the radar space information, and the coordinate conversion module uses a radar-to-camera projection matrix to draw the the radar point coordinates in the radar point coordinate map are projected onto the fused image to produce a fused image with radar information; (D) An image object recognition module of the image processing device recognizes the object in the image according to the corrected thermal image and the visible light image, and selects the recognized object in the fusion image to generate a display object information fused images; and (E) An image integration module of the image processing device excludes fusion images that do not correspond to the presentation object information in the fusion image with radar information according to the fusion image with radar information and the fusion image with the presentation object information The coordinates of the radar points in the framed object of the image, and the coordinates of the radar points in the framed object of the fused image that presents the information of the object corresponding to the coordinates of the radar point that is the closest to the millimeter wave radar Radar point coordinates, calculate an actual distance between the framed object of the fused image presenting object information and the millimeter-wave radar, and mark the actual distance at the framed object of the fused image presenting object information, to A final image showing the framed object and the actual distance is generated and output. The method for detecting objects and marking the distance as described in claim 1, in step (C), the coordinate conversion module needs to first obtain the projection matrix from the radar to the camera, and obtain the projection matrix from the radar to the camera include: (C1) placing a calibration plate in a space, and causing the visible light camera to photograph the calibration plate to obtain a visible light image for calibration, and causing the millimeter-wave radar to detect the space to obtain radar spatial information for calibration, the The radar spatial information for calibration includes coordinates of multiple radar points of the calibration plate; and (C2) A computer device draws a calibration radar point coordinate map showing the radar point coordinates of the calibration plate according to the calibration radar spatial information, and uses the principle of perspective transformation, according to the calibration plate in the calibration visible light image The correlation between the image and the radar point coordinates of the calibration plate in the calibration radar point coordinate map is calculated to obtain a homography that projects the radar point coordinates onto the calibration plate image in the calibration visible light image matrix, and use the homography matrix as the projection matrix from the radar to the camera. The method for detecting objects and marking the distance according to claim 2, wherein when the visible light camera uses a non-wide-angle lens, in step (B), the dual image fusion module also obtains an internal parameter matrix for correcting the visible light camera , and the dual image fusion module corrects the visible light image according to the internal parameter matrix to generate a corrected visible light image, and then superimposes the corrected thermal image and the corrected visible light image to form the fusion image; in step (C2 ), the computer device further calibrates the visible light image for correction according to the internal parameter matrix to generate a corrected visible light image, and according to the corrected The correlation of the radar point coordinates of the plate is calculated to obtain the homography matrix. The method for detecting objects and marking the distance according to claim 2, wherein when the visible light camera uses a wide-angle lens, in step (B), the dual image fusion module also obtains a fisheye correction matrix for correcting the visible light camera , and the dual image fusion module corrects the visible light image according to the fisheye correction matrix to generate a corrected visible light image, and then superimposes the corrected thermal image and the corrected visible light image to form the fusion image; in step ( In C2), the computer device further calibrates the visible light image for correction according to the fisheye correction matrix to generate a corrected visible light image, and according to the difference between the correction plate image in the corrected visible light image and the radar point coordinate map for correction The correlation of the coordinates of the radar points of the calibration plate is calculated to obtain the homography matrix. A system for detecting objects and marking distances, comprising: a thermal camera; a visible light camera fixed adjacent to the thermal camera at a photographing position; A millimeter-wave radar fixed below the photographing location; and An image processing device, which is electrically connected to the thermal camera, the visible light camera and the millimeter-wave radar and controls the synchronous action of the three, so that the thermal camera can shoot a scene to obtain a thermal image, and the visible light camera can shoot Obtaining a visible light image of the scene, and detecting radar spatial information obtained by detecting the scene with the millimeter-wave radar, the radar spatial information including a plurality of radars of each object detected by the millimeter-wave radar in the scene point coordinates; where A dual image fusion module of the image processing device corrects the thermal image according to the visible light image to generate a corrected thermal image, and superimposes the corrected thermal image and the visible light image to form a fusion image; The coordinate conversion module of the image processing device draws a radar point coordinate map showing the coordinates of the radar points according to the radar spatial information, and the coordinate conversion module draws the radar point coordinate map according to a radar-to-camera projection matrix The coordinates of the radar points in are projected onto the fused image to generate a fused image with radar information; An image object recognition module of the image processing device recognizes objects in the image according to the corrected thermal image and the visible light image, and selects the recognized object in the fusion image to generate a fusion image showing object information; An image integration module of the image processing device excludes frames in the fused image with radar information that do not correspond to the fused image with the presented object information according to the fused image with radar information and the fused image with the presented object information The coordinates of the radar points in the selected object, and the coordinates of the radar point closest to the millimeter wave radar corresponding to the coordinates of the radar points in the framed object of the fused image presenting the object information, calculating an actual distance between the framed object of the fused image presenting object information and the millimeter-wave radar, and marking the actual distance at the framed object of the fused image presenting object information, to generate and output a The final image, which presents the framed object and the actual distance. The system for detecting objects and marking distances as described in Claim 5, wherein the coordinate conversion module needs to first obtain the projection matrix from the radar to the camera, and the method of obtaining the projection matrix from the radar to the camera includes: placing a calibration plate in a space, and causing the visible light camera to photograph the calibration plate to obtain a visible light image for calibration, and causing the millimeter-wave radar to detect the space to obtain spatial information of a radar for calibration, the radar for calibration the spatial information includes the coordinates of a plurality of radar points for the calibration plate; and Draw a calibration radar point coordinate map showing the radar point coordinates of the calibration plate by a computer device according to the calibration radar spatial information, and use the principle of perspective transformation, according to the calibration plate image and the calibration plate image in the calibration visible light image The correlation of the radar point coordinates of the calibration plate in the calibration radar point coordinate map is calculated to obtain a homography matrix projecting the radar point coordinates onto the calibration plate image in the calibration visible light image, And use the homography matrix as the projection matrix from the radar to the camera. The system for detecting objects and marking distances as described in claim 6, wherein when the visible light camera uses a non-wide-angle lens, the dual image fusion module also obtains an internal parameter matrix for correcting the visible light camera, and the dual image fusion module The group corrects the visible light image according to the internal parameter matrix to generate a corrected visible light image, and then superimposes the corrected thermal image and the corrected visible light image to form the fusion image; and the computer device also corrects according to the internal parameter matrix The correction visible light image is used to generate a corrected visible light image, and according to the correlation between the calibration plate image in the corrected visible light image and the radar point coordinates of the calibration plate in the calibration radar point coordinate map, the calculation is obtained The homography matrix. The system for detecting objects and marking distances as described in claim 6, wherein when the visible light camera uses a wide-angle lens, the dual image fusion module also obtains a fisheye correction matrix for correcting the visible light camera, and the dual image fusion module The group corrects the visible light image according to the fisheye correction matrix to generate a corrected visible light image, and then superimposes the corrected thermal image and the corrected visible light image to form the fusion image; matrix correction of the calibration visible light image to generate a corrected visible light image, and according to the correlation between the calibration plate image in the corrected visible light image and the radar point coordinates of the calibration plate in the calibration radar point coordinate map, Calculate the homography matrix.

Images