CN108053449A - Three-dimensional rebuilding method, device and the binocular vision system of binocular vision system - Google Patents

Three-dimensional rebuilding method, device and the binocular vision system of binocular vision system Download PDF

Info

Publication number
CN108053449A
CN108053449A CN201711425236.XA CN201711425236A CN108053449A CN 108053449 A CN108053449 A CN 108053449A CN 201711425236 A CN201711425236 A CN 201711425236A CN 108053449 A CN108053449 A CN 108053449A
Authority
CN
China
Prior art keywords
vision system
dimensional reconstruction
images
binocular vision
target object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201711425236.XA
Other languages
Chinese (zh)
Inventor
左国玉
陈珂鑫
卢佳豪
潘婷婷
刘月雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing University of Technology
Original Assignee
Beijing University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing University of Technology filed Critical Beijing University of Technology
Priority to CN201711425236.XA priority Critical patent/CN108053449A/en
Publication of CN108053449A publication Critical patent/CN108053449A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本发明提供一种双目视觉系统的三维重建方法、装置及双目视觉系统，所述三维重建方法为：使用预先标定参数模型的双目摄像头获取两幅图像；基于所述两幅图像，使用YOLO目标检测算法获取目标物体图像的识别信息与检测窗口，对所述两幅图像的检测窗口作为特征点进行立体匹配，获得目标物体图像的空间离散点云；基于所述空间离散点云与所述参数模型，通过三角测绘的原理，获取实际空间中目标物体的位置坐标，以完成三维重建。本发明提供的双目视觉系统的三维重建方法，可以对目标物体进行识别，结合识别结果进行特征点匹配，完成立体视觉，从而为服务机器人完成抓取与操作任务提供必要参数。

The present invention provides a three-dimensional reconstruction method, device and binocular vision system of a binocular vision system. The three-dimensional reconstruction method is: using a binocular camera with a pre-calibrated parameter model to obtain two images; based on the two images, using The YOLO target detection algorithm obtains the identification information and detection window of the target object image, performs stereo matching on the detection windows of the two images as feature points, and obtains the spatially discrete point cloud of the target object image; based on the spatially discrete point cloud and the obtained The parametric model is used to obtain the position coordinates of the target object in the actual space through the principle of triangulation mapping, so as to complete the three-dimensional reconstruction. The three-dimensional reconstruction method of the binocular vision system provided by the present invention can identify the target object, perform feature point matching in combination with the identification results, and complete stereo vision, thereby providing necessary parameters for the service robot to complete the grasping and operation tasks.

Description

Three-dimensional reconstruction method and device of binocular vision system and binocular vision system
Technical Field
The invention relates to the technical field of artificial intelligence and stereoscopic vision, in particular to a three-dimensional reconstruction method and device of a binocular vision system and the binocular vision system.
Background
The object grabbing and operation are basic tasks required to be executed by the intelligent service robot, and the recognition and positioning of the target object through the vision system of the robot are the premise that the robot finishes the task of grabbing the target object. Robots cannot perceive complex environments as easily as humans.
The binocular stereoscopic vision is a typical human-like vision model, two vision sensors are adopted to carry out image acquisition, stereoscopic correction, stereoscopic matching and other work, rich environment information can be obtained, the perception capability of scene depth information is improved on the basis of traditional image acquisition, and the binocular stereoscopic vision is very suitable for constructing a vision system of an intelligent service robot.
In the current robot industry, the visual system technology of an industrial robot is mature, but the requirement on the working environment is harsh, the application requirement of the unstructured environment cannot be met, and the industrial robot is difficult to transplant and apply to an intelligent service robot. The vision function of the intelligent service robot is often single, and the intelligent service robot can only be used for collecting environmental information required by specific work.
Disclosure of Invention
In order to at least partially overcome the above problems in the prior art, the present invention provides a binocular vision system and a three-dimensional reconstruction method and apparatus thereof.
According to an aspect of the present invention, there is provided a three-dimensional reconstruction method of a binocular vision system, including: acquiring two images by using a binocular camera with a pre-calibrated parameter model; based on the two images, acquiring identification information and a detection window of a target object image by using a YOLO target detection algorithm, and performing stereo matching on the detection windows of the two images as feature points to obtain a space discrete point cloud of the target object image; and acquiring the position coordinates of the target object in the actual space based on the space discrete point cloud and the parameter model by a triangulation principle so as to complete three-dimensional reconstruction.
And carrying out parameter model calibration on the binocular camera by using an opencv-based camera plane calibration method.
Wherein the parametric model comprises an internal parameter, an external parameter, and a distortion parameter; the internal parameters are internal structure parameters of the binocular camera; the external parameters comprise a rotation matrix and a translation matrix of the binocular camera; the distortion parameters include radial distortion and tangential distortion.
Wherein, the neural network architecture model used in the YOLO target detection algorithm is Darknet 19; wherein, the Darknet19 comprises 19 connected layers and 5 maxporoling layers, and the conv layer comprises two Kernels of 1 × 1 and 3 × 3.
Wherein the training set used by the Darknet19 is the VOC2012 data set.
The obtaining of the identification information and the detection window of the target object image based on the two images by using a YOLO target detection algorithm specifically includes:
dividing the two images into grids of S, predicting B Bounding boxes and corresponding Confidence scales by each grid, and predicting C conditional class probability by the Bounding boxes;
the dimension of the detection information output by the YOLO target detection algorithm is S (B) 5+ C.
According to another aspect of the present invention, there is provided a three-dimensional reconstruction apparatus of a binocular vision system, including: the image acquisition module is used for acquiring two images by using a binocular camera with a pre-calibrated parameter model; the processing module is used for acquiring identification information and a detection window of a target object image by using a YOLO target detection algorithm based on the two images, and performing stereo matching on the detection windows of the two images as feature points to obtain a space discrete point cloud of the target object image; and the three-dimensional model generation module is used for acquiring the position coordinates of the target object in the actual space based on the space discrete point cloud and the parameter model by a triangulation principle so as to complete three-dimensional reconstruction.
Wherein, still include: and the parameter model calibration module is used for carrying out parameter model calibration on the binocular camera by using an opencv-based camera plane calibration method.
Wherein, still include: the target detection module is used for dividing the two images into grids of S, each grid predicts B Bounding boxes and corresponding Confidence Scores, and the Bounding boxes predict C conditional class probabilities; the dimension of the detection information output by the YOLO target detection algorithm is S (B) 5+ C.
According to still another aspect of the present invention, there is provided a binocular vision system including the above three-dimensional reconstruction apparatus of the robot binocular vision system.
In summary, the invention provides a binocular vision system and a three-dimensional reconstruction method and device thereof, wherein a binocular camera with a pre-calibrated parameter model is used for acquiring two images; based on the two images, acquiring identification information and a detection window of a target object image by using a YOLO target detection algorithm, and performing stereo matching on the detection windows of the two images as feature points to obtain a space discrete point cloud of the target object image; and acquiring the position coordinates of the target object in the actual space based on the space discrete point cloud and the parameter model by a triangulation principle so as to complete three-dimensional reconstruction. The three-dimensional reconstruction method of the binocular vision system can identify the target object, perform characteristic point matching by combining the identification result and finish stereoscopic vision, thereby providing necessary parameters for a service robot to finish grabbing and operating tasks.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a three-dimensional reconstruction method of a binocular vision system according to an embodiment of the present invention;
fig. 2 is a block diagram of a three-dimensional reconstruction apparatus of a binocular vision system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flowchart of a three-dimensional reconstruction method of a binocular vision system according to an embodiment of the present invention, as shown in fig. 1, including:
s1, acquiring two images by using a binocular camera with a pre-calibrated parameter model;
the acquisition of the stereo image pair uses a group of binocular vision cameras and adopts a parallel stereo vision model. The internal parameters of the two cameras are kept consistent and work independently, and the two cameras are set to be parallel structures when images are collected. And simultaneously shooting to obtain an image A and an image B, wherein the shooting pictures of the two cameras are positioned on the same horizontal line, so that the transformation model between the two images can be regarded as a translation model.
S2, based on the two images, obtaining identification information and a detection window of the target object image by using a YOLO target detection algorithm, and performing stereo matching on the detection windows of the two images as feature points to obtain a space discrete point cloud of the target object image;
and S3, acquiring the position coordinates of the target object in the actual space based on the space discrete point cloud and the parameter model through the principle of triangular mapping so as to complete three-dimensional reconstruction.
Specifically, the position coordinates of the target object in the actual space can be obtained based on the space discrete point cloud and the parameter model by the principle of triangulation to complete three-dimensional reconstruction.
In the embodiment of the invention, a binocular camera with a pre-calibrated parameter model is used for acquiring two images; based on the two images, acquiring identification information and a detection window of a target object image by using a YOLO target detection algorithm, and performing stereo matching on the detection windows of the two images as feature points to obtain a space discrete point cloud of the target object image; and acquiring the position coordinates of the target object in the actual space based on the space discrete point cloud and the parameter model by a triangulation principle so as to complete three-dimensional reconstruction. The three-dimensional reconstruction method of the binocular vision system provided by the embodiment of the invention can be used for identifying the target object, matching the characteristic points by combining the identification result and completing the stereoscopic vision, thereby providing necessary parameters for the service robot to complete the grabbing and operating tasks.
On the basis of the embodiment, parameter model calibration is carried out on the binocular camera based on an opencv camera plane calibration method.
In the embodiment of the invention, parameter model calibration is carried out on the binocular camera by using an opencv-based camera plane calibration method, and the relationship between the pixel position of the image acquired by the binocular camera and the position of the scene point is established.
On the basis of the above embodiment, the parametric model includes an internal parameter, an external parameter, and a distortion parameter; wherein,
the internal parameters are internal structural parameters of the binocular camera;
the external parameters comprise a rotation matrix and a translation matrix of the binocular camera;
the distortion parameters include radial distortion and tangential distortion.
Specifically, the internal parameters are basic parameters for imaging of the binocular camera, and represent internal structure parameters of the binocular camera; the external parameters comprise a rotation matrix and a translation matrix of the binocular double cameras and are used for determining the three-dimensional position and vector relation of a coordinate system of the binocular cameras relative to a world coordinate system.
On the basis of the above embodiment, the neural network architecture model used in the YOLO target detection algorithm is Darknet 19; wherein,
the Darknet19 includes 19 connected layers and 5 maxporoling layers, and the conv layer includes two Kernels of 1 × 1 and 3 × 3.
The Darknet19 network consisted of 19 confluent layers and 5 maxporoling layers, used a large number of 3 x 3 filters and doubled the number of channels after each pooling step, conv layers comprised of two Kernels 1 x 1 and 3 x 3, and used batch normalization to stabilize training, accelerate convergence, and regularize the model. The input of the network needs to cut the size of the input image to 416 × 416, and perform 32 times of dimensionality reduction sampling on the image, and output a feature map with the size of 13 × 13.
Based on the above examples, the training set used by the Darknet19 is the VOC2012 data set.
Specifically, each image in the VOC2012 data set has a corresponding annotation file, which gives a Bounding Box and a Class label where objects appear in the image, and multiple objects belonging to multiple categories appear in one picture.
On the basis of the above embodiment, the obtaining of the identification information and the detection window of the target object image based on the two images by using the YOLO target detection algorithm specifically includes:
dividing the two images into grids of S, predicting B Bounding boxes and corresponding Confidence scales by each grid, and predicting C conditional class probability by the Bounding boxes;
the dimension of the detection information output by the YOLO target detection algorithm is S (B) 5+ C.
Specifically, an input image is divided into grids of S, each grid predicts B Bounding boxes and Confidence scales of the Bounding boxes, C conditional classification probabilities are predicted by the Bounding boxes, information of each predicted target is 5-dimensional and comprises 4-dimensional coordinate information (center point coordinate + target length and width) and Confidence degree of the target, the final output dimension is S (B5 + C), and frame information and target categories required by target detection are regressed on each grid. In the embodiment, a YOLO target detection algorithm is used for respectively detecting input images of a binocular camera, and an identification window of a detected object, confidence of target classification and 4-dimensional coordinate information are returned.
Fig. 2 is a schematic structural diagram of a three-dimensional reconstruction apparatus of a binocular vision system according to an embodiment of the present invention, as shown in fig. 2, including: the system comprises an image acquisition module 101, a processing module 102 and a three-dimensional model generation module 103; wherein,
the image acquisition module 101 is configured to acquire two images by using a binocular camera with a pre-calibrated parameter model;
the acquisition of the stereo image pair uses a group of binocular vision cameras and adopts a parallel stereo vision model. The internal parameters of the two cameras are kept consistent and work independently, and the two cameras are set to be parallel structures when images are collected. And simultaneously shooting to obtain an image A and an image B, wherein the shooting pictures of the two cameras are positioned on the same horizontal line, so that the transformation model between the two images can be regarded as a translation model.
The processing module 102 is configured to obtain identification information and a detection window of a target object image based on the two images by using a YOLO target detection algorithm, perform stereo matching on the detection windows of the two images as feature points, and obtain a spatial discrete point cloud of the target object image;
preferably, the neural network architecture model used by the YOLO target detection algorithm is Darknet 19; wherein,
the Darknet19 includes 19 connected layers and 5 maxporoling layers, and the conv layer includes two Kernels of 1 × 1 and 3 × 3.
The Darknet19 network consisted of 19 confluent layers and 5 maxporoling layers, used a large number of 3 x 3 filters and doubled the number of channels after each pooling step, conv layers comprised of two Kernels 1 x 1 and 3 x 3, and used batch normalization to stabilize training, accelerate convergence, and regularize the model. The input of the network needs to cut the size of the input image to 416 × 416, and perform 32 times of dimensionality reduction sampling on the image, and output a feature map with the size of 13 × 13.
Preferably, the training set used by the Darknet19 is the VOC2012 data set.
Specifically, each image in the VOC2012 data set has a corresponding annotation file, which gives a Bounding Box and a Class label where objects appear in the image, and multiple objects belonging to multiple categories appear in one picture.
The three-dimensional model generation module 103 is configured to obtain a position coordinate of the target object in an actual space based on the spatial discrete point cloud and the parameter model according to a principle of triangulation, so as to complete three-dimensional reconstruction.
Specifically, the position coordinates of the target object in the actual space can be obtained based on the space discrete point cloud and the parameter model by the principle of triangulation to complete three-dimensional reconstruction.
In the embodiment of the invention, the image acquisition module is used for acquiring two images by using a binocular camera with a pre-calibrated parameter model; the processing module is used for acquiring identification information and a detection window of a target object image by using a YOLO target detection algorithm based on the two images, and performing stereo matching on the detection windows of the two images as feature points to acquire a space discrete point cloud of the target object image; and the three-dimensional model generation module is used for acquiring the position coordinates of the target object in the actual space based on the space discrete point cloud and the parameter model by a triangulation principle so as to complete three-dimensional reconstruction. The three-dimensional reconstruction device of the binocular vision system provided by the embodiment of the invention can identify the target object, and performs characteristic point matching by combining the identification result to complete stereoscopic vision, thereby providing necessary parameters for the service robot to complete the grabbing and operating tasks.
On the basis of the above embodiment, the method further includes: and the parameter model calibration module is used for carrying out parameter model calibration on the binocular camera by using an opencv-based camera plane calibration method.
In the embodiment of the invention, the parameter model calibration module is used for carrying out parameter model calibration on the binocular camera based on the opencv camera plane calibration method, and establishing the relationship between the pixel position of the image acquired by the binocular camera and the position of the scene point.
Wherein the parametric model comprises an internal parameter, an external parameter, and a distortion parameter; wherein,
the internal parameters are internal structural parameters of the binocular camera;
the external parameters comprise a rotation matrix and a translation matrix of the binocular camera;
the distortion parameters include radial distortion and tangential distortion.
Specifically, the internal parameters are basic parameters for imaging of the binocular camera, and represent internal structure parameters of the binocular camera; the external parameters comprise a rotation matrix and a translation matrix of the binocular double cameras and are used for determining the three-dimensional position and vector relation of a coordinate system of the binocular cameras relative to a world coordinate system.
On the basis of the above embodiment, the method further includes: the target detection module is used for dividing the two images into grids of S, each grid predicts B Bounding boxes and corresponding Confidence Scores, and the Bounding boxes predict C conditional class probabilities;
the dimension of the detection information output by the YOLO target detection algorithm is S (B) 5+ C.
Specifically, the target detection module is configured to divide an input image into grids of S × S, predict B Bounding boxes and Confidence Scores of the Bounding boxes for each grid, predict C conditional class probabilities by the Bounding boxes, predict 5-dimensional information of each predicted target, include 4-dimensional coordinate information (center coordinates + target length and width) and a Confidence of the target, finally output a dimension of S × S (B × 5+ C), and regress frame information and target classes required for target detection on each grid.
On the basis of the above embodiments, a binocular vision system, characterized in that the robot binocular vision system comprises the three-dimensional reconstruction apparatus of the robot binocular vision system of any one of claims 7 to 9.
According to the binocular vision system provided by the embodiment of the invention, the three-dimensional reconstruction device in the binocular vision system can identify the target object, and the characteristic point matching is carried out by combining the identification result, so that the stereoscopic vision is completed, and therefore, necessary parameters are provided for the service robot to complete the grabbing and operating tasks.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1.一种双目视觉系统的三维重建方法，其特征在于，包括：1. A three-dimensional reconstruction method of a binocular vision system, comprising: 使用预先标定参数模型的双目摄像头获取两幅图像；Acquire two images using a binocular camera with a pre-calibrated parameter model; 基于所述两幅图像，使用YOLO目标检测算法获取目标物体图像的识别信息与检测窗口，对所述两幅图像的检测窗口作为特征点进行立体匹配，获得目标物体图像的空间离散点云；Based on the two images, use the YOLO target detection algorithm to obtain the identification information and the detection window of the target object image, and perform stereo matching on the detection windows of the two images as feature points to obtain a spatially discrete point cloud of the target object image; 基于所述空间离散点云与所述参数模型，通过三角测绘的原理，获取实际空间中目标物体的位置坐标，以完成三维重建。Based on the spatially discrete point cloud and the parameter model, the position coordinates of the target object in the actual space are obtained through the principle of triangulation, so as to complete the three-dimensional reconstruction. 2.根据权利要求1所述的双目视觉系统的三维重建方法，其特征在于，基于opencv的摄像机平面标定法对所述双目摄像头进行参数模型标定。2. the three-dimensional reconstruction method of binocular vision system according to claim 1, is characterized in that, the camera plane calibration method based on opencv carries out parameter model calibration to described binocular camera. 3.根据权利要求1所述的双目视觉系统的三维重建方法，其特征在于，所述参数模型包括内部参数、外部参数以及畸变参数；其中，3. the three-dimensional reconstruction method of binocular vision system according to claim 1, is characterized in that, described parametric model comprises internal parameter, external parameter and distortion parameter; Wherein, 所述内部参数为双目摄像头的内部结构参数；Described internal parameter is the internal structure parameter of binocular camera; 所述外部参数包括双目摄像头的旋转矩阵和平移矩阵；The external parameters include a rotation matrix and a translation matrix of the binocular camera; 所述畸变参数包括径向畸变和切向畸变。The distortion parameters include radial distortion and tangential distortion. 4.根据权利要求1所述的双目视觉系统的三维重建方法，其特征在于，所述YOLO目标检测算法中使用的神经网络架构模型为Darknet19；其中，4. the three-dimensional reconstruction method of binocular vision system according to claim 1, is characterized in that, the neural network architecture model used in the described YOLO target detection algorithm is Darknet19; Wherein, 所述Darknet19包括19层convolutional layers和5层maxpooling layers，conv层包括1*1和3*3两种Kernel。The Darknet19 includes 19 layers of convolutional layers and 5 layers of maxpooling layers, and the conv layer includes 1*1 and 3*3 Kernels. 5.根据权利要求4所述的双目视觉系统的三维重建方法，其特征在于，所述Darknet19使用的训练集为VOC2012数据集。5. the three-dimensional reconstruction method of binocular vision system according to claim 4, is characterized in that, the training set that described Darknet19 uses is VOC2012 data set. 6.根据权利要求1所述的双目视觉系统的三维重建方法，其特征在于，所述基于所述两幅图像，使用YOLO目标检测算法获取目标物体图像的识别信息与检测窗口具体为：6. the three-dimensional reconstruction method of binocular vision system according to claim 1, is characterized in that, described based on described two images, use YOLO target detection algorithm to obtain the identification information and detection window of target object image specifically as: 将所述两幅图像分别划分为S*S的栅格，每一个栅格预测B个Bounding Box和与其相对应的Confidence Scores，Bounding Box预测C个conditional class probability；The two images are divided into S*S grids, each grid predicts B Bounding Boxes and Confidence Scores corresponding thereto, and the Bounding Box predicts C conditional class probability; 所述YOLO目标检测算法输出的检测信息的维度为S*S*(B*5+C)。The dimension of the detection information output by the YOLO target detection algorithm is S*S*(B*5+C). 7.一种双目视觉系统的三维重建装置，其特征在于，包括：7. A three-dimensional reconstruction device of a binocular vision system, characterized in that it comprises: 图像获取模块，用于使用预先标定参数模型的双目摄像头获取两幅图像；The image acquisition module is used to acquire two images using the binocular camera of the pre-calibrated parameter model; 处理模块，用于基于所述两幅图像，使用YOLO目标检测算法获取目标物体图像的识别信息与检测窗口，对所述两幅图像的检测窗口作为特征点进行立体匹配，获得目标物体图像的空间离散点云；The processing module is configured to use the YOLO target detection algorithm to obtain the identification information and the detection window of the target object image based on the two images, perform stereo matching on the detection windows of the two images as feature points, and obtain the space of the target object image discrete point cloud; 三维模型生成模块，用于基于所述空间离散点云与所述参数模型，通过三角测绘的原理，获取实际空间中目标物体的位置坐标，以完成三维重建。The three-dimensional model generation module is used to obtain the position coordinates of the target object in the actual space based on the spatially discrete point cloud and the parameter model through the principle of triangulation, so as to complete the three-dimensional reconstruction. 8.根据权利要求7所述的双目视觉系统的三维重建装置，其特征在于，还包括：8. The three-dimensional reconstruction device of binocular vision system according to claim 7, is characterized in that, also comprises: 参数模型标定模块，用于基于opencv的摄像机平面标定法对所述双目摄像头进行参数模型标定。The parameter model calibration module is used for performing parameter model calibration on the binocular camera based on the camera plane calibration method of opencv. 9.根据权利要求7所述的双目视觉系统的三维重建装置，其特征在于，还包括：9. The three-dimensional reconstruction device of binocular vision system according to claim 7, is characterized in that, also comprises: 目标检测模块，用于将所述两幅图像分别划分为S*S的栅格，每一个栅格预测B个Bounding Box和与其相对应的Confidence Scores，Bounding Box预测C个conditionalclass probability；The target detection module is used to divide the two images into S*S grids, each grid predicts B Bounding Boxes and corresponding Confidence Scores, and the Bounding Box predicts C conditionalclass probabilities; 所述YOLO目标检测算法输出的检测信息的维度为S*S*(B*5+C)。The dimension of the detection information output by the YOLO target detection algorithm is S*S*(B*5+C). 10.一种双目视觉系统，其特征在于，所述机器人双目视觉系统包括权利要求7至9任一项所述的机器人双目视觉系统的三维重建装置。10. A binocular vision system, characterized in that the robot binocular vision system comprises the three-dimensional reconstruction device of the robot binocular vision system according to any one of claims 7 to 9.
CN201711425236.XA 2017-12-25 2017-12-25 Three-dimensional rebuilding method, device and the binocular vision system of binocular vision system Pending CN108053449A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711425236.XA CN108053449A (en) 2017-12-25 2017-12-25 Three-dimensional rebuilding method, device and the binocular vision system of binocular vision system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711425236.XA CN108053449A (en) 2017-12-25 2017-12-25 Three-dimensional rebuilding method, device and the binocular vision system of binocular vision system

Publications (1)

Publication Number Publication Date
CN108053449A true CN108053449A (en) 2018-05-18

Family

ID=62131862

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711425236.XA Pending CN108053449A (en) 2017-12-25 2017-12-25 Three-dimensional rebuilding method, device and the binocular vision system of binocular vision system

Country Status (1)

Country Link
CN (1) CN108053449A (en)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108769644A (en) * 2018-06-06 2018-11-06 浙江大学 A kind of binocular animation style rendering intent based on deep learning
CN108982901A (en) * 2018-06-14 2018-12-11 哈尔滨工业大学 A kind of rotating speed measurement method of at the uniform velocity rotary body
CN109102547A (en) * 2018-07-20 2018-12-28 上海节卡机器人科技有限公司 Robot based on object identification deep learning model grabs position and orientation estimation method
CN109300154A (en) * 2018-11-27 2019-02-01 郑州云海信息技术有限公司 A binocular stereo-based ranging method and device
CN109377555A (en) * 2018-11-14 2019-02-22 江苏科技大学 Feature extraction and recognition method for 3D reconstruction of foreground field of view of autonomous underwater robot
CN109410316A (en) * 2018-09-21 2019-03-01 深圳前海达闼云端智能科技有限公司 Method, tracking, relevant apparatus and the storage medium of the three-dimensional reconstruction of object
CN109465809A (en) * 2018-12-17 2019-03-15 中北大学 An intelligent garbage sorting robot based on binocular stereo vision positioning and recognition
CN109548489A (en) * 2018-12-27 2019-04-02 海安智鹏自动化科技有限公司 A kind of apple picking robot based on yolo vision
CN109784333A (en) * 2019-01-22 2019-05-21 中国科学院自动化研究所 3D target detection method and system based on point cloud weighted channel features
CN110009698A (en) * 2019-05-15 2019-07-12 江苏弘冉智能科技有限公司 A kind of binocular vision system Intelligent Calibration device and scaling method
CN110012236A (en) * 2019-03-29 2019-07-12 联想(北京)有限公司 A kind of information processing method, device, equipment and computer storage medium
CN110340901A (en) * 2019-06-28 2019-10-18 深圳盈天下视觉科技有限公司 A kind of control method, control device and terminal device
CN110458947A (en) * 2019-08-09 2019-11-15 江苏濠汉信息技术有限公司 Method and system for three-dimensional reconstruction of sag of transmission conductor
WO2020029917A1 (en) * 2018-08-06 2020-02-13 北京旷视科技有限公司 Image processing method and apparatus, and image processing device
CN113256708A (en) * 2021-04-07 2021-08-13 深圳群宾精密工业有限公司 Calibration method for theoretical 3D space and actual robot space
CN113362248A (en) * 2021-06-23 2021-09-07 合肥朗云物联科技股份有限公司 Binocular detection system for detecting object strain
CN113379751A (en) * 2021-06-23 2021-09-10 苏州紫金港智能制造装备有限公司 Two-dimensional target detection-based most suitable workpiece capturing point cloud identification and segmentation method, device and system
CN113533145A (en) * 2021-07-09 2021-10-22 清华大学 Rockfill particle size identification and monitoring device and method based on camera set three-dimensional reconstruction
CN113763562A (en) * 2021-08-31 2021-12-07 哈尔滨工业大学（威海） Elevation feature detection and elevation feature processing method based on binocular vision
CN114018214A (en) * 2021-10-18 2022-02-08 武汉理工大学 Marker binocular sub-pixel distance measurement method based on hardware acceleration system
CN114581741A (en) * 2022-05-09 2022-06-03 广东电网有限责任公司佛山供电局 Circuit breaker testing robot wiring positioning method and device based on image enhancement
CN114842144A (en) * 2022-05-05 2022-08-02 上海交通大学 Binocular vision three-dimensional reconstruction method and system
CN114898317A (en) * 2022-05-18 2022-08-12 斯坦德机器人（深圳）有限公司 Fork-lift fork object detection positioning system and method
CN115115768A (en) * 2021-03-18 2022-09-27 上海宝信软件股份有限公司 Object coordinate recognition system, method, device and medium based on stereoscopic vision
CN115187730A (en) * 2022-07-19 2022-10-14 重庆交通大学 A structure strain measurement method, device, measurement terminal and storage medium
CN115442579A (en) * 2021-06-04 2022-12-06 同方威视技术股份有限公司 Robot, control device and method thereof, and vehicle chassis image generation system
CN115496853A (en) * 2021-06-18 2022-12-20 中移雄安信息通信科技有限公司 Assistive System for the Blind Based on 3D Reconstruction
CN115511809A (en) * 2022-09-16 2022-12-23 中国人民解放军空军军医大学 Brain vision detection method based on neural feedback for scientific research laboratory
CN115661337A (en) * 2022-09-23 2023-01-31 安徽南瑞继远电网技术有限公司 A 3D reconstruction method for substation operators based on binocular vision
CN116433639A (en) * 2023-04-19 2023-07-14 中国科学院上海微系统与信息技术研究所 A pantograph arc detection method and device based on a binocular vision system

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160328601A1 (en) * 2014-04-25 2016-11-10 Tencent Technology (Shenzhen) Company Limited Three-dimensional facial recognition method and system
CN106204731A (en) * 2016-07-18 2016-12-07 华南理工大学 A kind of multi-view angle three-dimensional method for reconstructing based on Binocular Stereo Vision System
CN106949836A (en) * 2017-05-25 2017-07-14 中国科学技术大学 A kind of stereoscopic vision camera homonymy target location caliberating device and method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160328601A1 (en) * 2014-04-25 2016-11-10 Tencent Technology (Shenzhen) Company Limited Three-dimensional facial recognition method and system
CN106204731A (en) * 2016-07-18 2016-12-07 华南理工大学 A kind of multi-view angle three-dimensional method for reconstructing based on Binocular Stereo Vision System
CN106949836A (en) * 2017-05-25 2017-07-14 中国科学技术大学 A kind of stereoscopic vision camera homonymy target location caliberating device and method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
丁明宇等: ""基于深度学习的图片中商品参数识别方法"", 《软件学报》 *
关卓威: ""基于临近空间平台的立体信息获取及可视化技术研究"", 《中国博士学位论文全文数据库 信息科技辑》 *
李传朋: ""基于机器视觉和深度学习的目标识别与抓取定位研究"", 《中国优秀硕士学位论文全文数据库 信息科技辑》 *

Cited By (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108769644A (en) * 2018-06-06 2018-11-06 浙江大学 A kind of binocular animation style rendering intent based on deep learning
CN108769644B (en) * 2018-06-06 2020-09-29 浙江大学 A stylized rendering method for binocular animation based on deep learning
CN108982901A (en) * 2018-06-14 2018-12-11 哈尔滨工业大学 A kind of rotating speed measurement method of at the uniform velocity rotary body
CN108982901B (en) * 2018-06-14 2020-06-09 哈尔滨工业大学 A method of measuring the rotational speed of a uniform rotating body
CN109102547A (en) * 2018-07-20 2018-12-28 上海节卡机器人科技有限公司 Robot based on object identification deep learning model grabs position and orientation estimation method
WO2020029917A1 (en) * 2018-08-06 2020-02-13 北京旷视科技有限公司 Image processing method and apparatus, and image processing device
US11461908B2 (en) 2018-08-06 2022-10-04 Beijing Kuangshi Technology Co., Ltd. Image processing method and apparatus, and image processing device using infrared binocular cameras to obtain three-dimensional data
CN109410316A (en) * 2018-09-21 2019-03-01 深圳前海达闼云端智能科技有限公司 Method, tracking, relevant apparatus and the storage medium of the three-dimensional reconstruction of object
CN109377555A (en) * 2018-11-14 2019-02-22 江苏科技大学 Feature extraction and recognition method for 3D reconstruction of foreground field of view of autonomous underwater robot
CN109377555B (en) * 2018-11-14 2023-07-25 江苏科技大学 Method for extracting and identifying three-dimensional reconstruction target features of foreground visual field of autonomous underwater robot
CN109300154A (en) * 2018-11-27 2019-02-01 郑州云海信息技术有限公司 A binocular stereo-based ranging method and device
CN109465809A (en) * 2018-12-17 2019-03-15 中北大学 An intelligent garbage sorting robot based on binocular stereo vision positioning and recognition
CN109548489A (en) * 2018-12-27 2019-04-02 海安智鹏自动化科技有限公司 A kind of apple picking robot based on yolo vision
CN109784333A (en) * 2019-01-22 2019-05-21 中国科学院自动化研究所 3D target detection method and system based on point cloud weighted channel features
CN109784333B (en) * 2019-01-22 2021-09-28 中国科学院自动化研究所 Three-dimensional target detection method and system based on point cloud weighted channel characteristics
CN110012236A (en) * 2019-03-29 2019-07-12 联想(北京)有限公司 A kind of information processing method, device, equipment and computer storage medium
CN110009698B (en) * 2019-05-15 2024-03-15 江苏弘冉智能科技有限公司 Intelligent calibration device and calibration method for binocular vision system
CN110009698A (en) * 2019-05-15 2019-07-12 江苏弘冉智能科技有限公司 A kind of binocular vision system Intelligent Calibration device and scaling method
CN110340901A (en) * 2019-06-28 2019-10-18 深圳盈天下视觉科技有限公司 A kind of control method, control device and terminal device
CN110340901B (en) * 2019-06-28 2022-09-27 深圳盈天下视觉科技有限公司 Control method, control device and terminal equipment
CN110458947A (en) * 2019-08-09 2019-11-15 江苏濠汉信息技术有限公司 Method and system for three-dimensional reconstruction of sag of transmission conductor
CN115115768A (en) * 2021-03-18 2022-09-27 上海宝信软件股份有限公司 Object coordinate recognition system, method, device and medium based on stereoscopic vision
CN113256708A (en) * 2021-04-07 2021-08-13 深圳群宾精密工业有限公司 Calibration method for theoretical 3D space and actual robot space
CN113256708B (en) * 2021-04-07 2022-09-20 深圳群宾精密工业有限公司 Calibration method for theoretical 3D space and actual robot space
CN115442579A (en) * 2021-06-04 2022-12-06 同方威视技术股份有限公司 Robot, control device and method thereof, and vehicle chassis image generation system
CN115496853A (en) * 2021-06-18 2022-12-20 中移雄安信息通信科技有限公司 Assistive System for the Blind Based on 3D Reconstruction
CN113362248A (en) * 2021-06-23 2021-09-07 合肥朗云物联科技股份有限公司 Binocular detection system for detecting object strain
CN113379751A (en) * 2021-06-23 2021-09-10 苏州紫金港智能制造装备有限公司 Two-dimensional target detection-based most suitable workpiece capturing point cloud identification and segmentation method, device and system
CN113533145A (en) * 2021-07-09 2021-10-22 清华大学 Rockfill particle size identification and monitoring device and method based on camera set three-dimensional reconstruction
WO2023280300A1 (en) * 2021-07-09 2023-01-12 清华大学 Rockfill particle size identification and monitoring apparatus and method based on three-dimensional reconstruction of camera group
CN113763562B (en) * 2021-08-31 2023-08-29 哈尔滨工业大学（威海） Facade Feature Detection and Facade Feature Processing Method Based on Binocular Vision
CN113763562A (en) * 2021-08-31 2021-12-07 哈尔滨工业大学（威海） Elevation feature detection and elevation feature processing method based on binocular vision
CN114018214A (en) * 2021-10-18 2022-02-08 武汉理工大学 Marker binocular sub-pixel distance measurement method based on hardware acceleration system
CN114842144A (en) * 2022-05-05 2022-08-02 上海交通大学 Binocular vision three-dimensional reconstruction method and system
CN114581741B (en) * 2022-05-09 2022-07-15 广东电网有限责任公司佛山供电局 Circuit breaker testing robot wiring positioning method and device based on image enhancement
CN114581741A (en) * 2022-05-09 2022-06-03 广东电网有限责任公司佛山供电局 Circuit breaker testing robot wiring positioning method and device based on image enhancement
CN114898317A (en) * 2022-05-18 2022-08-12 斯坦德机器人（深圳）有限公司 Fork-lift fork object detection positioning system and method
CN115187730A (en) * 2022-07-19 2022-10-14 重庆交通大学 A structure strain measurement method, device, measurement terminal and storage medium
CN115511809A (en) * 2022-09-16 2022-12-23 中国人民解放军空军军医大学 Brain vision detection method based on neural feedback for scientific research laboratory
CN115661337A (en) * 2022-09-23 2023-01-31 安徽南瑞继远电网技术有限公司 A 3D reconstruction method for substation operators based on binocular vision
CN116433639A (en) * 2023-04-19 2023-07-14 中国科学院上海微系统与信息技术研究所 A pantograph arc detection method and device based on a binocular vision system

Similar Documents

Publication Publication Date Title
CN108053449A (en) Three-dimensional rebuilding method, device and the binocular vision system of binocular vision system
CN112990310B (en) Artificial intelligence system and method serving electric robots
CN113524194B (en) Target grabbing method of robot vision grabbing system based on multi-mode feature deep learning
CN111179324B (en) Object pose estimation method based on fusion of color and depth information in six degrees of freedom
CN109255813B (en) Man-machine cooperation oriented hand-held object pose real-time detection method
CN105225269B (en) Object modelling system based on motion
CN105335696B (en) A kind of intelligence based on the identification of 3D abnormal gaits behavioral value is helped the elderly robot and implementation method
CN113450408A (en) Irregular object pose estimation method and device based on depth camera
CN115578460B (en) Robot grabbing method and system based on multi-mode feature extraction and dense prediction
CN113409384A (en) Pose estimation method and system of target object and robot
CN111080693A (en) Robot autonomous classification grabbing method based on YOLOv3
CN112950667A (en) Video annotation method, device, equipment and computer readable storage medium
CN110706267B (en) Mining process-based ore three-dimensional coordinate acquisition method and device
CN110276826A (en) A kind of construction method and system of electric network operation environmental map
CN110065068B (en) A method and device for robot assembly demonstration programming based on reverse engineering
CN113065546A (en) A target pose estimation method and system based on attention mechanism and Hough voting
CN108399643A (en) A kind of outer ginseng calibration system between laser radar and camera and method
CN111151463A (en) Mechanical arm sorting and grabbing system and method based on 3D vision
CN113327298A (en) Grabbing attitude estimation method based on image instance segmentation and point cloud PCA algorithm
CN113920191B (en) 6D data set construction method based on depth camera
CN114273826B (en) Automatic welding position identification method for large workpieces to be welded
CN112288815B (en) Target die position measurement method, system, storage medium and device
CN112200157A (en) Human body 3D posture recognition method and system for reducing image background interference
CN106970620A (en) A kind of robot control method based on monocular vision
CN113313824A (en) Three-dimensional semantic map construction method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20180518

RJ01 Rejection of invention patent application after publication