CN113033590B

CN113033590B - Image feature matching method, device, image processing equipment and storage medium

Info

Publication number: CN113033590B
Application number: CN201911357846.XA
Authority: CN
Inventors: 龙学雄; 易雨亭; 李建禹
Original assignee: Hangzhou Hikrobot Co Ltd
Current assignee: Hangzhou Hikrobot Co Ltd
Priority date: 2019-12-25
Filing date: 2019-12-25
Publication date: 2024-08-09
Anticipated expiration: 2039-12-25
Also published as: CN113033590A

Abstract

The application discloses an image feature matching method, an image feature matching device, image processing equipment and a storage medium, and belongs to the technical field of image processing. According to the application, most of the initial point pairs which are wrong in matching are screened out from a plurality of initial point pairs according to the rotation angle of the image acquisition equipment and the direction angle difference value between the direction angles of the two characteristic points included in each initial point pair, so that the characteristic point pairs used for representing the same object point are obtained. The method and the device utilize the principle that the direction angle difference value is approximately consistent with the rotation angle under the condition that the optical axis of the camera is approximately perpendicular to the observation plane, and can eliminate the error matching which is difficult to eliminate by other characteristic matching methods. In addition, the method for screening the characteristic point pairs is not influenced by the sparseness of the characteristic points in the image and the inner point rate, so that the robustness is high.

Description

Image feature matching method, device, image processing equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image feature matching method, an image feature matching device, an image processing apparatus, and a storage medium.

Background

With the development of image processing technology, feature matching is increasingly widely applied in the image processing technology, for example, in unmanned aerial vehicle image stitching and AGV (Automated Guided Vehicle ) map construction, and can be applied to process acquired images. Here, feature matching refers to matching feature points in a plurality of images to obtain feature point pairs representing the same object point.

In the related art, when feature matching is performed on a plurality of images, a violent matching method is adopted first, that is, feature point matching is performed according to similarity of descriptors of the plurality of images, so as to obtain a plurality of initial point pairs. And screening the plurality of initial point pairs to obtain characteristic point pairs, wherein the descriptors can be obtained by calculating according to symbols or operators for coding and describing the local information of the characteristic points. Common methods for screening the initial point pairs are two-way matching, GMS (Grid-based Motion Statistics ) method, RANSAC (Random Sample Consensus, random sample consensus) method, and the like.

When the two-way matching method is used, two violent matching needs to be performed, and the two violent matching is very time-consuming. When the GMS method is used, since the GMS method relies on the neighbor relation between a plurality of pairs of initial points to screen, it is difficult to use the method when the feature points are sparse. When the RANSAC method is used, since the method models by randomly selecting a small number of initial point pairs and iteratively adjusts the model by using other initial point pairs step by step to realize the screening of the initial point pairs, the iteration times are high and time is very consuming when the ratio of characteristic point pairs for representing the same object point included in the initial point pairs is low. Therefore, the image feature matching method commonly used in the related art is still time-consuming, and the robustness is still to be improved.

Disclosure of Invention

The application provides an image feature matching method, an image feature matching device, image processing equipment and a storage medium, which can solve the problems of time consumption and low robustness during feature matching in the related technology. The technical scheme is as follows:

in one aspect, there is provided an image feature matching method, the method comprising:

Acquiring a plurality of initial point pairs, wherein each initial point pair of the plurality of initial point pairs comprises two characteristic points, one characteristic point of the two characteristic points is a characteristic point in a first image, and the other characteristic point is a characteristic point in a second image;

Determining a rotation angle of an image acquisition device in a first period, wherein the first period is a period between the acquisition time of the first image and the acquisition time of the second image;

And determining the characteristic point pairs used for representing the same object point in the plurality of initial point pairs according to the rotation angle and the direction angle difference value between the direction angles of the two characteristic points included in each initial point pair.

Optionally, the acquiring a plurality of initial point pairs includes:

determining a descriptor of each feature point in the first image according to the main direction of each feature point in the first image;

Determining descriptors of each feature point in the second image according to the main direction of each feature point in the second image;

And determining the plurality of initial point pairs according to the descriptors of each characteristic point in the first image and the descriptors of each characteristic point in the second image.

Optionally, the determining the rotation angle of the image capturing device in the first period includes:

Receiving motion data acquired by the image acquisition equipment through a motion sensor included in the image acquisition equipment in the first period;

The rotation angle of the image acquisition device within the first period is estimated from the motion data.

The rotation angle of the image capturing apparatus in the first period is estimated from the direction angles of the two feature points included in each of the plurality of initial point pairs.

Optionally, the determining the feature point pairs of the plurality of initial point pairs for representing the same object point according to the rotation angle and the direction angle difference between the direction angles of the two feature points included in each of the initial point pairs includes:

The feature point pairs are determined from the plurality of initial point pairs based on at least one of a descriptor distance and a composite distance between two feature points included in each initial point pair, and a direction angle difference between the rotation angle and a direction angle of two feature points included in each initial point pair.

Optionally, the determining the feature point pairs from the plurality of initial point pairs according to at least one of a descriptor distance and a comprehensive distance between two feature points included in each initial point pair, and a direction angle difference between the rotation angle and a direction angle of two feature points included in each initial point pair includes:

determining a difference value between the direction angle difference value corresponding to each initial point pair and the rotation angle to obtain a rotation angle difference value corresponding to each initial point pair;

And acquiring initial point pairs with corresponding rotation angle difference values smaller than an angle threshold value from the plurality of initial point pairs, and taking the acquired initial point pairs as the plurality of first candidate point pairs.

Acquiring a first candidate point pair with a descriptor distance between two included feature points smaller than a descriptor distance threshold value from the plurality of first candidate point pairs, and taking the acquired first candidate point pair as a plurality of second candidate point pairs;

and acquiring a second candidate point pair with the comprehensive distance between the two included feature points smaller than a comprehensive distance threshold value from the plurality of second candidate point pairs, and taking the acquired candidate point pair as the feature point pair.

Optionally, the acquiring, from the plurality of second candidate point pairs, a second candidate point pair having a total distance between two feature points that is smaller than a total distance threshold, and before taking the acquired candidate point pair as the feature point pair, further includes:

acquiring a first weight and a second weight, wherein the first weight is a weight corresponding to a rotation angle difference value, and the second weight is a weight corresponding to a descriptor distance;

and determining the comprehensive distance between the two feature points included in the corresponding second candidate point pair according to the first weight, the rotation angle difference value corresponding to each second candidate point pair, the second weight and the descriptor distance between the two feature points included in the corresponding second candidate point pair.

In another aspect, there is provided an image feature matching apparatus, the apparatus comprising:

The device comprises an acquisition module, a first image acquisition module and a second image acquisition module, wherein the acquisition module is used for acquiring a plurality of initial point pairs, each initial point pair of the plurality of initial point pairs comprises two characteristic points, one characteristic point of the two characteristic points is a characteristic point in a first image, and the other characteristic point is a characteristic point in a second image;

a first determining module, configured to determine a rotation angle of an image capturing device within a first period, where the first period is a period between a capturing time of the first image and a capturing time of the second image;

And the second determining module is used for determining the characteristic point pairs which are used for representing the same object point in the plurality of initial point pairs according to the rotation angle and the direction angle difference value between the direction angles of the two characteristic points included in each initial point pair.

Optionally, the acquiring module includes:

a first determining unit, configured to determine a descriptor of each feature point in the first image according to a main direction of each feature point in the first image;

a second determining unit, configured to determine a descriptor of each feature point in the second image, where the main direction of each feature point in the second image is a main direction of each feature point;

And a third determining unit, configured to determine the plurality of initial point pairs according to the descriptor of each feature point in the first image and the descriptor of each feature point in the second image.

Optionally, the first determining module includes:

a receiving unit, configured to receive motion data acquired by the image acquisition apparatus through a motion sensor included in the image acquisition apparatus in the first period;

a first estimating unit configured to estimate the rotation angle of the image capturing device in the first period based on the motion data.

Optionally, the first determining module includes:

A second estimating unit configured to estimate the rotation angle of the image capturing apparatus within the first period based on the direction angles of the two feature points included in each of the plurality of initial point pairs.

Optionally, the second determining module includes:

A fourth determining unit configured to determine the feature point pairs from the plurality of initial point pairs based on at least one of a descriptor distance and a comprehensive distance between two feature points included in each of the initial point pairs, and a direction angle difference between the rotation angle and a direction angle of two feature points included in each of the initial point pairs.

Optionally, the fourth determining unit includes:

A first determining subunit, configured to determine a difference between the direction angle difference value corresponding to each initial point pair and the rotation angle, so as to obtain a rotation angle difference value corresponding to each initial point pair;

And the second determining subunit is used for acquiring initial point pairs, of which the corresponding rotation angle difference value is smaller than an angle threshold value, from the plurality of initial point pairs, and taking the acquired initial point pairs as the plurality of first candidate point pairs.

A third determining subunit, configured to obtain, from the plurality of first candidate point pairs, a first candidate point pair that includes two feature points, where a descriptor distance between the two feature points is smaller than a descriptor distance threshold, and use the obtained first candidate point pair as a plurality of second candidate point pairs;

and a fourth determination subunit, configured to acquire, from the plurality of second candidate point pairs, a second candidate point pair that includes two feature points whose integrated distance is smaller than an integrated distance threshold, and take the acquired second candidate point pair as the feature point pair.

Optionally, the fourth determining unit further includes:

the acquisition subunit is used for acquiring a first weight and a second weight, wherein the first weight is a weight corresponding to the rotation angle difference value, and the second weight is a weight corresponding to the descriptor distance;

and a fifth determining subunit, configured to determine, according to the first weight, the rotation angle difference value corresponding to each second candidate point pair, the second weight, and the descriptor distance between the two feature points included in the corresponding second candidate point pair, a comprehensive distance between the two feature points included in the corresponding second candidate point pair.

In another aspect, there is provided an image processing apparatus, the computer apparatus including a processor, a communication interface, a memory, and a communication bus, the processor, the communication interface, and the memory completing communication with each other through the communication bus, the memory storing a computer program, the processor being configured to execute the program stored on the memory to implement the steps of the above-described image feature matching method.

In another aspect, a computer readable storage medium is provided, in which a computer program is stored, which when executed by a processor, implements the steps of the image feature matching method described above.

In another aspect, a computer program product is provided comprising instructions which, when run on a computer, cause the computer to perform the steps of the image feature matching method described above.

The technical scheme provided by the application has at least the following beneficial effects:

In the present application, most of the pairs of mismatching initial points can be screened out from the pairs of initial points based on the rotation angle of the image capturing apparatus and the direction angle difference between the direction angles of the two feature points included in each pair of initial points, to obtain the pair of feature points for representing the same object point. The method and the device utilize the principle that the direction angle difference value is approximately consistent with the rotation angle under the condition that the optical axis of the camera is approximately perpendicular to the observation plane, and can eliminate the error matching which is difficult to eliminate by other characteristic matching methods. In addition, the method for screening the characteristic point pairs is not influenced by the sparseness of the characteristic points in the image and the inner point rate, so that the robustness is high.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of determining a direction angle of a feature point a according to an embodiment of the present application;

fig. 2 is a schematic diagram of determining a direction angle of a feature point B according to an embodiment of the present application;

FIG. 3 is a schematic diagram of an implementation environment of an image feature matching method according to an embodiment of the present application;

FIG. 4 is a flowchart of an image feature matching method according to an embodiment of the present application;

FIG. 5 is a flowchart of another image feature matching method provided by an embodiment of the present application;

fig. 6 is a schematic structural diagram of an image feature matching device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.

First, some terms involved in the embodiments of the present application are explained to facilitate understanding.

Observation plane: a plane perpendicular to an optical axis of a camera mounted on the image pickup apparatus. For example, when the image capture device is an unmanned aerial vehicle, the observation plane may be a plane parallel to the lower bottom surface of the unmanned aerial vehicle, and when the image capture device is an AGV, the observation plane may be a plane parallel to or coincident with the bearing surface of the AGV.

Rotation angle: the image acquisition device moves through an angle of rotation in a plane parallel to the viewing plane.

Characteristic points: points on the image that have significant features within a local area.

Feature matching: and matching the characteristic points on different images to obtain characteristic point pairs for representing the same object point.

Initial point pair: and carrying out preliminary matching on the characteristic points on different images to obtain matching point pairs.

Characteristic point pairs: and screening the initial point pairs to obtain matching point pairs which can be used for representing the same physical point, namely characteristic point pairs.

Interior point rate: the feature point pairs obtained through feature matching account for the ratio of the initial point pairs.

Main direction: the feature points can have rotational invariance in the stable direction and the main direction of the local area where the feature points are located.

Direction angle: the included angle between the main direction and the reference direction is the direction angle of the feature point. For example, referring to fig. 1 and 2,u, the u-axis positive direction may be taken as the reference direction, with the v-axis being the two coordinate axes of the pixel coordinate system of the image. Fig. 1 shows an acquired first image, wherein a feature point a is a feature point on the first image, and the main direction of the feature point a is assumed to be the directionThe direction angle of the feature point a is the directionThe angle with the positive direction of the u-axis, i.e. theta _A. FIG. 2 shows a second image acquired, wherein feature point B on the second image and feature point A shown in FIG. 1 may represent the same physical point, assuming that the main direction of feature point B is the directionThe direction angle of the feature point B is the directionThe angle with the positive direction of the u-axis, i.e. theta _B.

Before explaining the image feature matching method provided by the embodiment of the application in detail, an application scene and an implementation environment provided by the embodiment of the application are introduced.

With the development of image processing technology, feature matching is increasingly widely applied in the image processing technology, for example, in unmanned aerial vehicle image stitching and AGV map construction, and can be applied to process images acquired by image acquisition equipment.

For example, when the unmanned aerial vehicle is used to scan the ground to obtain a ground image, the unmanned aerial vehicle generally moves on a plane parallel to the ground, that is, the optical axis of the camera mounted on the unmanned aerial vehicle may be always perpendicular to the ground. In this case, the angle rotated on the plane during the process of the unmanned aerial vehicle collecting two images may be referred to as a rotation angle. The direction angle difference between the direction angles of the two feature points included in the pair of feature points corresponding to the two images will be approximately equal to the rotation angle, that is, the angle by which the direction angles of the two feature points rotate is approximately equal to the rotation angle of the image capturing apparatus, and in other words, the amount of change in the direction angle rotation of the two feature points is approximately equal to the amount of change in the rotation direction of the image capturing apparatus. In this case, the image processing apparatus may screen the matched feature point pairs according to the image feature matching method provided by the embodiment of the present application, and after obtaining the feature point pairs that may be used to represent the same object point, may perform image stitching or other processing on the acquired images.

For another example, when an image is acquired with the AGV looking down or up, the optical axis of the camera mounted on the AGV is also typically perpendicular to the plane of view. In this case, the rotation angle generated by the movement of the AGV is approximately equal to the direction angle difference between the direction angles of the two feature points included in the pair of feature points. In this case, the image processing apparatus may also screen the matched feature point pairs according to the image feature matching method provided by the embodiment of the present application, and after obtaining the feature point pairs that can be used to represent the same object point, image stitching may be performed on the acquired images to construct a map or the like.

As is clear from the foregoing description, in the case where the optical axis of the camera mounted on the image pickup apparatus is perpendicular to the observation plane, the change in the direction angle of the feature point is approximately coincident with the movement of the image pickup apparatus. The principle thereof will be explained next.

The direction vector of the principal direction of a feature point on the image can be represented by the pixel coordinates of two pixel points, and the two pixel points have corresponding world coordinates in the three-dimensional space, and the projection relationship is as follows:

Wherein K is an internal reference of the image capturing device, X ₁ and X ₂ respectively represent world coordinates of two pixel points corresponding to a main direction of a feature point on the first image, [ u ₁,v₁, 1] and [ u ₂,v₂, 1] respectively represent normalized pixel coordinates of the two pixel points, z ₁ and z ₂ respectively represent coordinates of the two pixel points on a z-axis of a camera coordinate system, and vector vec ₁ represents a direction vector of the main direction.

After the image acquisition device moves, a new projection relationship is generated:

Wherein R and T are external parameters of the image acquisition device, R is a rotation matrix, T is a translation matrix, RX ₁ +T and RX ₂ +T respectively represent world coordinates of the two pixel points after the image acquisition device moves, [ u ₁',v₁ ',1] and [ u ₂',v₂', 1] respectively represent pixel coordinates of the two pixel points after the image acquisition device rotates, and vector vec ₂ represents a direction vector of a main direction after the image acquisition device rotates.

Under the condition that the optical axis of a camera mounted on the image acquisition equipment is perpendicular to the observation plane, z ₁＝z₂＝z′₁＝z′₂ is set to beAt this time, the direction vector represented by the formula (3) may be converted into the formula (7), and the direction vector represented by the formula (6) may be converted into the formula (8):

equation (9) can be derived from equations (7) and (8):

Wherein, Θ represents a rotation angle of the image capturing apparatus, and θ is 0.ltoreq.θ <360 °.

When the image capturing device rotates about the z-axis, that is, the image capturing device rotates about the optical axis of the camera, and the aspect ratio of the image captured by the image capturing device is identical to that of the object, f _x in the internal reference K of the image capturing device is identical to f _y, and c _x and c _y are 0, in this case, the deduction result as shown in formula (10) can be obtained.

In equation (10), vector vec ₁ represents the direction vector of the principal direction of a feature point on the first image, vector vec ₂ represents the direction vector of the principal direction of the feature point on the second image,The rotational change of the direction vector of the main direction may be represented, where θ represents the rotational angle of the image acquisition device. From equation (10), it can be seen that the rotation amount of the direction vector of the main direction approximately coincides with the rotation angle of the image capturing apparatus. Since the direction angle is determined by the angle between the main direction and the reference direction, the rotation of the main direction can represent the change of the direction angle, so that the change of the direction angle of the feature point can be obtained to be approximately equal to the rotation angle of the image acquisition device.

When the angle by which the image capturing apparatus is rotated exceeds 360 degrees, the rotation angle of the image capturing apparatus is the remainder obtained by dividing the angle of actual rotation by 360 degrees.

It should be noted that the above is only two possible application scenarios provided in the embodiment of the present application. For scenes with the optical axes of other cameras perpendicular to the observation plane, the image feature matching method provided by the embodiment of the application can be adopted. In addition, in the application scenario described above, the optical axis of the camera in the image capturing device is perpendicular to the observation plane, in the embodiment of the present application, the optical axis of the camera may not be strictly perpendicular to the observation plane, that is, some deviation is allowed, and when the deviation is within a certain range, the feature point pair may also be screened according to the image matching method provided by the embodiment of the present application.

Referring to fig. 3, fig. 3 is a schematic diagram illustrating an implementation environment according to an example embodiment. The implementation environment includes an image processing apparatus 101 and an image capturing apparatus 102, and the image processing apparatus 101 may be communicatively connected to the image capturing apparatus 102. The communication connection may be a wired or wireless connection, which is not limited by the present application.

The image processing device 101 may be a notebook computer, a desktop computer, a tablet computer, a mobile phone, a wearable device, an intelligent vehicle, an intelligent television, or the like, or may be a server, or a server cluster formed by a plurality of servers, or may be a cloud computing service center, which is not limited in the embodiment of the present application. The image capturing device 102 may be any device having an image capturing function, such as an unmanned aerial vehicle, an AGV, etc. having an image capturing function, and the image capturing device 102 may include a motion sensor, etc. that may capture motion data, such as a motion mileage, a motion acceleration, a motion angular velocity, a gesture, etc., while the image capturing device 102 is in motion.

The image capturing device 102 is configured to capture an image and send the captured image to the image processing device 101, and the image capturing device 102 may also be configured to capture motion data of itself and send the captured motion data to the image processing device 101. The image processing apparatus 101 is configured to receive the image and the motion data acquired by the image acquisition apparatus 102, and process the received image according to the image feature matching method provided by the embodiment of the present application.

The image processing apparatus 101 and the image capturing apparatus 102 may be two apparatuses or one apparatus. For example, it is assumed that the image capturing device 102 is an unmanned aerial vehicle that can process the captured image, that is, the unmanned aerial vehicle may also be used as the image processing device 101, in which case the image processing device 101 and the image capturing device 102 are one device. Assuming that the image capturing device 102 is an unmanned aerial vehicle, the unmanned aerial vehicle transmits the captured image to a server that can process the received image, that is, the server is the image processing device 101, in this case, the image processing device 101 and the image capturing device 102 are two devices.

It will be appreciated by those skilled in the art that the above-described image processing apparatus and image capturing apparatus are merely examples, and that other image processing apparatus or image capturing apparatus, now known or later developed, may be adapted for use with the present application and are intended to be within the scope of the present application and are incorporated herein by reference.

The image feature matching method provided by the embodiment of the application is explained in detail below.

Fig. 4 is a flowchart of an image feature matching method according to an embodiment of the present application, which can be applied to an image processing apparatus. Referring to fig. 4, the method includes the following steps.

Step 401: a plurality of initial point pairs are acquired, each of the plurality of initial point pairs including two feature points, one of the two feature points being a feature point in the first image and the other being a feature point in the second image.

In the embodiment of the application, the image acquisition device can acquire images, the image processing device can receive the first image and the second image acquired by the image acquisition device, acquire the characteristic points in the two images and determine a plurality of initial point pairs matched in the two images.

In the embodiment of the present application, there are two cases of a method of determining a plurality of pairs of initial points, and the two cases will be described below.

In the first case, the image capturing apparatus may include a motion sensor that may capture motion data of the image capturing apparatus at the time of capturing an image, and the motion data may include a motion mileage, a motion acceleration, a motion angular velocity, a posture, and the like. In this case, in the process of capturing an image, after the image capturing device captures an image, the position of each pixel in the next image can be estimated according to the position of each pixel in the image and the motion acceleration, the motion angular velocity, the gesture and the like in the motion data, that is, the pixel that can represent a physical point on the matching image can be tracked. Based on this, the image capturing apparatus may first capture a first image, and after capturing a second image, send the first image and the second image to the image processing apparatus. The image processing apparatus may estimate positions of respective pixels in the second image based on the motion data and positions of the respective pixels in the first image, determine pixels to which the respective pixels are matched in the second image based on the estimated positions, and use each pixel in the first image and the pixel to which the corresponding pixel is matched in the second image as an initial point pair, thereby obtaining a plurality of initial point pairs. The motion sensor may be an odometer, a GPS locator, or the like.

In the second case, after the image capturing apparatus transmits the captured first image and second image to the image processing apparatus, the image processing apparatus may extract feature points on the first image and second image by calculating descriptors of the pixel points. Then, a plurality of initial point pairs are determined according to the descriptor distance between each feature point on the first image and each feature point on the second image.

In the embodiment of the application, the image processing device can determine the descriptors of each feature point in the first image according to the main direction of each feature point in the first image, determine the descriptors of each feature point in the second image according to the main direction of each feature point in the second image, and determine a plurality of initial point pairs according to the descriptors of each feature point in the first image and the descriptors of each feature point in the second image.

The image processing apparatus may first perform feature extraction on each pixel point in the first image and the second image, and determine the pixel point having the significant feature as the feature point on the corresponding image. For example, a pixel point having a significant feature on a contour or an edge is used as a feature point. The image processing device may then determine a main direction of each feature point on the first image and the second image, which main direction may provide the feature with rotational invariance, and may calculate a descriptor of each feature point on the corresponding image from the main directions of each feature point on the first image and the second image, which descriptor may be used to represent the feature of the feature point.

The method for determining the main direction of the feature point may be to calculate the gradient direction and the gradient amplitude of the gray value of the neighborhood pixel point of the feature point, obtain a gradient direction histogram according to the gradient direction and the gradient amplitude statistics, and determine the main direction of the feature point according to the gradient direction histogram. The neighborhood pixel point may be a pixel point in a circular area or a rectangular area with the feature point as a center.

After determining the main direction of the feature points, the reference direction of the image may be rotated to the main direction to calculate descriptors of the corresponding feature points, so that the features represented by the descriptors may have rotational invariance. For example, the u-axis of the pixel coordinate system is rotated to coincide with the main direction to calculate the descriptors of the corresponding feature points.

In the embodiment of the present application, the descriptor may be calculated by an operator such as SIFT (SCALE INVARIANT Feature Transform ), SURF (Speeded Up Robust Features, acceleration robust feature), or ORB, or may be calculated by another operator, which is not limited in the embodiment of the present application.

After determining the descriptors of each feature point on the first image and in the second image, the image processing apparatus may acquire, for each feature point in the first image, a matching feature point having the smallest distance from the descriptor of the corresponding feature point from among a plurality of feature points included in the second image, based on the descriptors of the corresponding feature point and the descriptors of each feature point in the second image, and use the matching feature point of each feature point in the first image and the acquired corresponding feature point as an initial point pair to obtain a plurality of initial point pairs.

In calculating the descriptor distance, a euclidean distance, a hamming distance, or the like may be used. For example, SIFT and SURF operators may use euclidean distances and ORB operators may use hamming distances.

For example, assuming that an operator used for calculating a descriptor is SIFT, a distance between descriptors is calculated, a distance threshold is dist _th, two feature points included in an initial point pair are a feature point B and a feature point C, a descriptor of the feature point B is x ₁,y₁, a descriptor of the feature point C is x ₂,y₂, and a distance between descriptors of the two feature points is

Step 402: a rotation angle of the image capturing device within a first period of time, the first period of time being a period of time between a capturing moment of the first image and a capturing moment of the second image, is determined.

As can be seen from the foregoing description of the application scenario, in the embodiment of the present application, the optical axis of the camera mounted on the image capturing device may be perpendicular to the observation plane, for example, the unmanned aerial vehicle performs ground scanning jigsaw, and the AGV performs downward looking or upward looking positioning navigation, where the movement of the image capturing device in the first period means that the image capturing device rotates on a plane parallel to the observation plane. I.e. from the moment the first image is acquired to the moment the second image is acquired, the image acquisition device rotates in a plane parallel to the observation plane. Based on this, the image processing apparatus can determine the rotation angle of the image capturing apparatus within the first period, that is, can determine the amount of change in the rotation direction of the image capturing apparatus within the first period.

From the foregoing, it is known that the image capturing apparatus may include a motion sensor or may not include a motion sensor, and the motion sensor may capture motion data such as motion acceleration, motion angular velocity, and posture. Based on this, the image processing apparatus determines that the rotation angle of the image capturing apparatus within the first period can be equally two cases, which will be described next.

In the first case, the image pickup apparatus may receive motion data picked up by the image pickup apparatus through a motion sensor included in the image pickup apparatus during a first period, and then may estimate a rotation angle of the image pickup apparatus during the first period based on the motion data.

In the embodiment of the application, the image acquisition device may acquire the motion data in the first period through the motion sensor, the motion data may include a gesture, for example, the motion sensor may include an inertial sensor, where the inertial sensor may be a gyroscope, the gyroscope may acquire the gesture of the image acquisition device at each moment, and the image processing device may determine, according to the gesture of the image acquisition device when acquiring the first image and the gesture of the image acquisition device when acquiring the second image, a rotation angle of the image acquisition device in the first period.

In the second case, the image processing apparatus may estimate the rotation angle of the image capturing apparatus in the first period from the direction angles of the two feature points included in each of the plurality of initial point pairs.

In the embodiment of the present application, when the rotation angle cannot be determined by the image capturing apparatus, or in other possible cases, the image processing apparatus may estimate the rotation angle of the image capturing apparatus in the first period from the direction angles of the two feature points included in each of the plurality of pairs of initial points.

It should be noted that, when the optical axis of the camera mounted on the image capturing apparatus is perpendicular to the observation plane, since both the two main directions of the two feature points included in the feature point pair for representing the same object point are determined by the same neighborhood pixel point, the angle of rotation of the two main directions will be equal to the rotation angle of the image capturing apparatus, that is, the difference in direction angle between the direction angles of the two feature points for the two feature points included in the feature point pair for representing the same object point will be equal to the rotation angle of the image capturing apparatus. Based on this, the image processing apparatus may use, as the direction angle of the corresponding feature point, the included angle between the main direction of each feature point and the reference direction according to the main directions of the two feature points included in each of the plurality of obtained initial point pairs, and further may estimate the rotation angle of the image capturing apparatus in the first period according to the direction angle.

In this case, the image capturing apparatus may first select an initial point pair having a consistent rotation of direction angles from among a plurality of initial point pairs by the RANSAC method, calculate a direction angle difference value between the direction angles of the two feature points included in each of the selected initial point pairs, estimate a rotation angle from the direction angle difference value corresponding to each of the selected initial point pairs, and may calculate an average value of the plurality of direction angle difference values as the estimated rotation angle.

Alternatively, since the ratio of the initial point pairs with the wrong matching among the plurality of initial point pairs is high, it may be time-consuming to use the RANSAC method, so the image capturing apparatus may not use the RANSAC method to screen the initial point pairs, but may directly calculate a direction angle difference between the direction angles of the two feature points in each of the plurality of initial point pairs, calculate an average value or a median value of all the direction angle differences, etc., and the direction angle difference obtained by the statistics may be used to represent the direction angle difference of the initial point pair with the consistent direction angle rotation among the plurality of initial point pairs, and may use the direction angle difference obtained by the statistics as the rotation angle of the image capturing apparatus in the first period.

It should be noted that, in the embodiment of the present application, when the image processing apparatus includes a motion sensor, the step of determining the rotation angle of the image capturing apparatus in the first period of time by the image processing apparatus may also be performed before the step of acquiring the plurality of pairs of initial points, which is not limited by the embodiment of the present application.

Step 403: a plurality of pairs of initial points are determined for representing the pairs of feature points of the same object point based on the rotation angle and the difference in direction angle between the direction angles of the two feature points included in each of the pairs of initial points.

In the embodiment of the present application, it is known from the foregoing that, when the camera optical axis of the image capturing device is perpendicular to the plane in which the captured image is located, the direction angle difference between the direction angles of the two feature points used to represent the same object point on the first image and the second image is approximately equal to the rotation angle of the image capturing device. Based on this, the image processing apparatus can screen a plurality of pairs of initial points according to the rotation angle, the direction angle difference value, and further determine pairs of characteristic points that can be used to represent the same object point.

In one possible implementation manner, the image processing apparatus may determine a difference between the direction angle difference value corresponding to each initial point pair and the rotation angle according to the rotation angle and the direction angle difference value between the direction angles of the two feature points included in each initial point pair, so as to obtain a corresponding rotation angle difference value of each initial point pair. Then, the image processing apparatus may acquire, from the plurality of pairs of initial points, pairs of initial points whose corresponding rotation angle difference is smaller than the angle threshold, the acquired pairs of initial points as pairs of feature points for representing the same object.

In the embodiment of the present application, in the case where the image processing apparatus can determine a plurality of pairs of initial points by tracking matching, the image processing apparatus can determine the main directions of two feature points included in each of the pairs of the plurality of initial points based on the description of the correlation of the determination main directions, and then determine the direction angles of the two feature points included in each of the pairs of the feature points. In the case where the image processing apparatus cannot determine a plurality of pairs of initial points by tracking matching, since the image processing apparatus has already determined the main direction of each feature point in the image at the time of calculating the description of the feature point in the process of determining the plurality of pairs of initial points, it is possible to determine the direction angles of the two feature points included in each pair of initial points directly from the main directions determined as described above.

The image processing apparatus may calculate a direction angle difference value between the direction angles of the respective two feature points, that is, a direction angle difference value corresponding to the respective initial point pair, after determining the direction angles of the two feature points included in each of the initial point pairs. Then, the image processing apparatus may set a difference between the direction angle difference value and the rotation angle corresponding to each of the plurality of initial point pairs as a rotation angle difference value corresponding to the corresponding initial point pair, and may then reserve an initial point pair whose rotation angle difference value is smaller than an angle threshold value, reject an initial point pair whose rotation angle difference value is greater than or equal to the angle threshold value, and set the reserved initial point pair as a feature point pair.

It should be noted that, the angle threshold may be an arc value greater than 0, for example, 1 arc, the direction angle difference may be an arc value greater than or equal to 0, and the rotation angle difference may be an arc value greater than or equal to 0. In the embodiment of the present application, the rotation angle difference Δθ _d may be calculated according to equation (11).

Δθ_d＝||θ₂-θ₁|-Δθ| (11)

Wherein θ ₁、θ₂ is the direction angle of the feature point 1 and the feature point 2 included in one initial point pair, respectively, and Δθ is the rotation angle of the image capturing apparatus in the first period.

For example, still taking fig. 1 and 2 as an example, assuming that the angle threshold is θ _th,θ_th >0, the rotation angle of the image capturing apparatus in the first period is Δθ, the feature point a on the first image and the feature point B on the second image are one initial point pair, the direction angle of the feature point a is θ _A, the direction angle of the feature point B is θ _B,θ_A and the direction angle difference of θ _B may be |θ _B-θ_A|＝Δθ_AB, and the rotation angle difference corresponding to the two feature points may be |Δθ _AB-Δθ|＝Δθ_d, where |·| represents the calculated absolute value. The feature point a and the feature point B may be regarded as one feature point pair when Δθ _d<θ_th, and may be culled when Δθ _d≥θ_th.

Therefore, according to the rotation angle and the direction angle difference value corresponding to each initial point pair, the initial point pair with the direction angle change exceeding the angle threshold value can be eliminated as the initial point pair with the matching error, namely the initial point pair which is difficult to eliminate by other characteristic matching methods.

In another possible implementation, the image processing apparatus may determine the feature point pairs from the plurality of initial point pairs based on at least one of a descriptor distance and a comprehensive distance between two feature points included in each of the initial point pairs, and a direction angle difference between a rotation angle and a direction angle of the two feature points included in each of the initial point pairs. Among such possible implementations, there may be various implementations of determining feature point pairs from a plurality of initial point pairs, three of which are described in detail below.

In the first implementation, the image processing apparatus may determine the feature point pairs from the plurality of initial point pairs based on the rotation angle, the direction angle difference between the direction angles of the two feature points included in each of the initial point pairs, and the descriptor distance between the two feature points included in each of the initial point pairs.

Since the descriptors can represent the features of the feature points, the image processing apparatus can screen the pairs of initial points in combination with the rotation angle difference value and the descriptor distance. The image capturing apparatus may first determine a plurality of first candidate point pairs from the plurality of initial point pairs based on the rotation angle and the direction angle difference between the direction angles of the two feature points included in each of the initial point pairs. Then, the image processing apparatus determines a feature point pair from the plurality of first candidate point pairs based on the descriptor distance between the two feature points included in each of the first candidate point pairs. That is, the image processing apparatus may reject pairs of initial points of the plurality of pairs of initial points that are erroneously matched, that is, two feature points that cannot be used to represent the same object point, according to the rotation angle and the direction angle difference value corresponding to each pair of initial points, and further screen according to the descriptor distance.

In this implementation, the image processing apparatus may acquire, from the plurality of initial point pairs, initial point pairs whose corresponding rotation angle difference values are smaller than the angle threshold value, based on the angle threshold value, and take the acquired initial point pairs as the plurality of first candidate point pairs.

It should be noted that, the image processing apparatus may determine the plurality of first candidate point pairs according to the formula (11) and referring to the foregoing description of filtering according to the angle threshold, which is not described herein.

After acquiring the plurality of first candidate point pairs, the image processing apparatus may determine a descriptor distance between two feature points included in each of the plurality of first candidate point pairs according to descriptors of the two feature points, and then acquire, from the plurality of first candidate point pairs, a first candidate point pair having a descriptor distance between the two feature points smaller than a descriptor distance threshold, with the acquired first candidate point pair as a feature point pair.

The descriptor distance threshold may be preset according to the actual situation, and as can be known from the foregoing, the descriptor may be calculated by using operators such as SIFT, SURF, ORB, etc., and when calculating the descriptor distance, the euclidean distance, hamming distance, etc. may be used. For example, the SIFT and SURF operators use euclidean distances and the ORB operator uses hamming distances.

Illustratively, assuming that the descriptor distance threshold is dist _th, two feature points included in one first candidate point pair are feature point B and feature point C, and the descriptor distance between descriptors of feature point B and feature point C is dist ₁₂. The feature point B and the feature point C may be regarded as feature points that can be used to represent the same physical point when dist ₁₂<dist_th, and may be culled when dist ₁₂≥dist_th.

Optionally, in this implementation manner, the image processing apparatus may first screen the plurality of initial point pairs according to the descriptor distance to obtain a plurality of first candidate point pairs, and then screen the plurality of first candidate point pairs according to the rotation angle difference value to obtain the feature point pairs. The method of screening according to the descriptor distance and the method of screening according to the rotation angle difference may refer to the foregoing implementation manner, and the embodiments of the present application are not described herein again.

In the second implementation, the image processing apparatus may determine the feature point pairs from the plurality of initial point pairs based on the rotation angle, the direction angle difference between the direction angles of the two feature points included in each of the initial point pairs, and the integrated distance between the two feature points included in each of the initial point pairs.

In the embodiment of the application, the image processing device can screen the plurality of first candidate point pairs by combining the descriptor distance and the rotation angle difference value, namely the plurality of first candidate point pairs can be further screened according to the comprehensive distance. Since the integrated distance is a combination of two filtering conditions, namely, the descriptor distance and the variation of the direction angle rotation, the integrated distance can be used for comprehensively evaluating the matching error rate of the candidate point pairs.

For example, the image processing apparatus may determine a rotation angle difference value corresponding to each of the initial point pairs from the direction angle and the rotation angle of the two feature points included in each of the initial point pairs, acquire an initial point pair having a rotation angle difference value between the two feature points included in the plurality of initial point pairs smaller than an angle threshold value, and use the acquired initial point pair as the plurality of first candidate point pairs. Then, the image processing apparatus may acquire, from the plurality of first candidate point pairs, a first candidate point pair including two feature points having a combined distance smaller than a combined distance threshold, and take the acquired first candidate point pair as the feature point pair. The integrated distance threshold may be preset according to the actual situation.

It should be noted that, the implementation manner of the image processing apparatus for screening the plurality of initial point pairs according to the rotation angle difference value to obtain the plurality of first candidate point pairs may also refer to the foregoing related description, which is not repeated herein.

After determining the plurality of first candidate point pairs, the image processing apparatus may determine a descriptor distance between two feature points included in each of the first candidate point pairs from descriptors between the two feature points included in the corresponding first candidate point pairs. Then, the image processing apparatus may determine the integrated distance between the two feature points included in the corresponding first candidate point pair from the difference in rotation angle between the two feature points included in each first candidate point pair and the descriptor distance. The implementation manner of determining the descriptor distance may refer to the foregoing related description, which is not repeated herein.

When the image processing device determines the comprehensive distance according to the rotation angle difference value and the descriptor distance, a first weight and a second weight can be obtained, wherein the first weight is the weight of the rotation angle difference value, and the second weight is the weight corresponding to the descriptor distance. Then, the image processing apparatus may determine a comprehensive distance between two feature points included in the corresponding first candidate point pair according to the first weight, the rotation angle difference value corresponding to each first candidate point pair, the second weight, and the descriptor distance between two feature points included in the corresponding first candidate point pair.

It should be noted that the first weight and the second weight may be preset parameters. In addition, since the descriptor and the angle belong to different calculation dimensions, when the integrated distance is calculated according to the descriptor and the angle, the first weight and the second weight can be understood as two conversion factors, that is, the first weight and the second weight can convert the angle and the descriptor to the same calculation dimension.

In the embodiment of the present application, the image processing apparatus may first determine the rotation angle difference value by the above-described formula (11), and then calculate the integrated distance T ₁₂ by the following formula (12).

T₁₂＝α₁·Δθ_d+α₂·dist₁₂ (13)

Wherein Δθ _d represents the rotation angle difference between the feature point 1 and the feature point 2, dist ₁₂ represents the descriptor distance between descriptors of the feature point 1 and the feature point 2, α ₁、α₂ is the first weight and the second weight, respectively, and the feature point 1 and the feature point 2 are two feature points in a first candidate point pair.

For example, assuming that the integrated distance threshold is T _th, two feature points included in one first candidate point pair are feature point D and feature point E, the descriptor distance between the descriptors of the two feature points calculated is dist _DE, the direction angle difference between the direction angles of the two feature points is Δθ _DE, the rotation angle of the image capturing apparatus in the first period is Δθ ₁, the rotation angle difference between Δθ _DE and Δθ ₁ is |Δθ _DE-Δθ₁ |, and the integrated distance T _DE＝α₁·(|Δθ_DE-Δθ₁|)+α₂·dist_DE of the two feature points is obtained by adding α ₁·(|Δθ_DE-Δθ₁ |) to α ₂·dist_DE. When T _DE<T_th is implemented, the two feature points can be used as a feature point pair which can be used for representing the same object. When T _DE≥T_th, the two feature points can be eliminated.

Optionally, in this implementation manner, the image processing apparatus may first screen the plurality of initial point pairs according to the integrated distance to obtain a plurality of first candidate point pairs, and then screen the plurality of first candidate point pairs according to the rotation angle difference value, so as to obtain the feature point pairs. The method of screening according to the comprehensive distance and the method of screening according to the rotation angle difference may refer to the foregoing implementation method, and the embodiments of the present application are not described herein again.

In the third implementation, the image processing apparatus may determine the feature point pairs from the plurality of initial point pairs based on the rotation angle, the direction angle difference between the direction angles of the two feature points included in each of the initial point pairs, and the descriptor distance and the integrated distance between the two feature points included in each of the initial point pairs.

In this implementation manner, the image capturing apparatus may determine the rotation angle difference value corresponding to the corresponding initial point pair according to the rotation angle and the direction angle difference value between the direction angles of the two feature points included in each initial point pair, acquire, from the plurality of initial point pairs, an initial point pair whose corresponding rotation angle difference value is smaller than the angle threshold, and use the acquired initial point pair as the plurality of first candidate point pairs. That is, the image capturing device may first screen the plurality of initial point pairs according to the rotation angle difference value, so as to obtain a plurality of first candidate point pairs. The method for screening according to the rotation angle difference may be described with reference to the foregoing related description, and the embodiments of the present application are not described herein again.

Then, the image processing apparatus may acquire, from the plurality of first candidate point pairs, a first candidate point pair in which a descriptor distance between two feature points included is smaller than a descriptor distance threshold, and take the acquired first candidate point pair as a plurality of second candidate point pairs. And then the image acquisition equipment acquires second candidate point pairs with the comprehensive distance between the two included characteristic points smaller than the comprehensive distance threshold value from the plurality of second candidate point pairs, and takes the acquired second candidate point pairs as the characteristic point pairs. That is, the image acquisition device may first use the descriptor distance threshold to screen the plurality of first candidate point pairs, and then use the comprehensive distance threshold to further screen the plurality of first candidate point pairs, so as to finally obtain the feature point pairs that may be used to represent the same object point.

In the embodiment of the application, after the screening is performed by using the angle threshold value and the descriptor distance threshold value, the feature point pairs with wrong matching may still exist, so that the comprehensive distance can be used for further screening to remove the feature point pairs with larger descriptor distance and direction angle errors at the same time.

The method for calculating the descriptor distance and the integrated distance may refer to the related description, and will not be described herein.

Optionally, in the third implementation manner, the sequence of the three steps of filtering according to the descriptor distance, filtering according to the rotation angle and the direction angle difference value, and filtering according to the comprehensive distance may be optionally changed and combined, which is not limited by the embodiment of the present application.

It should be noted that in the above-mentioned several implementations, under the condition that the image capturing device may track and match to obtain multiple initial point pairs, the image processing device may first screen the multiple initial point pairs according to the rotation angle and the direction angle difference value, so as to reject most of the initial point pairs with incorrect matching, and then further screen according to the descriptor distance and/or the comprehensive distance, so that the calculation amount of the descriptors can be reduced, and the matching speed is improved.

In the embodiment of the application, after the screening is finished, the RANSAC method can be used for screening the obtained characteristic point pairs again, the mismatching can be further removed, since there is little mismatch in the feature point pairs at this time, i.e., the interior point rate is high, the number of iterations using the RANSAC method can be small and the speed can be very fast.

Fig. 5 is a flowchart of another image feature matching method according to an embodiment of the present application. Referring to fig. 5, the image processing apparatus may receive the first image and the second image transmitted from the image capturing apparatus, and determine whether the image capturing apparatus has also transmitted motion data. When the judgment is yes, the motion data may be received, a plurality of initial point pairs may be determined according to the motion data, and the direction angles of the two feature points included in each of the initial point pairs may be acquired. Then, a rotation angle of the image acquisition device in a first period is determined from the motion data. And then screening the plurality of initial point pairs according to the direction angle, the rotation angle and the angle threshold value of the two included characteristic points of each initial point pair. And then, screening again by using the descriptor distance, screening again by using the comprehensive distance, and finally screening by using the RANSAC method to obtain the final characteristic point pair. And when the judgment is negative, the characteristic points can be firstly extracted, a plurality of initial point pairs are determined according to the descriptor distance of each characteristic point, then the RANSAC method is used for screening, the direction angles of the two characteristic points included in each screened initial point pair are obtained, the rotation angle is estimated, then the screening can be sequentially carried out according to the angle threshold value, the descriptor distance threshold value and the comprehensive distance threshold value, and finally the RANSAC method is used for screening.

It should be noted that, in the embodiment of the present application, RANSAC screening after screening according to descriptor distance or comprehensive distance is an optional step.

In the embodiment of the application, when the characteristics of the texture image which is complex and repeated are matched, as descriptors of a plurality of characteristic points possibly exist in the image and are approximately consistent, if the filtering matching is carried out only according to the distance of the descriptors, the finally obtained plurality of characteristic point pairs still have more characteristic points with wrong matching, namely the accuracy rate is lower. The technical scheme provided by the application can perform feature matching according to the rotation angle of the image acquisition equipment and the corresponding direction angle difference value of each initial point, namely, according to the change amount of the rotation direction of the image acquisition equipment and the change amount of the direction angle rotation corresponding to the two matched feature points, and can reduce the influence of repeated textures to a certain extent. For example, when an image of a road surface is acquired, the image of a tile on the road surface is repeated, in which case there may be a pair of tiles with identical textures on the acquired two images, but the two tiles are actually two different objects, and if the feature points on the two tiles match as the initial point pairs, the initial point pairs are all actually wrong in matching, but because the descriptors of the initial point pairs are approximately identical, when the features match, if the filtering is performed only according to the descriptor distance, the wrong initial point pairs may not be able to be removed, and if the filtering is performed according to the direction angle, the wrong initial point pairs may be removed.

In summary, in the embodiment of the present application, most of the initial point pairs with wrong matching may be screened from the plurality of initial point pairs according to the rotation angle of the image capturing device and the direction angle difference between the direction angles of the two feature points included in each initial point pair, so as to obtain the feature point pair for representing the same object point. The method and the device utilize the principle that the direction angle difference value is approximately consistent with the rotation angle under the condition that the optical axis of the camera is approximately perpendicular to the observation plane, and can eliminate the error matching which is difficult to eliminate by other characteristic matching methods. In addition, the method for screening the characteristic point pairs is not influenced by the sparseness of the characteristic points in the image and the inner point rate, so that the robustness is high.

Fig. 6 is a schematic structural diagram of an image feature matching device according to an embodiment of the present application, where the image feature matching device may be implemented by software, hardware, or a combination of both as part or all of an image processing apparatus, and the image processing apparatus may be the image processing apparatus shown in fig. 3. Referring to fig. 6, the apparatus includes: an acquisition module 601, a first determination module 602 and a second determination module 603.

An obtaining module 601, configured to obtain a plurality of initial point pairs, where each initial point pair of the plurality of initial point pairs includes two feature points, one feature point of the two feature points is a feature point in the first image, and the other feature point is a feature point in the second image;

a first determining module 602, configured to determine a rotation angle of the image capturing device within a first period, where the first period is a period between a capturing time of the first image and a capturing time of the second image;

the second determining module 603 is configured to determine, according to the rotation angle and the direction angle difference between the direction angles of the two feature points included in each of the plurality of initial point pairs, a feature point pair for representing the same object point.

Optionally, the acquiring module 601 includes:

A second determining unit, configured to determine a descriptor of each feature point in the second image, for a main direction of each feature point in the second image;

And a third determining unit for determining the plurality of pairs of initial points based on the descriptor of each feature point in the first image and the descriptor of each feature point in the second image.

Optionally, the first determining module includes:

A receiving unit for receiving motion data acquired by the image acquisition device through a motion sensor included in the image acquisition device in a first period;

a first estimating unit for estimating a rotation angle of the image capturing device within a first period of time based on the motion data.

Optionally, the first determining module includes:

And a second estimating unit configured to estimate a rotation angle of the image capturing apparatus within the first period based on the direction angles of the two feature points included in each of the plurality of initial point pairs.

Optionally, the second determining module includes:

And a fourth determining unit configured to determine a feature point pair from the plurality of initial point pairs based on at least one of a descriptor distance and a comprehensive distance between the two feature points included in each of the initial point pairs, and a direction angle difference between the rotation angle and a direction angle of the two feature points included in each of the initial point pairs.

Optionally, the fourth determining unit includes:

A first determining subunit, configured to determine a difference between the direction angle difference value and the rotation angle corresponding to each initial point pair, so as to obtain a rotation angle difference value corresponding to each initial point pair;

and the second determining subunit is used for acquiring initial point pairs, of which the corresponding rotation angle difference value is smaller than the angle threshold value, from the plurality of initial point pairs, and taking the acquired initial point pairs as a plurality of first candidate point pairs.

A third determination subunit configured to acquire, from the plurality of first candidate point pairs, a first candidate point pair in which a descriptor distance between two feature points included is smaller than a descriptor distance threshold, and take the acquired first candidate point pair as a plurality of second candidate point pairs;

And a fourth determination subunit configured to acquire, from the plurality of second candidate point pairs, a second candidate point pair in which a combined distance between two feature points included is smaller than a combined distance threshold, and take the acquired second candidate point pair as a feature point pair.

Optionally, the fourth determining unit further includes:

In the embodiment of the application, most of the initial point pairs which are wrong in matching can be screened out from a plurality of initial point pairs according to the rotation angle of the image acquisition equipment and the direction angle difference value between the direction angles of the two characteristic points included in each initial point pair, so that the characteristic point pairs used for representing the same physical point are obtained. The method and the device utilize the principle that the direction angle difference value is approximately consistent with the rotation angle under the condition that the optical axis of the camera is approximately perpendicular to the observation plane, and can eliminate the error matching which is difficult to eliminate by other characteristic matching methods. In addition, the method for screening the characteristic point pairs is not influenced by the sparseness of the characteristic points in the image and the inner point rate, so that the robustness is high.

It should be noted that: the image feature matching device provided in the above embodiment only uses the division of the above functional modules for illustration when the features are matched, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the image feature matching device and the image feature matching method provided in the foregoing embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments, which are not repeated herein.

Fig. 7 is a schematic structural diagram of an image processing apparatus 700 according to an embodiment of the present application. The image processing apparatus 700 may be a portable mobile image processing apparatus such as: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, motion picture expert compression standard audio plane 3), an MP4 (Moving Picture Experts Group Audio Layer IV, motion picture expert compression standard audio plane 4) player, a notebook computer, or a desktop computer. The image processing device 700 may also be referred to by other names as user device, portable image processing device, laptop image processing device, desktop image processing device, etc.

In general, the image processing apparatus 700 includes: a processor 701 and a memory 702.

Processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 701 may be implemented in at least one hardware form of DSP (DIGITAL SIGNAL Processing), FPGA (Field-Programmable gate array), PLA (Programmable Logic Array ). The processor 701 may also include a main processor and a coprocessor, wherein the main processor is a processor for processing data in an awake state, and is also called a CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 may integrate a GPU (Graphics Processing Unit, image processor) for rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 701 may also include an AI (ARTIFICIAL INTELLIGENCE ) processor for processing computing operations related to machine learning.

Memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 702 is used to store at least one instruction for execution by processor 701 to implement the image feature matching method provided by the method embodiments of the present application.

In some embodiments, the image processing apparatus 700 may further optionally include: a peripheral interface 703 and at least one peripheral. The processor 701, the memory 702, and the peripheral interface 703 may be connected by a bus or signal lines. The individual peripheral devices may be connected to the peripheral device interface 703 via buses, signal lines or a circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 704, touch display 705, camera 706, audio circuitry 707, positioning component 708, and power supply 709.

A peripheral interface 703 may be used to connect I/O (Input/Output) related at least one peripheral device to the processor 701 and memory 702. In some embodiments, the processor 701, memory 702, and peripheral interface 703 are integrated on the same chip or circuit board; in some other embodiments, either or both of the processor 701, the memory 702, and the peripheral interface 703 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The Radio Frequency circuit 704 is configured to receive and transmit RF (Radio Frequency) signals, also referred to as electromagnetic signals. The radio frequency circuitry 704 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 704 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 704 includes: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and so forth. The radio frequency circuitry 704 may communicate with other image processing devices via at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the world wide web, metropolitan area networks, intranets, generation mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (WIRELESS FIDELITY ) networks. In some embodiments, the radio frequency circuit 704 may further include NFC (NEAR FIELD Communication) related circuits, which is not limited by the present application.

The display screen 705 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display 705 is a touch display, the display 705 also has the ability to collect touch signals at or above the surface of the display 705. The touch signal may be input to the processor 701 as a control signal for processing. At this time, the display 705 may also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards. In some embodiments, the display 705 may be a front panel provided to the image processing apparatus 700; in other embodiments, the display 705 may be at least two, disposed on different surfaces of the image processing apparatus 700 or in a folded design; in other embodiments, the display 705 may be a flexible display disposed on a curved surface or a folded surface of the image processing device 700. Even more, the display 705 may be arranged in a non-rectangular irregular pattern, i.e. a shaped screen. The display 705 may be made of LCD (Liquid CRYSTAL DISPLAY), OLED (Organic Light-Emitting Diode) or other materials.

The camera assembly 706 is used to capture images or video. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, a front camera is provided on a front panel of the image processing apparatus, and a rear camera is provided on a rear surface of the image processing apparatus. In some embodiments, the at least two rear cameras are any one of a main camera, a depth camera, a wide-angle camera and a tele camera, so as to realize that the main camera and the depth camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize a panoramic shooting and Virtual Reality (VR) shooting function or other fusion shooting functions. In some embodiments, camera assembly 706 may also include a flash. The flash lamp can be a single-color temperature flash lamp or a double-color temperature flash lamp. The dual-color temperature flash lamp refers to a combination of a warm light flash lamp and a cold light flash lamp, and can be used for light compensation under different color temperatures.

The audio circuit 707 may include a microphone and a speaker. The microphone is used for collecting sound waves of users and environments, converting the sound waves into electric signals, and inputting the electric signals to the processor 701 for processing, or inputting the electric signals to the radio frequency circuit 704 for voice communication. The microphones may be provided in a plurality of different portions of the image processing apparatus 700 for the purpose of stereo sound collection or noise reduction, respectively. The microphone may also be an array microphone or an omni-directional pickup microphone. The speaker is used to convert electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only the electric signal can be converted into a sound wave audible to humans, but also the electric signal can be converted into a sound wave inaudible to humans for ranging and other purposes. In some embodiments, the audio circuit 707 may also include a headphone jack.

The locating component 708 is operative to locate the current geographic location of the image processing device 700 for navigation or LBS (Location Based Service, location-based service). The positioning component 708 may be a positioning component based on the United states GPS (Global Positioning System ), the Beidou system of China, or the Galileo system of Russia.

The power supply 709 is used to supply power to the respective components in the image processing apparatus 700. The power supply 709 may be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 709 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, the image processing device 700 further includes one or more sensors 710. The one or more sensors 710 include, but are not limited to: acceleration sensor 711, gyroscope sensor 712, pressure sensor 713, fingerprint sensor 714, optical sensor 715, and proximity sensor 716.

The acceleration sensor 711 can detect the magnitudes of accelerations on three coordinate axes of the coordinate system established by the image processing apparatus 700. For example, the acceleration sensor 711 may be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 701 may control the touch display screen 705 to display a user interface in a landscape view or a portrait view according to the gravitational acceleration signal acquired by the acceleration sensor 711. The acceleration sensor 711 may also be used for the acquisition of motion data of a game or a user.

The gyro sensor 712 may detect a body direction and a rotation angle of the image processing apparatus 700, and the gyro sensor 712 may collect a 3D motion of the user on the image processing apparatus 700 in cooperation with the acceleration sensor 711. The processor 701 may implement the following functions based on the data collected by the gyro sensor 712: motion sensing (e.g., changing UI according to a tilting operation by a user), image stabilization at shooting, game control, and inertial navigation.

The pressure sensor 713 may be disposed at a side frame of the image processing apparatus 700 and/or at a lower layer of the touch display screen 705. When the pressure sensor 713 is provided at a side frame of the image processing apparatus 700, a grip signal of the image processing apparatus 700 by a user can be detected, and the processor 701 performs left-right hand recognition or quick operation according to the grip signal acquired by the pressure sensor 713. When the pressure sensor 713 is disposed at the lower layer of the touch display screen 705, the processor 701 controls the operability control on the UI interface according to the pressure operation of the user on the touch display screen 705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

The fingerprint sensor 714 is used to collect a fingerprint of the user, and the processor 701 identifies the identity of the user based on the fingerprint collected by the fingerprint sensor 714, or the fingerprint sensor 714 identifies the identity of the user based on the collected fingerprint. Upon recognizing that the user's identity is a trusted identity, the processor 701 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying for and changing settings, etc. The fingerprint sensor 714 may be provided on the front, back, or side of the image processing apparatus 700. When a physical key or vendor Logo is provided on the image processing device 700, the fingerprint sensor 714 may be integrated with the physical key or vendor Logo.

The optical sensor 715 is used to collect the ambient light intensity. In one embodiment, the processor 701 may control the display brightness of the touch display 705 based on the ambient light intensity collected by the optical sensor 715. Specifically, when the intensity of the ambient light is high, the display brightness of the touch display screen 705 is turned up; when the ambient light intensity is low, the display brightness of the touch display screen 705 is turned down. In another embodiment, the processor 701 may also dynamically adjust the shooting parameters of the camera assembly 706 based on the ambient light intensity collected by the optical sensor 715.

A proximity sensor 716, also referred to as a distance sensor, is typically provided on the front panel of the image processing apparatus 700. The proximity sensor 716 is used to capture the distance between the user and the front of the image processing apparatus 700. In one embodiment, when the proximity sensor 716 detects that the distance between the user and the front face of the image processing apparatus 700 gradually decreases, the processor 701 controls the touch display screen 705 to switch from the bright screen state to the off screen state; when the proximity sensor 716 detects that the distance between the user and the front surface of the image processing apparatus 700 gradually increases, the touch display screen 705 is controlled by the processor 701 to switch from the off-screen state to the on-screen state.

It will be appreciated by those skilled in the art that the structure shown in fig. 7 is not limiting of the image processing apparatus 700 and may include more or fewer components than shown, or may combine certain components, or may employ a different arrangement of components.

In some embodiments, there is also provided a computer readable storage medium having stored therein a computer program which, when executed by a processor, implements the steps of the image feature matching method of the above embodiments. For example, the computer readable storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

It is noted that the computer readable storage medium mentioned in the present application may be a non-volatile storage medium, in other words, a non-transitory storage medium.

It should be understood that all or part of the steps to implement the above-described embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions may be stored in the computer-readable storage medium described above.

That is, in some embodiments, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the steps of the image feature matching method described above.

The above embodiments are not intended to limit the present application, and any modifications, equivalent substitutions, improvements, etc. within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A method of image feature matching, the method comprising:

Determining a rotation angle of an image acquisition device in a first period, wherein the first period is a period between the acquisition time of the first image and the acquisition time of the second image, and the optical axis of a camera carried by the image acquisition device is vertical to an observation plane;

Determining a characteristic point pair used for representing the same object point in the plurality of initial point pairs according to the rotation angle and a direction angle difference value between direction angles of two characteristic points included in each initial point pair, wherein the direction angle is an included angle between a main direction of the characteristic point and a reference direction, and the main direction is a stable direction of the characteristic point in a local area where the characteristic point is located;

The acquiring a plurality of initial point pairs includes:

Determining a descriptor of each feature point in the first image according to the main direction of each feature point in the first image; determining descriptors of each feature point in the second image according to the main direction of each feature point in the second image; and determining the plurality of initial point pairs according to the descriptors of each characteristic point in the first image and the descriptors of each characteristic point in the second image.

2. The method of claim 1, wherein determining the rotation angle of the image capture device over the first period of time comprises:

3. The method of claim 1, wherein determining the rotation angle of the image capture device over the first period of time comprises:

4. A method according to any one of claims 1-3, wherein said determining pairs of feature points for representing the same object point from said rotation angle and a direction angle difference between the direction angles of two feature points included in each pair of initial points comprises:

5. The method of claim 4, wherein said determining said pairs of feature points from said plurality of pairs of initial points based on at least one of a descriptor distance and a composite distance between two feature points included in each pair of initial points, and a direction angle difference between said rotation angle and a direction angle of two feature points included in each pair of initial points, comprises:

acquiring initial point pairs with corresponding rotation angle difference values smaller than an angle threshold value from the plurality of initial point pairs, and taking the acquired initial point pairs as a plurality of first candidate point pairs;

and acquiring a second candidate point pair with the comprehensive distance between the two included feature points smaller than a comprehensive distance threshold value from the plurality of second candidate point pairs, and taking the acquired second candidate point pair as the feature point pair.

6. The method of claim 5, wherein the acquiring, from the plurality of second candidate point pairs, a second candidate point pair having a combined distance between two feature points that is less than a combined distance threshold, further comprises:

7. An image feature matching device, the device comprising:

An acquisition module, configured to acquire a plurality of initial point pairs, where each initial point pair in the plurality of initial point pairs includes two feature points, one feature point in the two feature points is a feature point in the first image, and the other feature point is a feature point in the second image;

a first determining module, configured to determine a rotation angle of an image capturing device in a first period, where the first period is a period between a capturing time of the first image and a capturing time of the second image, and an optical axis of a camera mounted on the image capturing device is perpendicular to an observation plane;

The second determining module is used for determining a characteristic point pair used for representing the same object point in the plurality of initial point pairs according to the rotation angle and a direction angle difference value between direction angles of two characteristic points included in each initial point pair, wherein the direction angle is an included angle between a main direction of the characteristic point and a reference direction, and the main direction is a stable direction of the characteristic point in a local area where the characteristic point is located;

The acquisition module comprises:

8. The apparatus of claim 7, wherein the first determining module comprises:

9. The apparatus of claim 7, wherein the first determining module comprises:

10. The apparatus according to any one of claims 7-9, wherein the second determining module comprises:

11. The apparatus according to claim 10, wherein the fourth determining unit includes:

A second determining subunit, configured to obtain, from the plurality of initial point pairs, an initial point pair whose corresponding rotation angle difference is smaller than an angle threshold, and use the obtained initial point pair as the plurality of first candidate point pairs;

A third determining subunit, configured to obtain, from the plurality of first candidate point pairs, a first candidate point pair in which a descriptor distance between two feature points included is smaller than a descriptor distance threshold, and take the obtained first candidate point pair as the plurality of second candidate point pairs;

12. The apparatus of claim 11, wherein the fourth determining unit further comprises:

13. An image processing device comprising a processor, a communication interface, a memory and a communication bus, the processor, the communication interface and the memory performing communication with each other via the communication bus, the memory being for storing a computer program, the processor being for executing the program stored on the memory to implement the steps of the method according to any one of claims 1-6.

14. A computer-readable storage medium, characterized in that the storage medium has stored therein a computer program which, when executed by a processor, implements the steps of the method according to any of claims 1-6.