Ocean structure swaying measurement system and method based on binocular image processing
Technical Field
The invention relates to the field of situation perception of marine structures, in particular to a binocular image processing-based marine structure swaying measurement system and method.
Background
In recent years, with the rapid development of technologies such as electronic information, automatic control, artificial intelligence and the like, the research of unmanned ships and aircrafts is gradually maturing, and great application prospects are shown. Unmanned ships and craft with intelligent navigation ability are core components of future intelligent navigation network, and will play an important role in ocean engineering. Wherein the motion perception of the marine structure plays an important role in the safe operation and unmanned control of the marine structure.
The shaking of the marine structure comprises pitching and rolling, the current sensing mode mainly comprises the steps that sensors such as an inertia measuring unit are fixed on a certain plane of the floating body, and the shaking of the floating body is measured through the inclination of the measuring plane. However, the conventional measurement method has great limitations:
a. due to the fact that the sizes of structures such as ships are large, the structures can generate large deformation when the structures navigate in waves, the inclination of each plane of the floating body is inconsistent with the shaking of the floating body, sensors such as an inertia measurement unit estimate the shaking of the floating body by measuring the inclination of a certain point, and accordingly the deformation of the floating body has large influence on a measurement result.
b. The application field has larger limitation. Sensor devices such as inertial measurement units are expensive and, due to the above-mentioned disadvantages, a plurality of such devices need to be arranged on the floating body, which requires a large communication overhead.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a binocular image processing-based marine structure sway measuring system and method, which are used for restoring the real depth information of waves by using a binocular camera under the condition of actually measured wind waves of the sea, further fitting to obtain an average water surface, and then obtaining the sway of the marine structure by using the position of the binocular camera relative to the average water surface.
In order to achieve the aim, the invention provides a binocular image processing-based marine structure sway measuring system, which comprises a binocular camera, a PC end of the marine structure sway measuring system and a supporting platform, wherein the PC end is connected with the binocular camera; the support platform is fixed on a side of a ship and is arranged opposite to the outer side of the side, and the binocular camera is fixed on the support platform; and the binocular camera is in communication connection with the PC end of the marine structure swaying measurement system.
Preferably, the binocular camera comprises two camera unit assemblies fixed on the support platform; the two camera unit components are positioned at the same height.
The invention relates to a binocular image processing ocean structure swaying measurement method, which comprises the following steps:
s1: the binocular camera transmits the acquired wave images to the PC end of the marine structure swaying measurement system;
s2: the PC end of the marine structure oscillation measuring system processes the wave image in real time;
the step of S2 further includes the steps of:
s21: carrying out binocular calibration on the wave image;
s22: performing stereo correction on the current wave image;
s23: carrying out stereo matching on the current wave image;
s24: calculating the depth information of the current wave image;
s25: calculating world coordinates of the characteristic points of the wave image;
s26: fitting the characteristic points of the current wave image to obtain an average water surface;
s27: the swaying of the marine structure is measured.
Preferably, the wave image collected by the binocular camera is restored to real depth information by the PC end of the marine structure swaying measurement system through an average wave surface based on binocular vision and a ship situation measurement algorithm, an average water surface is obtained through fitting, and the swaying of the marine structure is obtained according to the position of the binocular camera relative to the average water surface.
Preferably, in the step S23:
comparing the wave images shot by the two camera unit assemblies of the binocular camera, matching the same pixel blocks in the wave images, finding corresponding feature points, and calculating parallax disparity:
disparity=X R -X T
wherein X R And X T Is the column coordinate of the imaging point of the feature point on the photoreceptors of the two camera unit assemblies.
Preferably, in the step S24:
after the parallax is obtained, the real distance Z between the point P and the optical center of the camera can be obtained by applying the principle of triangulation:
wherein f is the focal length of the camera, and B is the center-to-center distance between the two camera unit components.
Preferably, in the step S25:
conversion relationship using camera coordinate system:
and obtaining world coordinates (X, Y, Z) of the characteristic points.
Wherein Z c The coordinates of the camera are u and v, the coordinates of the pixel are dx and dy, which are intrinsic parameters of the camera and respectively represent how many millimeters each pixel occupies in the x and y directions, f is the focal length of the camera, R is a rotation matrix, t is a displacement matrix, and X, Y, Z is the coordinate value of the world coordinate.
Preferably, in the step S26:
in the camera coordinate system, the average water surface equation is expressed as:
z m =[α 1 α 2 α 3 ][x m y m 1] T
wherein [ x ] m ,y m ,z m ]Is the coordinate of each feature point on the average water surface relative to the camera coordinate system, z m Also known as mean water depth; alpha is alpha i Is the coefficient of the plane of the mean water surface, provided that the depths of a sufficient number of characteristic points, alpha, on the wave surface are known i Can be obtained by least square fitting;
and after an equation expression of the average water surface is obtained, a depth map of the average water surface is further obtained.
Preferably, in the step S27:
obtaining the normal vector of the average water surface according to the average water surface equation
The roll angle of the marine structure may be determined by the camera's vector along the optical axis
Normal vector to mean water surface
The pitch angle can be obtained from the vector of the camera along the x-axis direction of the camera coordinate system
Normal vector to mean water surface
The included angle between the two is obtained, and the swaying of the ocean structure is obtained.
Due to the adoption of the technical scheme, the invention has the following beneficial effects:
1. and (4) introducing binocular image processing into ocean structure swing measurement of ocean actual measurement. Due to the fact that the sizes of structures such as ships are large, the structures can generate large deformation when the structures navigate in waves, the inclination of each plane of the floating body is inconsistent with the shaking of the floating body, sensors such as an inertia measurement unit estimate the shaking of the floating body by measuring the inclination of a certain point, and accordingly the deformation of the floating body has large influence on a measurement result. However, the binocular image processing-based sea structure swaying measurement system can restore the real depth information of waves through the binocular camera, further fit to obtain the average water surface, and then obtain the swaying of the sea structure according to the position of the binocular camera relative to the average water surface to avoid the problem.
2. Has wide application field. In comparison, sensor devices such as an inertial measurement unit are expensive, and the application field is limited greatly, so that a plurality of floating bodies need to be arranged, and a large communication overhead is required. The ocean structure oscillation measuring system based on binocular image processing can be suitable for various non-extreme sea conditions, is low in cost and can be carried on ships, unmanned aerial vehicles and other equipment.
3. The binocular vision-based average wave surface and ship situation measurement algorithm is used for processing the binocular images, and the method has the advantages of being high in speed, high in precision and real-time.
Drawings
Fig. 1 and fig. 2 are schematic structural diagrams of a marine structure swaying measurement system based on binocular image processing according to an embodiment of the invention;
fig. 3 is a flowchart of the marine structure sway measurement based on binocular image processing according to the embodiment of the present invention.
Fig. 4 is a schematic coordinate system diagram of a binocular camera according to an embodiment of the invention.
Detailed Description
The following description of the preferred embodiments of the present invention will be provided in conjunction with the accompanying drawings, which are set forth in the accompanying drawings and figures 1-4, to provide a better understanding of the function and features of the invention.
Referring to fig. 1 to 4, an ocean structure swaying measurement system based on binocular image processing according to an embodiment of the present invention includes a binocular camera 1, a PC end of the ocean structure swaying measurement system, and a support platform; the support platform is fixed on the side of a ship 2 and is arranged opposite to the outer side of the side, and the binocular camera 1 is fixed on the support platform; the binocular camera 1 is in communication connection with a PC end of the marine structure swaying measurement system.
The binocular camera 1 comprises two camera unit components fixed on a supporting platform; the two camera unit components are located at the same height.
The image obtained by the binocular camera 1 is transmitted to the PC end of the marine structure swaying measurement system through a network cable.
The PC end of the marine structure swaying measurement system runs an average wave surface and ship situation measurement algorithm program based on binocular vision, wave images of the sea level 3 collected by the binocular camera 1 can be processed in real time and fitted to obtain an average water surface, and swaying of the marine structure is obtained according to the position of the binocular camera 1 relative to the average water surface.
The invention uses a binocular image processing method to measure the swaying of marine structures, and the core algorithm based on the binocular image processing method is an average wave surface and ship situation measuring algorithm based on binocular vision. After the binocular device is built, an average wave surface based on binocular vision and a ship situation measuring program are operated to measure the swaying of the marine structure.
The binocular image processing ocean structure swaying measurement method comprises the following steps:
s1: the binocular camera 1 transmits the collected wave images to a PC end of the marine structure swaying measurement system;
s2: the PC end of the marine structure oscillation measurement system processes the wave image in real time;
the step of S2 further includes the steps of:
s21: carrying out binocular calibration on the wave image;
for the left and right two lenses of the binocular camera 1Respectively calibrating to obtain the internal reference, external reference and distortion coefficient of the left and right lenses, wherein the internal reference comprises the focal length f of the left and right lenses x ,f y Offset C of the lens optical axis in the image coordinate system x ,C y The external parameter comprises a rotation matrix R and a translation vector t of the left lens relative to the right lens, and the distortion coefficient comprises a radial distortion coefficient k 1 ,k 2 ,k 3 And tangential distortion coefficient p 1 ,p 2 。
S22: performing stereo correction on the current wave image;
the purpose of stereo correction is to perform mathematical projection transformation on the left view and the right view shot in the same scene, so that two imaging planes are parallel to a base line, and the same point is located in the same line in the left view and the right view, which are called coplanar line alignment for short. The distance can be calculated using trigonometric principles only after alignment of the coplanar rows is achieved.
S23: carrying out stereo matching on the current wave image;
comparing wave images shot by two camera unit components of the binocular camera 1, matching the same color blocks in the wave images, finding corresponding feature points, and calculating parallax disparity:
disparity=X R -X T
wherein X R And X T Is the column coordinate of the imaged point of the feature point on the photoreceptors of the two camera unit assemblies.
S24: calculating the depth information of the current wave image;
after the parallax is obtained, the real distance Z between the point P and the optical center of the camera can be obtained by applying the principle of trigonometry:
wherein f is the focal length of the camera, and B is the center distance of the two camera unit components.
S25: calculating world coordinates of the characteristic points of the wave image;
after the feature points are found by matching the pixel blocks in S23, the pixel coordinates corresponding to each feature point and the depth information of each feature point in S24 are obtained simultaneously using the transformation relationship of the camera coordinate system:
world coordinates (X, Y, Z) of the feature points are obtained.
Wherein Z c The coordinates of the camera are u and v, the coordinates of the pixel are dx and dy, which are intrinsic parameters of the camera and respectively represent how many millimeters each pixel occupies in the x and y directions, f is the focal length of the camera, R is a rotation matrix, t is a displacement matrix, and X, Y, Z is the coordinate value of the world coordinate.
S26: fitting the characteristic points of the current wave image to obtain an average water surface;
in the camera coordinate system, the plane of the average water surface can be represented mathematically by a three-dimensional plane, and the equation is as follows:
z m =[α 1 α 2 α 3 ][x m y m 1] T
wherein [ x ] m ,y m ,z m ]Is the coordinate of each feature point on the average water surface relative to the camera coordinate system, z m Also known as mean water depth; alpha is alpha i Is the coefficient of the plane of the mean water surface, provided that the depths of a sufficient number of characteristic points, alpha, on the wave surface are known i Can be obtained by least square fitting;
and after an equation expression of the average water surface is obtained, a depth map of the average water surface is further obtained.
S27: the swaying of the marine structure is measured.
According to the mean water surface equation z m =[α 1 α 2 α 3 ][x m y m 1] T
Normal vector of average water surface can be obtained
The roll angle of the marine structure may be determined by the camera's vector along the optical axis (z-axis)
Normal vector to mean water surface
The pitch angle can be obtained from the vector of the camera along the x-axis direction of the camera coordinate system
Normal vector to mean water surface
The included angle between the two is obtained, and the swaying of the marine structure is obtained.
While the present invention has been described in detail and with reference to the embodiments thereof as illustrated in the accompanying drawings, it will be apparent to one skilled in the art that various changes and modifications can be made therein. Therefore, certain details of the embodiments are not to be interpreted as limiting, and the scope of the invention is to be determined by the appended claims.