CN114567758A - Automobile 360-degree panoramic video processing system and method - Google Patents

Automobile 360-degree panoramic video processing system and method Download PDF

Info

Publication number
CN114567758A
CN114567758A CN202210230400.6A CN202210230400A CN114567758A CN 114567758 A CN114567758 A CN 114567758A CN 202210230400 A CN202210230400 A CN 202210230400A CN 114567758 A CN114567758 A CN 114567758A
Authority
CN
China
Prior art keywords
video
camera
car
automobile
environment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210230400.6A
Other languages
Chinese (zh)
Inventor
高宏稳
陈群飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Chaoyue Intelligent Electronics Co ltd
Original Assignee
Shenzhen Chaoyue Intelligent Electronics Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Chaoyue Intelligent Electronics Co ltd filed Critical Shenzhen Chaoyue Intelligent Electronics Co ltd
Priority to CN202210230400.6A priority Critical patent/CN114567758A/en
Publication of CN114567758A publication Critical patent/CN114567758A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/165—Anti-collision systems for passive traffic, e.g. including static obstacles, trees
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/363—Image reproducers using image projection screens
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N17/00—Diagnosis, testing or measuring for television systems or their details
    • H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

本发明提出了一种汽车360度全景视频处理系统和方法，其系统包括：全景视频采集模块，用于采集汽车近身四周环境的视频，生成第一视频集；全景视频处理模块，用于根据所述第一视频集中的视频生成三维仿真场景，利用三维仿真场景，获得影响汽车安全行驶的信息；全景安全提示模块，用于根据影响汽车安全行驶的信息，提供安全提示。本发明通过对360度全景视频的获取和处理，能够帮助驾驶员从适宜视角和视野看到汽车近身四周环境情况，提高驾驶的安全性。

Figure 202210230400

The present invention provides a system and method for processing a 360-degree panoramic video of an automobile. The system includes: a panoramic video acquisition module, which is used to collect the video of the surrounding environment of the automobile to generate a first video set; and a panoramic video processing module, which is used for The videos in the first video set generate a three-dimensional simulation scene, and the three-dimensional simulation scene is used to obtain information that affects the safe driving of the car; the panoramic safety prompt module is used to provide safety prompts according to the information affecting the safe driving of the car. Through the acquisition and processing of 360-degree panoramic video, the present invention can help the driver to see the surrounding environment of the car from a suitable angle of view and field of vision, thereby improving the safety of driving.

Figure 202210230400

Description

Automobile 360-degree panoramic video processing system and method
Technical Field
The invention relates to the technical field of intelligent traffic, in particular to a 360-degree panoramic video processing system and method for an automobile.
Background
The 360-degree automobile panoramic video is used for providing 360-degree video image information in a driving environment for an automobile driver; in the driving process of the automobile, the parking of the automobile and the crowded urban traffic condition of the automobile, a driver can better judge the road environment around the automobile through the 360-degree panoramic video, so that the collision with surrounding pedestrians and the vehicle is avoided, and the driving process is safer and more convenient; the existing 360-degree panoramic video processing of the automobile also has the problems of insufficient acquisition efficiency, relatively backward video processing method, insufficient timeliness for providing safety prompts and the like; accordingly, there is a need for an improved automotive 360 degree panoramic video processing system and method.
Disclosure of Invention
The invention provides a system and a method for processing a 360-degree panoramic video of an automobile, which can help a driver to see the surrounding situation of the automobile from a proper visual angle and a proper visual field by acquiring and processing the 360-degree panoramic video, and improve the driving safety.
The invention provides an automobile 360-degree panoramic video processing system, which comprises:
the panoramic video acquisition module is used for acquiring videos of the near-body surrounding environment of the automobile and generating a first video set;
the panoramic video processing module is used for generating a three-dimensional simulation scene according to the video in the first video set and acquiring information influencing the safe driving of the automobile by using the three-dimensional simulation scene;
and the panoramic safety prompt module is used for providing safety prompt according to the information influencing the safe driving of the automobile.
Further, the panoramic safety prompt module comprises a main display screen and an auxiliary display screen, wherein the main display screen is embedded in the central console in the automobile and is used for displaying a 360-degree panoramic video aerial view around the vehicle; the auxiliary display screen is arranged right in front of the sight line of the driver and used for displaying the near-body and peripheral aerial view of the automobile within the range of 20 degrees left and right by taking the visual angle direction of the driver as the center;
the near-body surrounding aerial view of the automobile is obtained by performing projection mapping on a near-body 360-degree panoramic video of the automobile based on an image re-projection program and in combination with the head movement angle information of a driver;
when the environment of the automobile is an open space environment, displaying a bird's-eye view around the automobile close to the automobile only through the auxiliary display screen; when the environment of the automobile is a narrow space environment, the primary display screen is opened to display the aerial view of 360-degree panoramic video around the automobile near the automobile body while the secondary display screen displays the aerial view around the automobile near the automobile body.
Furthermore, the panoramic video acquisition module acquires videos by using a plurality of 180-degree ultra-wide-angle fisheye cameras; the ultra-wide angle fisheye camera is arranged at the front, the back, the left and the right of the vehicle body according to a preset angle, and the preset angle is adjusted and set according to the area shot by the fisheye camera;
the panoramic video acquisition module comprises a video primary selection unit, a video inspection unit and a video set generation unit;
the video primary selection unit is used for periodically acquiring videos of the surrounding environment of the close body of the automobile shot by the camera according to a preset period to obtain a plurality of video sets;
the video inspection unit is used for inspecting the videos in the plurality of video sets according to a preset video inspection rule; the video inspection rule comprises: evaluating the image definition in the video and judging the area range corresponding to the video; when the image definition in the video is smaller than a preset image definition threshold value, or the image boundary in the video is larger than or smaller than a preset image boundary, judging that the video does not meet the requirement, and discarding the video; when the image definition in the video is greater than or equal to a preset image definition threshold value or the image boundary in the video is superposed with a preset image boundary, judging that the video is qualified in quality and reserving the video;
the video set generating unit is used for generating a video set; and in the same period, taking the video reserved by the video inspection unit as a quality qualified video, and converging the quality qualified video to generate a first video set.
Further, the panoramic video processing module comprises a three-dimensional simulation scene construction unit; the three-dimensional simulation scene construction unit comprises a camera calibration subunit, a model parameter acquisition subunit, a projection model construction subunit and a three-dimensional simulation scene generation subunit;
the camera calibration subunit is used for calibrating the internal parameters and the external parameters of the camera; calculating to obtain a first texture mapping table and a second texture mapping table according to the internal parameters and the external parameters of the camera;
the model parameter acquiring subunit is used for calculating the distance between the camera in the first video set image and the main planes of the environments at the left side and the right side of the automobile; matching the distance with the first texture mapping table to obtain a first projection model parameter; matching the distance with the second texture mapping table to obtain a second projection model parameter;
the projection model construction subunit is used for establishing a first projection model according to the first projection model parameter; establishing a second projection model according to the second projection model parameters;
the three-dimensional panorama generating subunit is used for generating a first three-dimensional simulation scene according to the first projection model, wherein the first three-dimensional simulation scene corresponds to an automobile in an open space environment; and generating a second three-dimensional simulation scene according to the second projection model, wherein the second three-dimensional simulation scene corresponds to an automobile in a narrow space environment.
Further, the model parameter obtaining subunit includes a space environment determining subunit, configured to determine that an environment in which the automobile is located is an open space or a narrow space; the method comprises the following specific steps:
performing optical flow tracking and matching on videos of the environments on the left side and the right side of the automobile, determining feature points on a main plane of a side-looking obstacle according to the change rule of the optical flow, and estimating relative depth values of the feature points according to epipolar geometry and triangulation;
transforming the side-view image of the rear-view camera to a top view, and calculating to obtain the inter-frame motion pose of the camera:
obtaining a scale factor according to a calibration result of the camera;
calculating the distance between the camera and the main planes of the environments at the left side and the right side of the automobile according to the relative depth value, the inter-frame motion pose and the scale factor;
judging whether the environment of the automobile is an open space or a narrow space according to the distance and the preset distance threshold value; when the distance is smaller than a preset distance threshold value, judging that the automobile is in a narrow space environment; and when the distance is larger than a preset distance threshold value, judging that the automobile is in an open space environment.
A360-degree panoramic video processing method for an automobile comprises the following steps:
s1, collecting videos of the environment around the automobile close to the automobile, and generating a first video set;
s2, generating a three-dimensional simulation scene according to the video in the first video set, and observing the three-dimensional simulation scene to obtain information influencing the safe driving of the automobile;
and S3, providing safety prompt according to the information influencing the safe driving of the automobile.
Further, S1 includes
S101, arranging two adjacent radars on an automobile, selecting a certain moment, and respectively acquiring data of a first target at a first fixed position in the environment around the automobile by using the two radars to obtain first data and second data; analyzing the overlapping content of the first data and the second data, taking the overlapping content of the first data as third data, taking the overlapping content of the second data as fourth data, and preliminarily judging that the two radars work normally if the third data and the fourth data both correspond to the first target; performing data acquisition and analysis again after the two radars are shut down and restarted, and if the two radars are still judged to work normally, determining that the two radars work normally;
s102, selecting the same time as that in the step S101, and acquiring data of the first target located at the first fixed position in the environment around the automobile to be detected by using the camera to be detected to obtain fifth data; judging whether the corresponding targets in the fifth data and the first data or the second data are the first targets, if so, judging that the camera to be detected works normally; if not, performing correction operation on the camera to be detected;
s103, executing a correction operation on the camera, wherein the correction operation comprises the following steps: the method comprises the following steps of shutting down, resetting and restarting a camera, or adjusting the angle of a camera lens to a preset normal working angle, or adjusting the focal length of the camera lens to a preset normal working focal length; after the correction operation is finished, the operation is transferred to the step S102 to continue to be checked until the camera works normally, and the operation of detecting and correcting the performance of the camera is finished;
s104, regularly acquiring videos of the surrounding environment of the automobile close to the body, which are shot by a camera, according to a preset period to obtain a plurality of video sets;
s105, checking videos in the plurality of video sets according to a preset video checking rule; the video inspection rule comprises: evaluating the image definition in the video and judging the area range corresponding to the video; when the image definition in the video is smaller than a preset image definition threshold value or the area range boundary corresponding to the video is larger than or smaller than a preset image boundary, judging that the video does not meet the requirement, and discarding the video; when the image definition in the video is greater than or equal to a preset image definition threshold value or the image boundary in the video is superposed with a preset image boundary, judging that the video is qualified in quality and reserving the video;
and S106, in the same period, taking the video reserved by the video inspection unit as a quality qualified video, and converging the quality qualified video to generate a first video set.
Further, S2 includes constructing a three-dimensional simulation scene, and the specific steps include:
s201, calibrating internal parameters and external parameters of a camera; calculating to obtain a first texture mapping table and a second texture mapping table according to the internal parameters and the external parameters of the camera;
s202, calculating the distance between a camera in the first video set image and the main planes of the environments on the left side and the right side of the automobile; matching the distance with the first texture mapping table to obtain a first projection model parameter; matching the distance with the second texture mapping table to obtain a second projection model parameter;
s203, establishing a first projection model according to the first projection model parameters; establishing a second projection model according to the second projection model parameters;
s204, generating a first three-dimensional simulation scene according to the first projection model, wherein the first three-dimensional simulation scene corresponds to an automobile in an open space environment; and generating a second three-dimensional simulation scene according to the second projection model, wherein the second three-dimensional simulation scene corresponds to an automobile in a narrow space environment.
Further, S202 includes determining that the environment in which the vehicle is located is an open space or a narrow space; the method comprises the following specific steps:
s2021, carrying out optical flow tracking and matching on videos of the environment on the left side and the right side of the automobile, determining feature points on a main plane of the side-looking obstacle according to a change rule of the optical flow, and estimating relative depth values of the feature points according to epipolar geometry and triangulation;
s2022, transforming the side-view image of the rear-view camera to a top view, and calculating to obtain the inter-frame motion pose of the camera:
s2023, obtaining a scale factor according to a calibration result of the camera;
s2024, calculating to obtain the distance between the camera and the main planes of the left and right environments of the automobile according to the relative depth value, the inter-frame motion pose and the scale factor;
s2025, judging that the environment where the automobile is located is a wide space or a narrow space according to the distance and the preset distance threshold value; when the distance is smaller than a preset distance threshold value, judging that the automobile is in a narrow space environment; and when the distance is larger than a preset distance threshold value, judging that the automobile is in an open space environment.
Further, the step of providing a safety prompt in S3 includes the specific steps of:
s301, acquiring first distance data between a camera and main planes of environments on the left side and the right side of an automobile, second distance data between the camera and the main planes of the environments on the front side and the rear side of the automobile and third distance data between a plane of the top of the automobile and the main planes of the environments outside the automobile in real time, and storing the first distance data and the second distance data into a preset database;
s302, constructing a collision risk analysis model based on an artificial neural network according to historical first interval data, historical second interval data and historical third interval data in the database; the collision risk analysis model inputs data as interval data and outputs data as a collision risk value;
s303, according to a collision risk analysis model, collision risk values of the first interval data, the second interval data and the third interval data are respectively determined, and a first collision risk value, a second collision risk value and a third collision risk value are obtained; sending out corresponding safety prompts according to the analysis results of the first collision risk value, the second collision risk value and the third collision risk value;
judging whether the first collision risk value is greater than or equal to a preset first collision risk threshold value or not, and if so, sending a first safety prompt to an automobile driver;
judging whether the second collision risk value is greater than or equal to a preset second collision risk threshold value or not, and if so, sending a second safety prompt to the automobile driver;
judging whether the third collision risk value is greater than or equal to a preset third collision risk threshold value, and if so, sending a third safety prompt to the automobile driver;
and S304, the automobile driver takes corresponding risk avoiding driving operation according to the first safety prompt, the second safety prompt or the third safety prompt.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 is a block diagram of an automotive 360 degree panoramic video processing system according to the present invention;
FIG. 2 is a flow chart of a method for processing a 360-degree panoramic video of an automobile according to the present invention;
FIG. 3 is a flowchart of a method for providing a security prompt of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
The invention provides a 360-degree panoramic video processing system of an automobile, which comprises the following components as shown in figure 1:
the panoramic video acquisition module is used for acquiring videos of the environment around the automobile close to the body and generating a first video set;
the panoramic video processing module is used for generating a three-dimensional simulation scene according to the video in the first video set and acquiring information influencing the safe driving of the automobile by using the three-dimensional simulation scene;
and the panoramic safety prompt module is used for providing safety prompt according to the information influencing the safe driving of the automobile.
The working principle of the technical scheme is as follows: the 360-degree panoramic video of the automobile can help a driver to better observe the surrounding environment of the automobile nearby, acquire traffic information in the environment where the automobile is located, and provide help on the visual field for the driver to drive the automobile, so that the safe driving of the automobile is guaranteed. The system comprises a panoramic video acquisition module, a panoramic video acquisition module and a panoramic video acquisition module, wherein the panoramic video acquisition module is used for acquiring videos of the environment around the automobile close to the body and generating a first video set; the panoramic video processing module is used for generating a three-dimensional simulation scene according to the video in the first video set and acquiring information influencing the safe driving of the automobile by using the three-dimensional simulation scene; and the panoramic safety prompt module is used for providing safety prompt according to the information influencing the safe driving of the automobile.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the driver can be helped to see the environment conditions around the automobile from a proper visual angle and a proper visual field by acquiring and processing the 360-degree panoramic video, and the driving safety is improved.
In one embodiment, the panoramic safety prompting module comprises a main display screen and a secondary display screen, wherein the main display screen is embedded in a central console in the automobile and is used for displaying a 360-degree panoramic video aerial view around the near body of the automobile; the auxiliary display screen is arranged right in front of the sight line of the driver and used for displaying the near-body and peripheral aerial view of the automobile within the range of 20 degrees left and right by taking the visual angle direction of the driver as the center;
the near-body surrounding aerial view of the automobile is obtained by performing projection mapping on a near-body 360-degree panoramic video of the automobile based on an image re-projection program and in combination with the head movement angle information of a driver;
when the environment of the automobile is an open space environment, displaying a bird's-eye view around the automobile close to the automobile only through the auxiliary display screen; when the environment of the automobile is a narrow space environment, the primary display screen is opened to display the aerial view of 360-degree panoramic video around the automobile near the automobile body while the secondary display screen displays the aerial view around the automobile near the automobile body.
The working principle of the technical scheme is as follows: considering the difference of the environment where the automobile runs, different display modes are necessary to be set; when the automobile passes through a narrow road and a road with obstacles, and is backed and climbed to a slope, the environment of other parts around the automobile body cannot be seen at the same time, and the panorama around the automobile body needs to be displayed through a main display screen; and only through the main display screen display, under the open environment condition of car, can not provide the observation visual angle of geometric proportion according to the observation custom of human eye, build real external environment for the driver, it is necessary only to open vice display screen work. The specific composition structure and implementation mode of the implementation are as follows: the panoramic safety prompt module comprises a main display screen and an auxiliary display screen, wherein the main display screen is embedded in a central console in the automobile and is used for displaying a 360-degree panoramic video aerial view around the automobile close to the body; the auxiliary display screen is arranged right in front of the sight line of the driver and used for displaying the near-body and peripheral aerial view of the automobile within the range of 20 degrees left and right by taking the visual angle direction of the driver as the center;
the near-body surrounding aerial view of the automobile is obtained by performing projection mapping on a near-body 360-degree panoramic video of the automobile based on an image re-projection program and in combination with the head movement angle information of a driver;
when the environment of the automobile is an open space environment, displaying a bird's-eye view around the automobile close to the automobile only through the auxiliary display screen; when the environment of the automobile is a narrow space environment, the primary display screen is opened to display the aerial view of 360-degree panoramic video around the automobile near the automobile body while the secondary display screen displays the aerial view around the automobile near the automobile body.
The beneficial effects of the above technical scheme are: adopt the scheme that this embodiment provided, utilize to set up main display screen and vice display screen, can go the difference of environment in place according to the car, show car close-in panoramic video all around, improved the pertinence of observing car close-in environment all around.
In one embodiment, the panoramic video acquisition module acquires videos by using a plurality of 180-degree ultra-wide-angle fisheye cameras; the ultra-wide angle fisheye camera is arranged at the front, the back, the left and the right of the vehicle body according to a preset angle, and the preset angle is adjusted and set according to the area shot by the fisheye camera;
the panoramic video acquisition module comprises a video primary selection unit, a video inspection unit and a video set generation unit;
the video primary selection unit is used for periodically acquiring videos of the surrounding environment of the close body of the automobile shot by the camera according to a preset period to obtain a plurality of video sets;
the video inspection unit is used for inspecting the videos in the plurality of video sets according to a preset video inspection rule; the video inspection rule comprises: evaluating the image definition in the video and judging the area range corresponding to the video; when the image definition in the video is smaller than a preset image definition threshold value, or the image boundary in the video is larger than or smaller than a preset image boundary, judging that the video does not meet the requirement, and discarding the video; when the image definition in the video is greater than or equal to a preset image definition threshold value or the image boundary in the video is superposed with a preset image boundary, judging that the video is qualified in quality and reserving the video;
the video set generating unit is used for generating a video set; and in the same period, taking the video reserved by the video inspection unit as a quality qualified video, and converging the quality qualified video to generate a first video set.
The working principle of the technical scheme is as follows: the video is formed by combining a series of continuous images, so the video image splicing technology can use the principle of the image splicing technology for reference; in order to ensure the quality of the obtained video images and the image splicing effect, the quality control inspection needs to be carried out on the video collected by the fisheye camera, when the video images to be spliced all accord with the preset quality qualified condition, the obtained video is reserved, the reserved video is integrated, and the video set meeting the requirements is obtained.
The method comprises the steps of firstly, regularly acquiring videos of the surrounding environment of the close body of the automobile shot by a camera according to a preset period to obtain a plurality of video sets; then, the videos in the plurality of video sets are inspected according to a preset video inspection rule; the video inspection rule comprises: evaluating the image definition in the video and judging the area range corresponding to the video; when the image definition in the video is smaller than a preset image definition threshold value, or the image boundary in the video is larger than or smaller than a preset image boundary, judging that the video does not meet the requirement, and discarding the video; when the image definition in the video is greater than or equal to a preset image definition threshold value and the boundary of an image in the video is superposed with a preset image boundary, judging that the video is qualified in quality and reserving the video; and finally, in the same period, taking the video reserved by the video inspection unit as a quality-qualified video, and converging the quality-qualified video to generate a first video set.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the quality control inspection is carried out on the video collected by the fisheye camera, so that the quality of the obtained video image can be ensured, and the image splicing effect is ensured.
In one embodiment, the panoramic video processing module comprises a three-dimensional simulation scene construction unit; the three-dimensional simulation scene construction unit comprises a camera calibration subunit, a model parameter acquisition subunit, a projection model construction subunit and a three-dimensional simulation scene generation subunit;
the camera calibration subunit is used for calibrating the internal parameters and the external parameters of the camera; calculating to obtain a first texture mapping table and a second texture mapping table according to the internal parameters and the external parameters of the camera;
the model parameter acquiring subunit is used for calculating the distance between the camera in the first video set image and the main planes of the environments at the left side and the right side of the automobile; matching the distance with the first texture mapping table to obtain a first projection model parameter; matching the distance with the second texture mapping table to obtain a second projection model parameter;
the projection model construction subunit is used for establishing a first projection model according to the first projection model parameter; establishing a second projection model according to the second projection model parameters;
the three-dimensional panorama generating subunit is configured to generate a first three-dimensional simulation scene according to the first projection model, where the first three-dimensional simulation scene corresponds to an environment where an automobile is in an open space; and generating a second three-dimensional simulation scene according to the second projection model, wherein the second three-dimensional simulation scene corresponds to an automobile in a narrow space environment.
The working principle of the technical scheme is as follows: in order to better meet the panoramic view observation requirements of automobiles in different environments, it is necessary to establish corresponding three-dimensional simulation scenes for the different environments. In this embodiment, a first texture mapping table and a second texture mapping table are obtained by calculation according to an internal parameter and an external parameter of a camera calibrated in advance; the method comprises the steps of calculating the distance between a camera in a video image and main planes of the environment on the left side and the right side of an automobile according to different environments by analyzing actual environment information of the video image, obtaining corresponding projection model parameters through a texture mapping table, constructing a first projection model and a second projection model according to the projection model parameters, and completing construction of a three-dimensional simulation scene for an open space environment and a three-dimensional simulation scene for a narrow space environment according to the projection models.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the three-dimensional simulation panorama matched with the camera is constructed according to the different distances between the camera and the main planes of the environments at the left side and the right side of the automobile in the video image, the three-dimensional simulation panorama can be switched in a self-adaptive manner according to the different environments of the automobile, and a driver can conveniently observe the panoramic video of the environment around the automobile in a targeted manner.
In one embodiment, the model parameter obtaining subunit includes a space environment determining subunit, configured to determine that an environment where the automobile is located is an open space or a narrow space; the method specifically comprises the following steps:
performing optical flow tracking and matching on videos of the environments on the left side and the right side of the automobile, determining feature points on a main plane of a side-looking obstacle according to the change rule of the optical flow, and estimating relative depth values of the feature points according to epipolar geometry and triangulation;
transforming the side-view image of the rear-view camera to a top view, and calculating to obtain the inter-frame motion pose of the camera:
obtaining a scale factor according to a calibration result of the camera;
calculating the distance between the camera and the main planes of the environments at the left side and the right side of the automobile according to the relative depth value, the inter-frame motion pose and the scale factor;
judging whether the environment of the automobile is an open space or a narrow space according to the distance and the preset distance threshold value; when the distance is smaller than a preset distance threshold value, judging that the automobile is in a narrow space environment; and when the distance is larger than a preset distance threshold value, judging that the automobile is in an open space environment.
The working principle of the technical scheme is as follows: in order to calculate the distance between the camera and the main planes of the left and right environments of the automobile, the video needs to be tracked to obtain feature points, and then matching is carried out, so that the relative motion of the camera is estimated; the optical flow can detect small motion changes of objects through a continuous image sequence, acquire image frames of videos on two sides, perform feature point tracking matching, and judge feature points on main planes of side-looking nearest obstacles (planes of objects on two sides, such as vehicle vertical planes of parking lots, green belts on driving roads and building planes nearest to roadsides) according to tracking results. In the feature point selection, firstly, an image processing interested area with larger feature point optical flow values and concentrated distribution on the same main plane is selected, the optical flow values of the feature points in the area are calculated, an optical flow value distribution histogram is calculated, a plurality of groups with larger horizontal component values and smaller vertical component values and larger quantity of optical flows are used as feature points to be selected, and then a group of feature points with pixel coordinate point distribution span meeting a preset threshold value is selected from the feature points to be selected and used as the main plane feature points.
For corresponding feature points located on the principal plane, relative depth values are estimated from epipolar geometry and triangulation. Epipolar geometry is used for describing the geometric relationship between two images, and only needs the internal reference and the relative posture of a camera; epipolar geometry is commonly used in the aspects of camera matrix recovery, pose estimation, three-dimensional reconstruction and the like; calculating an essential matrix and a basic matrix according to epipolar geometry, recovering a camera matrix through the essential matrix and decomposing to obtain an inter-frame motion pose; according to the corresponding relation between the size of the top view and the actual distance, the scale factor between the pixel distance of the top view and the actual distance is determined by the actual width of the top view area and the width of the top view; the distance between the camera and the main planes of the environment at the left side and the right side of the automobile can be calculated through the rotation matrix and the translational motion vector value in the interframe motion pose, and the method specifically comprises the following steps:
Figure BDA0003540290810000131
in the above formula, d is the distance between the camera and the main planes of the left and right environments of the automobile, and alpha1Is the normalized coordinate, alpha, of the feature point corresponding to the previous frame image2The normalized coordinates of the feature points corresponding to the next frame of image; p is a rotation matrix in the inter-frame motion pose of the camera from the previous frame image to the next frame image, n is a translation vector in the inter-frame motion pose of the camera from the previous frame image to the next frame image, and deltarThe actual width of the top view area of the video image is a preset value; δ is the width of the top view of the image, which is a preset value;
comparing the distance between the camera and the main planes of the environments on the left side and the right side of the automobile with a preset distance threshold value, and judging whether the environment where the automobile is located is a wide space or a narrow space; when the distance is smaller than a preset distance threshold value, judging that the automobile is in a narrow space environment; and when the distance is larger than a preset distance threshold value, judging that the automobile is in an open space environment.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the distance between the camera and the main planes of the environments at the left side and the right side of the automobile is calculated, the environment where the automobile is located is judged to be a wide space or a narrow space according to the distance, and the judgment accuracy can be ensured.
A method for processing a 360-degree panoramic video of an automobile, as shown in fig. 2, includes:
s1, collecting videos of the environment around the automobile close to the automobile, and generating a first video set;
s2, generating a three-dimensional simulation scene according to the video in the first video set, and observing the three-dimensional simulation scene to obtain information influencing the safe driving of the automobile;
and S3, providing safety prompt according to the information influencing the safe driving of the automobile.
The working principle of the technical scheme is as follows: the 360-degree panoramic video of the automobile can help a driver to better observe the environment around the automobile, acquire traffic information in the environment where the automobile is located, and provide help on the visual field for the driver to drive the automobile, so that the automobile is safely driven. The method comprises the steps of collecting videos of the environment around the automobile close to the body to generate a first video set; generating a three-dimensional simulation scene according to the video in the first video set, and acquiring information influencing the safe driving of the automobile by using the three-dimensional simulation scene; and providing a safety prompt according to the information influencing the safe driving of the automobile.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the driver can be helped to see the environment conditions around the automobile from a proper visual angle and a proper visual field by acquiring and processing the 360-degree panoramic video, and the driving safety is improved.
In one embodiment, S1 includes:
s101, arranging two adjacent radars on an automobile, selecting a certain moment, and respectively acquiring data of a first target at a first fixed position in the environment around the automobile by using the two radars to obtain first data and second data; analyzing the overlapping content of the first data and the second data, taking the overlapping content of the first data as third data, taking the overlapping content of the second data as fourth data, and preliminarily judging that the two radars work normally if the third data and the fourth data both correspond to the first target; performing data acquisition and analysis again after the two radars are shut down and restarted, and if the two radars are still judged to work normally, determining that the two radars work normally;
s102, selecting the same time as that in the step S101, and acquiring data of the first target located at the first fixed position in the environment around the automobile to be detected by using the camera to be detected to obtain fifth data; judging whether the corresponding targets in the fifth data and the first data or the second data are the first targets, if so, judging that the camera to be detected works normally; if not, performing correction operation on the camera to be detected;
s103, executing a correction operation on the camera, wherein the correction operation comprises the following steps: carrying out shutdown reset restart on the camera, or adjusting the angle of a camera lens to a preset normal working angle, or adjusting the focal length of the camera lens to a preset normal working focal length; after the correction operation is finished, the operation is transferred to the step S102 to continue to be checked until the camera works normally, and the operation of detecting and correcting the performance of the camera is finished;
s104, regularly acquiring videos of the surrounding environment of the close body of the automobile shot by a camera according to a preset period to obtain a plurality of video sets;
s105, checking videos in the plurality of video sets according to a preset video checking rule; the video inspection rule comprises: evaluating the image definition in the video and judging the area range corresponding to the video; when the image definition in the video is smaller than a preset image definition threshold value or the area range boundary corresponding to the video is larger than or smaller than a preset image boundary, judging that the video does not meet the requirement, and discarding the video; when the image definition in the video is greater than or equal to a preset image definition threshold value or the image boundary in the video is superposed with a preset image boundary, judging that the video is qualified in quality and reserving the video;
and S106, in the same period, taking the video reserved by the video inspection unit as a quality qualified video, and converging the quality qualified video to generate a first video set.
The working principle of the technical scheme is as follows: in order to ensure the quality of the obtained video images and the image splicing effect, the quality control inspection needs to be carried out on the video collected by the fisheye camera, when the video images to be spliced all accord with the preset quality qualified condition, the obtained video is reserved, the reserved video is integrated, and the video set meeting the requirements is obtained. Meanwhile, the working performance of the camera needs to be checked, and the mechanical problems are corrected and adjusted in time so as to ensure the normal and stable operation of the camera.
The specific steps of the embodiment include:
s101, arranging two adjacent radars on an automobile, selecting a certain moment, and respectively acquiring data of a first target at a first fixed position in the environment around the automobile by using the two radars to obtain first data and second data; analyzing the overlapping content of the first data and the second data, taking the overlapping content of the first data as third data, taking the overlapping content of the second data as fourth data, and preliminarily judging that the two radars work normally if the third data and the fourth data both correspond to the first target; performing data acquisition and analysis again after the two radars are shut down and restarted, and if the two radars are still judged to work normally, determining that the two radars work normally;
s102, selecting the same time as that in the step S101, and acquiring data of the first target located at the first fixed position in the environment around the automobile to be detected by using the camera to be detected to obtain fifth data; judging whether the corresponding targets in the fifth data and the first data or the second data are the first targets, if so, judging that the camera to be detected works normally; if not, performing correction operation on the camera to be detected;
s103, executing a correction operation on the camera, wherein the correction operation comprises the following steps: carrying out shutdown reset restart on the camera, or adjusting the angle of a camera lens to a preset normal working angle, or adjusting the focal length of the camera lens to a preset normal working focal length; after the correction operation is finished, the operation is transferred to the step S102 to continue to be checked until the camera works normally, and the operation of detecting and correcting the performance of the camera is finished;
s104, regularly acquiring videos of the surrounding environment of the automobile close to the body, which are shot by a camera, according to a preset period to obtain a plurality of video sets;
s105, checking videos in the plurality of video sets according to a preset video checking rule; the video inspection rule comprises: evaluating the image definition in the video and judging the area range corresponding to the video; when the image definition in the video is smaller than a preset image definition threshold value or the area range boundary corresponding to the video is larger than or smaller than a preset image boundary, judging that the video does not meet the requirement, and discarding the video; when the image definition in the video is greater than or equal to a preset image definition threshold value or the image boundary in the video is superposed with a preset image boundary, judging that the video is qualified in quality and reserving the video;
and S106, in the same period, taking the video reserved by the video inspection unit as a quality qualified video, and converging the quality qualified video to generate a first video set.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the normal working performance of the camera can be ensured by checking and correcting the working performance of the camera, so that the quality of the shot video is ensured; through carrying out quality control inspection on the video collected by the fisheye camera, the quality of the obtained video image can be ensured, and the image splicing effect is ensured.
In one embodiment, S2 includes constructing a three-dimensional simulation scene, specifically:
s201, calibrating internal parameters and external parameters of a camera; calculating to obtain a first texture mapping table and a second texture mapping table according to the internal parameters and the external parameters of the camera;
s202, calculating the distance between a camera in the first video set image and the main planes of the environments on the left side and the right side of the automobile; matching the distance with the first texture mapping table to obtain a first projection model parameter; matching the distance with the second texture mapping table to obtain a second projection model parameter;
s203, establishing a first projection model according to the first projection model parameters; establishing a second projection model according to the second projection model parameters;
s204, generating a first three-dimensional simulation scene according to the first projection model, wherein the first three-dimensional simulation scene corresponds to an automobile in an open space environment; and generating a second three-dimensional simulation scene according to the second projection model, wherein the second three-dimensional simulation scene corresponds to an automobile in a narrow space environment.
The working principle of the technical scheme is as follows: in order to better meet the panoramic view observation requirements of automobiles in different environments, corresponding three-dimensional simulation scenes need to be established for the different environments. In this embodiment, a first texture mapping table and a second texture mapping table are obtained by calculation according to an internal parameter and an external parameter of a camera calibrated in advance; the method comprises the steps of calculating the distance between a camera in a video image and main planes of the environment on the left side and the right side of an automobile according to different environments by analyzing actual environment information of the video image, obtaining corresponding projection model parameters through a texture mapping table, constructing a first projection model and a second projection model according to the projection model parameters, and completing construction of a three-dimensional simulation scene for an open space environment and a three-dimensional simulation scene for a narrow space environment according to the projection models.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the three-dimensional simulation panorama matched with the camera is constructed according to the different distances between the camera and the main planes of the environments at the left side and the right side of the automobile in the video image, the three-dimensional simulation panorama can be switched in a self-adaptive manner according to the different environments of the automobile, and a driver can conveniently observe the panoramic video of the environment around the automobile in a targeted manner.
In one embodiment, S202 includes determining whether the environment in which the vehicle is located is an open space or a narrow space; the method comprises the following specific steps:
s2021, carrying out optical flow tracking and matching on videos of the environment on the left side and the right side of the automobile, determining feature points on a main plane of the side-looking obstacle according to a change rule of the optical flow, and estimating relative depth values of the feature points according to epipolar geometry and triangulation;
s2022, transforming the side-view image of the rear-view camera to a top view, and calculating to obtain the inter-frame motion pose of the camera:
s2023, obtaining a scale factor according to a calibration result of the camera;
s2024, calculating to obtain the distance between the camera and the main planes of the left and right environments of the automobile according to the relative depth value, the inter-frame motion pose and the scale factor;
s2025, judging that the environment where the automobile is located is a wide space or a narrow space according to the distance and the preset distance threshold value; when the distance is smaller than a preset distance threshold value, judging that the automobile is in a narrow space environment; and when the distance is larger than a preset distance threshold value, judging that the automobile is in an open space environment.
The working principle of the technical scheme is as follows: in order to calculate the distance between the camera and the main planes of the environment on the left side and the right side of the automobile, the video is tracked to obtain feature points, and then matching is carried out, so that the relative motion of the camera is estimated; the optical flow can detect small motion changes of objects through a continuous image sequence, acquire image frames of videos on two sides, perform feature point tracking matching, and judge feature points on main planes of side-looking nearest obstacles (planes of objects on two sides, such as vehicle vertical planes of parking lots, green belts on driving roads and building planes nearest to roadsides) according to tracking results. In the feature point selection, firstly, an image processing interested area with larger feature point optical flow values and concentrated distribution on the same main plane is selected, the optical flow values of the feature points in the area are calculated, an optical flow value distribution histogram is calculated, a plurality of groups with larger horizontal component values and smaller vertical component values and larger quantity of optical flows are used as feature points to be selected, and then a group with the distribution span of pixel coordinate points meeting a preset threshold value is taken out from the feature points to be selected and used as the main plane feature points.
For corresponding feature points located on the principal plane, relative depth values are estimated from epipolar geometry and triangulation. Epipolar geometry is used for describing the geometric relationship between two images, and only needs the internal reference and the relative posture of a camera; epipolar geometry is commonly used in the aspects of camera matrix recovery, pose estimation, three-dimensional reconstruction and the like; calculating an essential matrix and a basic matrix according to the epipolar geometry, recovering a camera matrix through the essential matrix and decomposing to obtain an inter-frame motion pose; according to the corresponding relation between the size of the top view and the actual distance, the scale factor between the pixel distance of the top view and the actual distance is determined by the actual width of the top view area and the width of the top view; the distance between the camera and the main planes of the environment on the left side and the right side of the automobile can be calculated through the rotation matrix and the translational motion vector value in the interframe motion pose; judging whether the environment of the automobile is an open space or a narrow space according to the calculated distance and the preset distance threshold value; when the distance is smaller than a preset distance threshold value, judging that the automobile is in a narrow space environment; and when the distance is larger than the preset distance threshold value, judging that the automobile is in the open space environment.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the distance between the camera and the main planes of the environments at the left side and the right side of the automobile is calculated, the environment where the automobile is located is judged to be a wide space or a narrow space according to the distance, and the judgment accuracy can be ensured.
In one embodiment, as shown in fig. 3, the step of providing the safety prompt in S3 includes the following specific steps:
s301, acquiring first distance data between a camera and main planes of environments on the left side and the right side of an automobile, second distance data between the camera and the main planes of the environments on the front side and the rear side of the automobile and third distance data between a plane of the top of the automobile and the main planes of the environments outside the automobile in real time, and storing the first distance data and the second distance data into a preset database;
s302, constructing a collision risk analysis model based on an artificial neural network according to historical first interval data, historical second interval data and historical third interval data in the database; the collision risk analysis model inputs data as interval data and outputs data as a collision risk value;
s303, according to a collision risk analysis model, collision risk values of the first interval data, the second interval data and the third interval data are respectively determined, and a first collision risk value, a second collision risk value and a third collision risk value are obtained; sending out corresponding safety prompts according to the analysis results of the first collision risk value, the second collision risk value and the third collision risk value;
judging whether the first collision risk value is greater than or equal to a preset first collision risk threshold value or not, and if so, sending a first safety prompt to an automobile driver;
judging whether the second collision risk value is greater than or equal to a preset second collision risk threshold value or not, and if so, sending a second safety prompt to the automobile driver;
judging whether the third collision risk value is greater than or equal to a preset third collision risk threshold value, and if so, sending a third safety prompt to the automobile driver;
and S304, the automobile driver takes corresponding risk avoiding driving operation according to the first safety prompt, the second safety prompt or the third safety prompt.
The working principle of the technical scheme is as follows: in order to better provide safety prompts for drivers, it is necessary to distinguish different safety risks and provide corresponding prompts, so as to help the drivers to take corresponding risk avoidance driving operations to ensure that the automobiles are safely driven. The method comprises the following specific steps:
acquiring first distance data between a camera and main planes of the environment on the left side and the right side of the automobile, second distance data between the camera and the main planes of the environment behind the front part of the automobile and third distance data between the plane of the top of the automobile and the main plane of the environment outside the automobile in real time, and storing the first distance data and the second distance data into a preset database;
constructing a collision risk analysis model based on an artificial neural network according to historical first interval data, historical second interval data and historical third interval data in the database; the collision risk analysis model inputs data as interval data and outputs data as a collision risk value;
according to a collision risk analysis model, collision risk values of the first interval data, the second interval data and the third interval data are respectively determined, and a first collision risk value, a second collision risk value and a third collision risk value are obtained; sending out corresponding safety prompts according to the analysis results of the first collision risk value, the second collision risk value and the third collision risk value;
judging whether the first collision risk value is greater than or equal to a preset first collision risk threshold value or not, and if so, sending a first safety prompt to an automobile driver;
judging whether the second collision risk value is greater than or equal to a preset second collision risk threshold value or not, and if so, sending a second safety prompt to the automobile driver;
judging whether the third collision risk value is greater than or equal to a preset third collision risk threshold value, and if so, sending a third safety prompt to the automobile driver;
and the automobile driver takes corresponding risk avoiding driving operation according to the first safety prompt, the second safety prompt or the third safety prompt.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the possible collision risk is analyzed by analyzing the distance between the automobile and the main plane of the environment where the automobile is located according to the difference of the distances, so that safety prompts can be provided in a targeted manner, a driver is helped to avoid the collision risk in time, and safe driving is ensured.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims (10)

1.一种汽车360度全景视频处理系统，其特征在于，包括：1. a car 360 degree panoramic video processing system, is characterized in that, comprises: 全景视频采集模块，用于采集汽车近身四周环境的视频，生成第一视频集；The panoramic video acquisition module is used to collect the video of the surrounding environment of the car to generate the first video set; 全景视频处理模块，用于根据所述第一视频集中的视频生成三维仿真场景，利用三维仿真场景，获得影响汽车安全行驶的信息；a panoramic video processing module, configured to generate a three-dimensional simulation scene according to the videos in the first video set, and obtain the information affecting the safe driving of the vehicle by using the three-dimensional simulation scene; 全景安全提示模块，用于根据影响汽车安全行驶的信息，提供安全提示。The panoramic safety prompt module is used to provide safety prompts according to the information that affects the safe driving of the car. 2.根据权利要求1所述的一种汽车360度全景视频处理系统，其特征在于，所述全景安全提示模块，包括主显示屏和副显示屏，所述主显示屏嵌于汽车内中控台，用于显示车辆近身四周360度全景视频鸟瞰图；所述副显示屏设置于驾驶员视线正前方位置，用于显示以驾驶员视角方向为中心左右20度范围内的汽车近身四周鸟瞰图；2. An automobile 360-degree panoramic video processing system according to claim 1, wherein the panoramic safety prompt module comprises a main display screen and a secondary display screen, and the main display screen is embedded in a central control panel in the car The display screen is used to display a 360-degree panoramic video bird's-eye view around the vehicle; the secondary display screen is arranged in front of the driver's line of sight, and is used to display the surrounding of the car within 20 degrees of the driver's viewing angle. Aerial View; 所述汽车近身四周鸟瞰图基于图像重投影程序，结合驾驶员头部运动角度信息，对车辆近身360度全景视频进行投影映射获得；The bird's-eye view around the vehicle is obtained by performing projection mapping on the vehicle's close-up 360-degree panoramic video based on the image re-projection program, combined with the driver's head movement angle information; 当汽车所处环境为开阔空间环境时，仅通过所述副显示屏显示汽车近身四周鸟瞰图；当汽车所处环境为狭窄空间环境时，在通过所述副显示屏显示汽车近身四周鸟瞰图的同时，开启所述主显示屏显示车辆近身四周360度全景视频鸟瞰图。When the environment where the car is located is an open space environment, only the bird's-eye view around the car is displayed through the auxiliary display screen; when the environment where the car is located is a narrow space environment, the bird's-eye view around the car is displayed through the auxiliary display screen. At the same time, the main display screen is turned on to display a 360-degree panoramic video bird's-eye view around the vehicle. 3.根据权利要求1所述的一种汽车360度全景视频处理系统，其特征在于，所述全景视频采集模块，利用若干台180度超广角鱼眼摄像机进行视频采集；所述超广角鱼眼摄像机按照预设的角度置于车体前后左右，所述预设角度根据鱼眼摄像机所拍摄的区域调节设定；3. a kind of automobile 360 degree panoramic video processing system according to claim 1, is characterized in that, described panoramic video acquisition module, utilizes several 180 degree super wide angle fisheye cameras to carry out video collection; Described super wide angle fisheye The cameras are placed at the front, back, left and right sides of the vehicle body according to a preset angle, and the preset angle is adjusted and set according to the area captured by the fisheye camera; 所述全景视频采集模块包括视频初选单元、视频检验单元和视频集生成单元；The panoramic video acquisition module includes a video primary selection unit, a video inspection unit and a video set generation unit; 所述视频初选单元，用于按照预设的周期，定期获取摄像机拍摄的汽车近身四周环境的视频，获得若干个视频集；The video primary selection unit is used for regularly obtaining the video of the surrounding environment of the car shot by the camera according to a preset period, and obtaining several video sets; 所述视频检验单元，用于根据预设的视频检验规则对所述若干个视频集中的视频进行检验；所述视频检验规则包括：对视频中图像清晰度进行评估、对视频中图像的边界进行判定；当视频中图像清晰度小于预设的图像清晰度阈值，或者视频中图像的边界大于或小于预设的图像边界时，则判定视频不符合要求，将该视频舍弃；当视频中图像清晰度大于等于预设的图像清晰度阈值，或视频中图像的边界与预设的图像边界重合时，则判定视频符合要求，将该视频保留；The video inspection unit is configured to inspect the videos in the several video sets according to preset video inspection rules; the video inspection rules include: evaluating the clarity of the images in the videos, and inspecting the boundaries of the images in the videos. Judgment; when the image definition in the video is less than the preset image definition threshold, or the boundary of the image in the video is larger or smaller than the preset image boundary, it is determined that the video does not meet the requirements, and the video is discarded; when the image in the video is clear If the degree is greater than or equal to the preset image definition threshold, or when the border of the image in the video coincides with the preset image border, it is determined that the video meets the requirements and the video is retained; 所述视频集生成单元，用于生成视频集；在同一周期内，将所述视频检验单元保留的视频作为质量合格视频，将所述质量合格视频汇合，生成第一视频集。The video set generating unit is configured to generate a video set; in the same period, the videos retained by the video inspection unit are regarded as quality qualified videos, and the quality qualified videos are merged to generate a first video set. 4.根据权利要求1所述的一种汽车360度全景视频处理系统，其特征在于，所述全景视频处理模块，包括三维仿真场景构建单元；所述三维仿真场景构建单元包括摄像机标定子单元、模型参数获取子单元、投影模型构建子单元和三维仿真场景生成子单元；4. a kind of automobile 360 degree panoramic video processing system according to claim 1, is characterized in that, described panoramic video processing module, comprises three-dimensional simulation scene construction unit; Described three-dimensional simulation scene construction unit comprises camera calibration subunit, Model parameter acquisition subunit, projection model construction subunit and 3D simulation scene generation subunit; 所述摄像机标定子单元，用于标定摄像机的内参数和外参数；并根据所述摄像机的内参数和外参数，计算得到第一纹理映射表和第二纹理映射表；The camera calibration subunit is used to calibrate the internal parameters and external parameters of the camera; and according to the internal parameters and external parameters of the camera, the first texture mapping table and the second texture mapping table are obtained by calculation; 所述模型参数获取子单元，用于计算所述第一视频集图像中的摄像机与汽车左右两侧环境主平面的间距；将所述间距与所述第一纹理映射表进行匹配，得到第一投影模型参数；将所述间距与所述第二纹理映射表进行匹配，得到第二投影模型参数；The model parameter acquisition subunit is used to calculate the distance between the camera in the first video set and the main plane of the environment on the left and right sides of the car; match the distance with the first texture mapping table to obtain the first Projection model parameters; match the spacing with the second texture mapping table to obtain second projection model parameters; 投影模型构建子单元，用于根据所述第一投影模型参数，建立第一投影模型；根据所述第二投影模型参数，建立第二投影模型；a projection model construction subunit, configured to establish a first projection model according to the first projection model parameters; establish a second projection model according to the second projection model parameters; 所述三维全景图生成子单元，用于根据所述第一投影模型，生成第一三维仿真场景，所述第一三维仿真场景对应为汽车处于开阔空间环境；根据所述第二投影模型，生成第二三维仿真场景，所述第二三维仿真场景对应为汽车处于狭窄空间环境。The three-dimensional panorama image generating subunit is configured to generate a first three-dimensional simulation scene according to the first projection model, and the first three-dimensional simulation scene corresponds to a car in an open space environment; according to the second projection model, generate a The second three-dimensional simulation scene corresponds to that the car is in a narrow space environment. 5.根据权利要求3所述的一种汽车360度全景视频处理系统，其特征在于，所述模型参数获取子单元，包括空间环境判断分子单元，用于判断汽车所处的环境为开阔空间或狭窄空间；具体为：5. a kind of car 360 degree panoramic video processing system according to claim 3, is characterized in that, described model parameter acquisition subunit, comprises space environment judgment molecular unit, is used for judging that the environment where the car is located is open space or narrow spaces; specifically: 对汽车左右两侧环境的视频进行光流跟踪和匹配，根据光流的变化规律确定侧视障碍物主平面上的特征点，根据对极几何和三角测量，估计出特征点的相对深度值；Perform optical flow tracking and matching on the video of the environment on the left and right sides of the car, determine the feature points on the main plane of the side-view obstacle according to the change law of the optical flow, and estimate the relative depth value of the feature points according to the epipolar geometry and triangulation; 将后视摄像机的侧视图像变换到顶视图，计算得到摄像机的帧间运动位姿：Transform the side-view image of the rear-view camera to the top view, and calculate the inter-frame motion pose of the camera: 根据摄像机的标定结果，获得比例因子；According to the calibration result of the camera, the scale factor is obtained; 根据所述相对深度值、帧间运动位姿和比例因子，计算得到摄像机与汽车左右两侧环境主平面的间距；According to the relative depth value, the motion pose and scale factor between frames, the distance between the camera and the main plane of the environment on the left and right sides of the car is calculated; 根据所述间距与预设间距阈值的大小判断汽车所处的环境为开阔空间或狭窄空间；当所述间距小于预设间距阈值时，判定汽车处于狭窄空间环境；当所述间距大于预设间距阈值时，判定汽车处于开阔空间环境。According to the size of the distance and the preset distance threshold, it is determined that the environment where the car is located is an open space or a narrow space; when the distance is less than the preset distance threshold, it is determined that the car is in a narrow space environment; when the distance is greater than the preset distance When the threshold is set, it is determined that the car is in an open space environment. 6.一种汽车360度全景视频处理方法，其特征在于，包括：6. A method for processing a 360-degree panoramic video of an automobile, comprising: S1、采集汽车近身四周环境的视频，生成第一视频集；S1. Collect videos of the surrounding environment of the car to generate a first video set; S2、根据所述第一视频集中的视频生成三维仿真场景，通过观察三维仿真场景，获得影响汽车安全行驶的信息；S2. Generate a three-dimensional simulation scene according to the video in the first video set, and obtain information that affects the safe driving of the vehicle by observing the three-dimensional simulation scene; S3、根据影响汽车安全行驶的信息，提供安全提示。S3. Provide safety tips according to the information that affects the safe driving of the car. 7.根据权利要求6所述的一种汽车360度全景视频处理方法，其特征在于，S1包括7. A kind of automobile 360 degree panoramic video processing method according to claim 6, is characterized in that, S1 comprises S101、在汽车上设置相邻的两个雷达，选定某一时刻，采用两个雷达对位于汽车近身四周环境内第一固定位置的第一目标分别进行数据采集，获得第一数据和第二数据；分析所述第一数据和第二数据的重叠内容，将所述第一数据的重叠内容作为第三数据，将第二数据的重叠内容作为第四数据，若所述第三数据和第四数据均对应于所述第一目标，则初步判断两个雷达工作正常；将两个雷达进行关机重启后再次进行数据采集分析，若仍判断为工作正常，则确认两个雷达工作正常；S101. Set up two adjacent radars on the car, select a certain moment, and use the two radars to collect data on the first target located at the first fixed position in the surrounding environment of the car, respectively, to obtain the first data and the first target. Second data; analyze the overlapping content of the first data and the second data, take the overlapping content of the first data as the third data, and take the overlapping content of the second data as the fourth data, if the third data and If the fourth data all correspond to the first target, it is preliminarily judged that the two radars are working normally; after the two radars are shut down and restarted, data collection and analysis are performed again, and if it is still judged that they are working normally, it is confirmed that the two radars are working normally; S102、选定与步骤S101中的相同时刻，对所述位于汽车近身四周环境内第一固定位置的第一目标，利用待检测摄像机进行数据采集，获得第五数据；判断所述第五数据与所述第一数据或第二数据中所对应的目标是否同为第一目标，若是，则判定所述待检测摄像机工作正常；若否，则对所述待检测摄像机进行矫正操作；S102, select the same time as in step S101, use the camera to be detected to collect data for the first target located at the first fixed position in the surrounding environment of the car to obtain fifth data; determine the fifth data Whether the target corresponding to the first data or the second data is the same as the first target, if so, it is determined that the camera to be detected is working normally; if not, a correction operation is performed on the camera to be detected; S103、执行对摄像机的矫正操作，所述矫正操作包括：对摄像机进行关机复位重启、或将摄像机镜头角度调整到预设的正常工作角度、或将摄像机镜头焦距调整到预设的正常工作焦距；矫正操作结束后，转至步骤S102继续进行检验，直到判定摄像机工作正常，完成对摄像机性能进行检测和矫正的操作；S103, performing a correction operation on the camera, where the correction operation includes: shutting down, resetting and restarting the camera, or adjusting the camera lens angle to a preset normal working angle, or adjusting the camera lens focal length to a preset normal working focal length; After the correction operation is completed, go to step S102 to continue the inspection until it is determined that the camera works normally, and the operation of detecting and correcting the performance of the camera is completed; S104、按照预设的周期，定期获取摄像机拍摄的汽车近身四周环境的视频，获得若干个视频集；S104, according to a preset period, regularly acquire the video of the surrounding environment of the car shot by the camera, and acquire several video sets; S105、根据预设的视频检验规则对所述若干个视频集中的视频进行检验；所述视频检验规则包括：对视频中图像清晰度进行评估、对视频对应的区域范围进行判定；当视频中图像清晰度小于预设的图像清晰度阈值，或者视频对应的区域范围边界大于或小于预设的图像边界时，则判定视频不符合要求，将该视频舍弃；当视频中图像清晰度大于等于预设的图像清晰度阈值，或视频中图像的边界与预设的图像边界重合时，则判定视频质量合格，将该视频保留；S105. Check the videos in the several video sets according to preset video checking rules; the video checking rules include: evaluating the clarity of the images in the videos, and judging the area corresponding to the videos; When the definition is less than the preset image definition threshold, or the boundary of the region corresponding to the video is larger or smaller than the preset image boundary, it is determined that the video does not meet the requirements and the video is discarded; when the image definition in the video is greater than or equal to the preset If the image definition threshold is set, or the border of the image in the video coincides with the preset image border, the video quality is judged to be qualified, and the video is retained; S106、在同一周期内，将所述视频检验单元保留的视频作为质量合格视频，将所述质量合格视频汇合，生成第一视频集。S106. In the same cycle, take the videos retained by the video inspection unit as quality qualified videos, and combine the quality qualified videos to generate a first video set. 8.根据权利要求6所述的一种汽车360度全景视频处理方法，其特征在于，S2包括构建三维仿真场景，具体为：8. a kind of automobile 360 degree panoramic video processing method according to claim 6, is characterized in that, S2 comprises constructing three-dimensional simulation scene, is specifically: S201、标定摄像机的内参数和外参数；并根据所述摄像机的内参数和外参数，计算得到第一纹理映射表和第二纹理映射表；S201, calibrate the internal parameters and external parameters of the camera; and calculate and obtain a first texture mapping table and a second texture mapping table according to the internal parameters and external parameters of the camera; S202、计算所述第一视频集图像中的摄像机与汽车左右两侧环境主平面的间距；将所述间距与所述第一纹理映射表进行匹配，得到第一投影模型参数；将所述间距与所述第二纹理映射表进行匹配，得到第二投影模型参数；S202: Calculate the distance between the camera in the first video set and the main plane of the environment on the left and right sides of the car; match the distance with the first texture mapping table to obtain first projection model parameters; calculate the distance Matching with the second texture mapping table to obtain the second projection model parameter; S203、根据所述第一投影模型参数，建立第一投影模型；根据所述第二投影模型参数，建立第二投影模型；S203, establishing a first projection model according to the first projection model parameters; establishing a second projection model according to the second projection model parameters; S204、根据所述第一投影模型，生成第一三维仿真场景，所述第一三维仿真场景对应为汽车处于开阔空间环境；根据所述第二投影模型，生成第二三维仿真场景，所述第二三维仿真场景对应为汽车处于狭窄空间环境。S204. Generate a first 3D simulation scene according to the first projection model, where the first 3D simulation scene corresponds to that the car is in an open space environment; according to the second projection model, generate a second 3D simulation scene, the first 3D simulation scene is The 2D and 3D simulation scene corresponds to the car in a narrow space environment. 9.根据权利要求8所述的一种汽车360度全景视频处理方法，其特征在于，S202包括判断汽车所处的环境为开阔空间或狭窄空间，具体步骤为：9. a kind of automobile 360 degree panoramic video processing method according to claim 8, is characterized in that, S202 comprises judging that the environment where the automobile is located is open space or narrow space, and concrete steps are: S2021、对汽车左右两侧环境的视频进行光流跟踪和匹配，根据光流的变化规律确定侧视障碍物主平面上的特征点，根据对极几何和三角测量，估计出特征点的相对深度值；S2021. Perform optical flow tracking and matching on the video of the environment on the left and right sides of the car, determine the feature points on the main plane of the side-view obstacle according to the change rule of the optical flow, and estimate the relative depth of the feature points according to epipolar geometry and triangulation value; S2022、将后视摄像机的侧视图像变换到顶视图，计算得到摄像机的帧间运动位姿：S2022, transform the side view image of the rear-view camera to the top view, and calculate the motion pose of the camera between frames: S2023、根据摄像机的标定结果，获得比例因子；S2023, obtaining a scale factor according to the calibration result of the camera; S2024、根据所述相对深度值、帧间运动位姿和比例因子，计算得到摄像机与汽车左右两侧环境主平面的间距；S2024, calculating the distance between the camera and the main plane of the environment on the left and right sides of the car according to the relative depth value, the motion pose and scale factor between frames; S2025、根据所述间距与预设间距阈值的大小判断汽车所处的环境为开阔空间或狭窄空间；当所述间距小于预设间距阈值时，判定汽车处于狭窄空间环境；当所述间距大于预设间距阈值时，判定汽车处于开阔空间环境。S2025. Determine whether the environment where the car is located is an open space or a narrow space according to the distance and a preset distance threshold; when the distance is less than a preset distance threshold, determine that the car is in a narrow space environment; when the distance is greater than a predetermined distance When the distance threshold is set, it is determined that the car is in an open space environment. 10.根据权利要求6所述的一种汽车360度全景视频处理方法，其特征在于，S3中包括提供安全提示，具体步骤为：10. The method for processing a 360-degree panoramic video of an automobile according to claim 6, wherein S3 includes providing a safety prompt, and the specific steps are: S301、实时获取摄像机与汽车左右两侧环境主平面的第一间距数据，摄像机与汽车前方后方环境主平面的第二间距数据，以及汽车车顶平面与汽车外部环境主平面的第三间距数据，并存储到预设的数据库中；S301. Acquire in real time the first distance data between the camera and the main plane of the environment on the left and right sides of the car, the second distance data between the camera and the main plane of the environment in front of and behind the car, and the third distance data between the roof plane of the car and the main plane of the external environment of the car, and stored in the preset database; S302、根据所述数据库中的历史第一间距数据、历史第二间距数据和历史第三间距数据，基于人工神经网络构建磕碰风险分析模型；所述磕碰风险分析模型输入数据为间距数据，输出数据为磕碰风险值；S302. According to the historical first spacing data, historical second spacing data and historical third spacing data in the database, build a collision risk analysis model based on an artificial neural network; the input data of the collision risk analysis model is spacing data, and output data is the collision risk value; S303、根据磕碰风险分析模型，分别确定所述第一间距数据、第二间距数据和第三间距数据的磕碰风险值，获得第一磕碰风险值、第二磕碰风险值和第三磕碰风险值；根据对所述第一磕碰风险值、第二磕碰风险值和第三磕碰风险值的分析结果，发出相应的安全提示；S303, according to the collision risk analysis model, respectively determine the collision risk value of the first distance data, the second distance data and the third distance data, and obtain the first collision risk value, the second collision risk value and the third collision risk value; According to the analysis results of the first collision risk value, the second collision risk value and the third collision risk value, a corresponding safety prompt is issued; 判断所述第一磕碰风险值是否大于等于预设的第一磕碰风险阈值，若是则向汽车驾驶员发出第一安全提示；Judging whether the first collision risk value is greater than or equal to a preset first collision risk threshold, and if so, issuing a first safety prompt to the driver; 判断所述第二磕碰风险值是否大于等于预设的第二磕碰风险阈值，若是则向汽车驾驶员发出第二安全提示；Determine whether the second collision risk value is greater than or equal to a preset second collision risk threshold, and if so, issue a second safety prompt to the driver; 判断所述第三磕碰风险值是否大于等于预设的第三磕碰风险阈值，若是则向汽车驾驶员发出第三安全提示；Determine whether the third collision risk value is greater than or equal to a preset third collision risk threshold, and if so, issue a third safety prompt to the driver; S304、汽车驾驶员根据所述第一安全提示、或第二安全提示、或第三安全提示，采取相应的避险驾驶操作。S304, the driver of the vehicle takes a corresponding risk-avoiding driving operation according to the first safety prompt, the second safety prompt, or the third safety prompt.
CN202210230400.6A 2022-03-10 2022-03-10 Automobile 360-degree panoramic video processing system and method Pending CN114567758A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210230400.6A CN114567758A (en) 2022-03-10 2022-03-10 Automobile 360-degree panoramic video processing system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210230400.6A CN114567758A (en) 2022-03-10 2022-03-10 Automobile 360-degree panoramic video processing system and method

Publications (1)

Publication Number Publication Date
CN114567758A true CN114567758A (en) 2022-05-31

Family

ID=81717024

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210230400.6A Pending CN114567758A (en) 2022-03-10 2022-03-10 Automobile 360-degree panoramic video processing system and method

Country Status (1)

Country Link
CN (1) CN114567758A (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030151563A1 (en) * 2002-02-08 2003-08-14 Kulas Charles J. Reduction of blind spots by using display screens
CN102791509A (en) * 2010-03-11 2012-11-21 丰田自动车株式会社 Image position adjustment device
CN103108152A (en) * 2011-11-09 2013-05-15 能晶科技股份有限公司 Blind Spot Detection Warning System
CN104590120A (en) * 2014-12-02 2015-05-06 重庆交通大学 Automobile self-adaptive image assistance passing method and system
CN105172701A (en) * 2015-09-16 2015-12-23 浙江吉利汽车研究院有限公司 Control method of vehicle panorama image system
CN106226910A (en) * 2016-09-08 2016-12-14 邹文韬 HUD system and image regulating method thereof
CN108508872A (en) * 2018-04-18 2018-09-07 鄂尔多斯市普渡科技有限公司 A kind of fault detection method of pilotless automobile information acquisition system
CN111462172A (en) * 2020-02-24 2020-07-28 西安电子科技大学 Three-dimensional panoramic image self-adaptive generation method based on driving scene estimation

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030151563A1 (en) * 2002-02-08 2003-08-14 Kulas Charles J. Reduction of blind spots by using display screens
CN102791509A (en) * 2010-03-11 2012-11-21 丰田自动车株式会社 Image position adjustment device
CN103108152A (en) * 2011-11-09 2013-05-15 能晶科技股份有限公司 Blind Spot Detection Warning System
CN104590120A (en) * 2014-12-02 2015-05-06 重庆交通大学 Automobile self-adaptive image assistance passing method and system
CN105172701A (en) * 2015-09-16 2015-12-23 浙江吉利汽车研究院有限公司 Control method of vehicle panorama image system
CN106226910A (en) * 2016-09-08 2016-12-14 邹文韬 HUD system and image regulating method thereof
CN108508872A (en) * 2018-04-18 2018-09-07 鄂尔多斯市普渡科技有限公司 A kind of fault detection method of pilotless automobile information acquisition system
CN111462172A (en) * 2020-02-24 2020-07-28 西安电子科技大学 Three-dimensional panoramic image self-adaptive generation method based on driving scene estimation

Similar Documents

Publication Publication Date Title
US9858639B2 (en) Imaging surface modeling for camera modeling and virtual view synthesis
CN103617606B (en) For assisting the vehicle multi-angle panorama generation method of driving
JP2022095776A (en) Rear-stitched view panorama for rear-view visualization
CN107856667B (en) Parking assist system and method
CN110910453B (en) Vehicle pose estimation method and system based on non-overlapping view field multi-camera system
US8872920B2 (en) Camera calibration apparatus
CN110796711B (en) Panoramic system calibration method and device, computer readable storage medium and vehicle
CN111768332B (en) Method for splicing vehicle-mounted panoramic real-time 3D panoramic images and image acquisition device
US20090167856A1 (en) Reverse sensing system displaying overhead vehicle image
JP2012075060A (en) Image processing device, and imaging device using the same
JP2006053890A (en) Obstacle detection apparatus and method
CN102368137B (en) Embedded calibrating stereoscopic vision system
CN109883433B (en) Vehicle localization method in structured environment based on 360-degree panoramic view
Li et al. Easy calibration of a blind-spot-free fisheye camera system using a scene of a parking space
CN113834463B (en) Intelligent vehicle side pedestrian/vehicle monocular depth ranging method based on absolute size
EP3629292A1 (en) Reference point selection for extrinsic parameter calibration
JP2010028803A (en) Image displaying method for parking aid
Li et al. VAVM: a flexible technique for variable-angle around view monitor system towards articulated engineering vehicle
CA3122865A1 (en) Method for detecting and modeling of object on surface of road
CN115546761B (en) A Deep Learning-Based Parking Space and Obstacle Detection Method and Its Application
GB2513703A (en) Method and apparatus for three-dimensional imaging of at least a partial region of a vehicle environment
CN114567758A (en) Automobile 360-degree panoramic video processing system and method
CN119228904A (en) A 360-degree vehicle-mounted surround view camera calibration method and device based on ground texture features
CN117710517A (en) Parking track line drawing method, system and vehicle
TW202305405A (en) Method and system for detecting and analyzing objects

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20220531

RJ01 Rejection of invention patent application after publication