CN119904841A - Image processing method, device, electronic equipment and vehicle - Google Patents
Image processing method, device, electronic equipment and vehicle Download PDFInfo
- Publication number
- CN119904841A CN119904841A CN202411988235.6A CN202411988235A CN119904841A CN 119904841 A CN119904841 A CN 119904841A CN 202411988235 A CN202411988235 A CN 202411988235A CN 119904841 A CN119904841 A CN 119904841A
- Authority
- CN
- China
- Prior art keywords
- image
- vehicle
- panoramic image
- environment
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three-dimensional [3D] modelling for computer graphics
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Traffic Control Systems (AREA)
Abstract
本申请提供一种图像处理方法、装置、电子设备及车辆，所述方法包括：采集当前车辆所处环境的图像，得到真实环境图像；识别所述真实环境图像中的环境要素的类型，得到至少一个类型标签；从外置数据库中分别获取与每个所述类型标签相对应的场景图像，得到多张场景元素图像；基于多张场景元素图像构建得到模拟当前车辆周边环境的三维全景图像。本申请公开的图像处理方法、装置、电子设备及车辆，能构建得到模拟当前车辆周边环境的三维全景图像，避免驾驶人员停车、下车观察环境，有效提高驾驶安全性与驾驶体验。
The present application provides an image processing method, device, electronic device and vehicle, the method comprising: collecting an image of the environment in which the current vehicle is located to obtain a real environment image; identifying the type of environmental elements in the real environment image to obtain at least one type label; acquiring scene images corresponding to each type label from an external database to obtain multiple scene element images; constructing a three-dimensional panoramic image simulating the current vehicle's surrounding environment based on the multiple scene element images. The image processing method, device, electronic device and vehicle disclosed in the present application can construct a three-dimensional panoramic image simulating the current vehicle's surrounding environment, avoiding the driver from stopping or getting off the vehicle to observe the environment, and effectively improving driving safety and driving experience.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing method, an image processing device, an electronic device, and a vehicle.
Background
During the running of the vehicle, the driver is located in the vehicle, and the visible line is limited, and sometimes has to get off to observe the surrounding environment better. However, the time for parking and getting off the vehicle on the road is limited, so that the vehicle cannot be parked at any time to view the environment, and the driving safety and the driving experience are affected.
Disclosure of Invention
In view of the above, the present application is directed to an image processing method, an image processing device, an electronic device and a vehicle, so as to improve driving experience and driving safety.
Based on the above object, the present application provides an image processing method comprising:
acquiring an image of the environment where the current vehicle is located, and obtaining a real environment image;
identifying the type of the environmental element in the real environmental image to obtain at least one type tag;
respectively acquiring scene images corresponding to each type tag from an external database to obtain a plurality of scene element images;
and constructing a three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on the plurality of scene element images.
Optionally, the constructing a three-dimensional panoramic image simulating the current vehicle surrounding environment based on the plurality of scene element images includes:
Splicing the plurality of scene element images into a planar panoramic image by using an image splicing technology;
And mapping the planar panoramic image into a spherical panoramic image by utilizing a three-dimensional engine technology to obtain the three-dimensional panoramic image.
Optionally, before the mapping the planar panoramic image into the spherical panoramic image by using the three-dimensional engine technology, the method further comprises:
And removing objects below a preset height threshold from the ground in the planar panoramic image by using an image removing technology.
Optionally, the method further comprises:
acquiring three-dimensional model data of a current vehicle;
and fusing the three-dimensional model data of the current vehicle with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing the three-dimensional model of the current vehicle.
Optionally, the real environment image includes a vehicle forward environment image, and the method further includes:
and generating and sending prompt information in response to the type tag corresponding to the vehicle forward environment image accords with a preset tag range which influences driving safety.
Optionally, the generating and sending out the prompt information includes:
And generating corresponding early warning information and/or driving advice according to the type tag, and fusing the early warning information and/or the driving advice with the three-dimensional panoramic image to obtain the three-dimensional panoramic image containing the front environment prompt information.
Optionally, the method further comprises:
Acquiring navigation information of a current vehicle;
generating driving direction guide information based on the navigation information and the real environment image;
and fusing the driving direction guide information with the three-dimensional panoramic image to obtain the three-dimensional panoramic image containing driving direction guide.
The second aspect of the present application also provides an image processing apparatus including:
The acquisition module is used for acquiring an image of the environment where the current vehicle is located to obtain a real environment image;
The element identification module is used for identifying the type of the environment element in the real environment image and obtaining at least one type tag;
The image acquisition module is used for respectively acquiring scene images corresponding to each type tag from an external database to obtain a plurality of scene element images;
the generation module is used for constructing and obtaining a three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on the plurality of scene element images.
A third aspect of the application also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, the processor implementing the method as described above when executing the computer program.
A fourth aspect of the application also provides a vehicle comprising an electronic device as described above.
According to the image processing method, the device, the electronic equipment and the vehicle, the real environment image is acquired firstly, the environment element types in the real environment image are identified, the corresponding type labels are obtained, the type labels can represent the real environment states, the scene element images obtained from the external database by the type labels can accurately represent the real environment conditions, finally, the three-dimensional panoramic image is constructed according to the scene element images, the current surrounding environment conditions of the vehicle can be effectively simulated, the real image of the environment where the current vehicle is located can be observed through the three-dimensional panoramic image in the running process of the vehicle, no parking or getting off observation is needed, driving safety is high, and driving experience is good. Meanwhile, the scene images are expanded by using the type labels and the external database, a large number of real environment images are not required to be acquired for analysis, the processing efficiency and the computing efficiency are high, the scene element images are stored in the external database, the limitation on the storage space of the database is small, different scene images can be stored as much as possible, the application range is wider, and the computing efficiency is higher.
Drawings
In order to more clearly illustrate the technical solutions of the present application or related art, the drawings that are required to be used in the description of the embodiments or related art will be briefly described below, and it is apparent that the drawings in the following description are only embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort to those of ordinary skill in the art.
FIG. 1 is a flow chart of an image processing method according to an embodiment of the application;
fig. 2 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;
fig. 3 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
The present application will be further described in detail below with reference to specific embodiments and with reference to the accompanying drawings, in order to make the objects, technical solutions and advantages of the present application more apparent.
It should be noted that unless otherwise defined, technical or scientific terms used in the embodiments of the present application should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present application belongs. The terms "first," "second," and the like, as used in embodiments of the present application, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.
As described in the background art, in order to ensure driving safety during running of a vehicle, a driver needs to constantly observe the surrounding environment of the vehicle and constantly adjust driving parameters according to the surrounding environment of the vehicle. For example, when there are water accumulation, ice formation, etc. on the road surface of the road, it is necessary to reduce the vehicle speed at this time, when the road is sandy, it is necessary to reduce the drive gear at this time, and when the surrounding environment is city, the vehicle flow is dense, it is necessary to reduce the vehicle speed at this time, etc. Therefore, the surrounding environment of the vehicle is closely related to the driving parameters and the driving safety. However, the driver sits in the vehicle, the visual field is limited, the observable range is limited, the observation accuracy is poor, and when the driver needs to accurately observe the surrounding environment of the vehicle, the driver has to stop and get off the vehicle for observation, so that the driving safety is affected, and the driving experience is reduced. In addition, with the development of intelligent driving technology, the requirements of people on driving experience are gradually increased, and technologies such as intelligent cabins and intelligent driving are increasingly applied. However, in the existing intelligent cabin and intelligent driving technology, a driver and a rider cannot accurately know the external real environment, and driving experience is poor.
In view of this, the present application provides an image processing method, capable of constructing and obtaining a three-dimensional panoramic image simulating the surrounding environment of the current vehicle, so that the driver and the passengers can observe the real situation outside the vehicle without getting off, the driving safety is high, and the driving experience is good, as shown in fig. 1, the method includes:
S101, acquiring an image of the environment where a current vehicle is located, and obtaining a real environment image;
Specifically, an image of the environment in which the current vehicle is located may be acquired by using an image capturing device (such as a vehicle recorder or the like) on the vehicle. When the image of the environment where the current vehicle is located is acquired by using the image pickup device, real-time video can be shot by using the image pickup device to the environment outside the vehicle, then image frames are intercepted from the video stream as real-time environment images, and the real-time environment images of the environment outside the vehicle can be obtained by shooting pictures of the environment outside the vehicle in real time by using the image pickup device, so that the method is not particularly limited. The image pickup apparatus may be mounted inside the vehicle or may be mounted outside the vehicle, and is not particularly limited. In general, the view in front of the vehicle is relatively wide, the range of observation is relatively wide, the panoramic range image of the vehicle can be basically simulated through the content observed by the view in front, and the forward image at the previous moment can be changed into the backward image at the next moment along with the running of the vehicle, so that only the real environment image in front of the vehicle is acquired, and the panoramic image of the vehicle can be simulated. Therefore, the image pickup apparatus used in the present application to collect an image of the environment in which the vehicle is currently located may be a general image pickup apparatus, that is, only an image of the environment in the forward direction of the vehicle can be collected. Of course, in order to acquire more real environment images in multiple directions, the imaging device may also adopt panoramic imaging devices, such as panoramic cameras, panoramic driving recorders, and the like, and may acquire 360 ° panoramic images around the vehicle, so as to obtain real environment images in multiple directions, which provides a precondition for obtaining accurate three-dimensional panoramic images in subsequent construction, and is not particularly limited.
S102, identifying the type of the environment element in the real environment image to obtain at least one type tag;
Specifically, the environmental element includes at least one of a running road surface, a road surface condition, weather, a surrounding environment, a traffic condition, a surrounding building, and a surrounding article. The type of the driving road surface refers to the type of the road on which the current vehicle is driving, and includes highways, expressways, mud lands, sand lands, rock roads, gravel roads, grasslands, etc. The type of road surface condition refers to the specific condition of the road on which the current vehicle is running, and comprises icing, potholes, snow accumulation, water accumulation and the like. The weather type refers to the actual weather of the current environment of the vehicle, and includes light rain, heavy rain, snowing, heavy snow, heavy fog, hail, sunny days, etc. The type of the surrounding environment refers to the surrounding environment condition of the current vehicle, including villages, cities, deserts, grasslands, seas, forests and the like. The traffic situation refers to the traffic situation of the road on which the current vehicle is traveling, and comprises unblocked, basically unblocked, slightly congested, moderately congested, severely congested and the like. The types of surrounding buildings refer to the types of buildings around the current vehicle, including residential buildings, commercial buildings, tile houses, malls, and the like. The type of the surrounding articles refers to the specific article type of the surrounding of the current vehicle, such as automobiles, motorcycles, bicycles, tricycles, pedestrians, enclosing walls and the like. The weather type tag can further comprise visibility information, the surrounding building type tag can further comprise building sizes such as total height, layer height and width, the road surface condition type tag can further comprise road surface flatness information, the surrounding article type tag can further comprise vehicle type information, namely vehicle type information of vehicles is further recognized when the surrounding article is recognized as being vehicles, such as sports vehicles, off-road vehicles and the like, and specific vehicle brands and accurate models can be further recognized, and therefore information of real environments with more dimensions can be obtained. According to the actual application situation, the content included in the environment elements can be further widened, the type of each environment element can be further expanded, or the types of the environment elements can be divided into different dimensions according to different application situations, and the method is not limited in particular.
When the real environment image is obtained, the type of the environment element in the real environment image can be identified through an image identification algorithm, for example, the image that the real environment image is obtained currently is an image of a city road in a rainy day, and the corresponding type label can be identified through the image identification algorithm to include city, highway, rainy day and ponding, so that the environment element of the environment where the vehicle is located currently can be analyzed, and a basis is provided for obtaining accurate scene element images subsequently.
The image recognition algorithm may be a Convolutional Neural Network (CNN), a generating countermeasure network (GAN), a regional convolutional neural network (R-CNN), YOLO, a visual transducer, etc., and other algorithms capable of realizing image recognition may be applied to the present application, which is not particularly limited.
S103, respectively acquiring scene images corresponding to each type tag from an external database to obtain a plurality of scene element images;
Specifically, after the type labels are obtained, scene images corresponding to each type label are respectively obtained from an external database by using the type labels, and when the type labels are city, highway, light rain and water accumulation, the scene images corresponding to the city, highway, light rain and water accumulation are respectively obtained from the external database, so that a plurality of scene element images are obtained. When the external database is built, the types of the environment elements of all the images in the external database are firstly identified so that each image and the like can correspond to at least one preset label, then the images are grouped and classified according to the environment elements, namely, the groups of the driving road surface, the road surface condition, the weather, the surrounding environment, the traffic condition, the surrounding buildings and the surrounding objects are firstly classified, and then all the images in the external database are classified according to the groups. When a plurality of scene element images are required to be acquired from the external database, firstly calling the images under the corresponding environment element groups in the external database according to the environment elements corresponding to the type labels, and then comparing the type labels with preset labels of the images under the corresponding environment element groups so as to obtain the corresponding scene element images. Because the images in the external database are grouped and classified according to the environmental elements, the labels of all the images in the database are not required to be compared, and the acquisition efficiency of the scene element images can be effectively improved.
The external database may be a conventional relational database, such as MySQL, postgreSQL, a non-relational database (NoSQL), such as MongoDB, cassandra, a distributed file system (HDFS), etc., which is not limited in particular. The external database can be carried on the cloud server, and after the type label is obtained, the type label is transmitted to the cloud server in a wireless mode and the like, and then the corresponding scene element image is obtained from the external database of the cloud server. The external database can also be deployed in a hybrid cloud architecture mode, and the external database is connected through the cloud server, namely after the type tag is obtained, the type tag is transmitted to the cloud server, and then the cloud server issues a command to specific external data to call the scene element image, so that the method is not particularly limited.
S104, constructing a three-dimensional panoramic image simulating the surrounding environment of the current vehicle based on the plurality of scene element images.
Specifically, after a plurality of scene element images are obtained through the type tag, the surrounding environment of the current vehicle is simulated by utilizing the plurality of scene element images, so that a three-dimensional panoramic image is obtained. The scene image is obtained according to the environmental elements in the real environment image, so that the three-dimensional panoramic image obtained through final simulation can accurately restore the real environment where the current vehicle is located, drivers and passengers can know the real environment conditions outside the vehicle through the three-dimensional panoramic image, the visual field range is wider, the information is more accurate, the vehicle does not need to be parked or taken off, and the driving safety and experience are better.
According to the method, based on the steps S101-S104, a real environment image is acquired firstly, then the environment element types in the real environment image are identified to obtain corresponding type labels, the type labels can represent real environment states, scene element images obtained from an external database by means of the type labels can accurately represent the real environment conditions, finally, the three-dimensional panoramic image constructed according to a plurality of scene element images can effectively simulate the current surrounding environment conditions of a vehicle, the real image of the environment where the current vehicle is located can be observed through the three-dimensional panoramic image in the running process of the vehicle, parking and getting-off observation are not needed, driving safety is high, and driving experience is good. Meanwhile, the scene images are expanded by using the type labels and the external database, a large number of real environment images are not required to be acquired for analysis, the processing efficiency and the computing efficiency are high, the scene element images are stored in the external database, the limitation on the storage space of the database is small, different scene images can be stored as much as possible, the application range is wider, and the computing efficiency is higher.
The image processing method can be applied to the intelligent cabin or intelligent driving and other technologies. The intelligent cabin is an automotive interior environment manufactured by utilizing advanced information technology, artificial intelligence, the Internet of things, big data and other technologies, and can improve driving experience, convenience and safety of an owner. The intelligent cabin not only comprises traditional hardware equipment such as an instrument panel and a seat, but also integrates various intelligent systems and services. The intelligent driving technology is to make the automobile capable of autonomous sensing, understanding, deciding and controlling by utilizing artificial intelligence, machine learning, computer vision, laser radar, camera, radar and other technologies, so as to realize unmanned or auxiliary driving function. Whether it be intelligent cabins or intelligent driving technologies, it is necessary to load a control system, i.e. a vehicle-to-machine system, on the vehicle. In the car machine system, a central control screen is usually arranged, and man-machine interaction can be realized through the central control screen. When the method is applied to intelligent cabins or intelligent driving, the three-dimensional panoramic image can be displayed in the central control screen in the vehicle, and interaction with a user is realized through the central control screen. In the driving process of the vehicle, a real environment image can be acquired in real time, the types of environment elements in the real environment image are identified in real time to obtain type labels, then a plurality of scene element images are obtained from an external database according to the type labels in real time, a three-dimensional panoramic image is constructed in real time, finally the three-dimensional panoramic image generated in real time is displayed in a central control screen, so that the three-dimensional panoramic image can be changed in real time according to the driving condition of the vehicle, the environment outside the vehicle can be displayed in the vehicle in time, a driver and a passenger can observe the surrounding environment of the vehicle without getting off the vehicle, the driving efficiency and the safety are higher, the driving experience and the driving safety of intelligent cabin and intelligent driving are improved, meanwhile, the external database is selected for storing and obtaining the scene element images, a vehicle system does not need to be additionally provided with the external database, the vehicle system resources are not occupied, and the computing efficiency is higher.
The construction of the three-dimensional panoramic image can be realized by utilizing an image stitching technology and a three-dimensional engine technology. In some embodiments, the constructing a three-dimensional panoramic image simulating the current vehicle surroundings based on the plurality of scene element images includes:
Splicing the plurality of scene element images into a planar panoramic image by using an image splicing technology;
And mapping the planar panoramic image into a spherical panoramic image by utilizing a three-dimensional engine technology to obtain the three-dimensional panoramic image.
Specifically, the image stitching technique is a technique for combining a plurality of images into a single wide or high-resolution image, and is commonly used for panoramic image production. The image stitching technology of the application can be a stitching algorithm based on a spatial domain image, a stitching algorithm based on normalized cross correlation, an image stitching algorithm based on mutual information, a stitching algorithm based on bottom layer characteristics, and the like, and other technologies capable of realizing image stitching can be applied to the application without limitation. The image stitching technique may include image preprocessing, feature detection and matching, image registration, image transformation and stitching, and image fusion and optimization steps, where (1) the image preprocessing includes performing preliminary processing, such as denoising, graying, contrast adjustment, etc., on the input multiple scene element images to improve stability of subsequent processing, (2) feature detection and matching includes using feature point detection algorithm to find key feature points (such as corner points or texture points) in the respective images, and finding out matching point pairs between the different scene element images by matching algorithm, (3) image registration includes calculating a transformation matrix (perspective transformation matrix) from one scene element image to another scene element image according to the matching feature points to ensure that geometric relationships between the scene element images are accurately aligned, (4) image transformation and stitching includes using the calculated transformation matrix to map each scene element image onto a common plane according to the transformation relationships thereof, where the geometric transformations, such as rotation, translation, etc., are required to enable the different scene element images to be found out, so that the initial image can be compared with the panoramic image, and the final image can be fused with a seamless joint quality, such as to reduce the problem of course, and the final image is fused with the final image, and the image is fused with a seamless joint quality, such as to improve the image quality, other steps can be adopted to process the multiple scene element images to obtain the planar panoramic image, and the method is not particularly limited.
Three-dimensional engine technology (3D engine technology) is a computer graphics tool and software framework for creating and rendering three-dimensional graphics, scene interactions, and virtual environments, capable of providing functionality for processing and exposing elements of three-dimensional models, scenes, lighting, textures, animations, and the like. In three-dimensional engine technology, sky sphere (Skybox) technology is commonly used to simulate a wide range of environments, such as sky, distant horizon, cloud layer, starry sky, etc., and the sky sphere can make an observer perceive the wide of the virtual world in a large and endless environment. In the application, the plane panoramic image can be mapped into the spherical panoramic image by using the sky sphere technology in the three-dimensional engine technology, so as to obtain the three-dimensional panoramic image. The three-dimensional panoramic image is a spherical panoramic image, so that a user can observe the all-round information around the vehicle conveniently, the user can observe the environment around the vehicle in real time without stopping, and the driving efficiency, the driving safety and the driving experience are better.
After the planar panoramic image is obtained, some sundries or objects affecting the three-dimensional effect may exist in the planar panoramic image, and at this time, some techniques may be adopted to remove the sundries. In some embodiments, before the mapping the planar panoramic image into the spherical panoramic image using the three-dimensional engine technique, the method further comprises:
And removing objects below a preset height threshold from the ground in the planar panoramic image by using an image removing technology.
In particular, image stripping techniques can automatically or semi-automatically remove certain objects or elements from an image while maintaining consistency and naturalness of other portions of the image, often applied to scenes such as image editing, visual restoration, landscaping, and the like. In the three-dimensional reconstruction process, parallax refers to the change of the position of an object in an image along with the change of an observation angle, and the smaller the size of the object is, the less obvious the parallax effect is. When objects are very small (e.g., below 10 cm), their displacement at multiple viewing angles is hardly captured, which makes depth information of these objects not accurately estimated in three-dimensional reconstruction. At the same time, small objects often introduce more image noise, especially in low resolution images or in situations where the shooting environment is not sufficiently light. These noises may interfere with the effect of the three-dimensional reconstruction algorithm, resulting in that small objects in the reconstruction result cannot be effectively removed or repaired, affecting the final three-dimensional image quality. The goal of the three-dimensional panorama is to create a panoramic effect with visual consistency that can be freely browsed in virtual space. When these small objects are present, unnatural seams or shape distortions may occur during stitching, especially between different image overlap areas. In summary, when some small objects exist in the planar panoramic image, problems of unobvious parallax, inaccurate depth reconstruction, noise interference and the like are caused during three-dimensional construction, and the small objects are difficult to provide enough depth information between multiple visual angles, so that the three-dimensional reconstruction effect is finally affected, and the quality of the three-dimensional panoramic image is reduced. Therefore, before the three-dimensional panoramic image is constructed by utilizing the planar panoramic image, objects which are below a preset height threshold value from the ground in the planar panoramic image are removed by utilizing an image removing technology, and then the three-dimensional panoramic image is constructed, so that the precision and the visual consistency can be effectively improved, smooth transition of the panoramic image under different visual angles can be ensured, meanwhile, small objects are removed, the calculated amount is reduced, and the calculation efficiency is effectively improved. Alternatively, the preset height threshold may be 10cm, or 15, 14, 13, 12, 11, 9, 8, 7, 6,5, 4, 3, 2, or 1cm, or may be set to other values according to practical situations, which is not particularly limited.
The three-dimensional model of the vehicle is fused into the three-dimensional panoramic image, so that driving experience can be further improved. In some embodiments, further comprising:
acquiring three-dimensional model data of a current vehicle;
and fusing the three-dimensional model data of the current vehicle with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing the three-dimensional model of the current vehicle.
Specifically, the three-dimensional model data of the current vehicle is fused with the three-dimensional panoramic image to obtain the three-dimensional panoramic image containing the three-dimensional model of the current vehicle, and a user can more truly, accurately and intuitively observe the environment around the vehicle, so that driving experience and driving safety are effectively improved.
After the type label is obtained, prompt information can be generated and sent out according to the type label to remind a driver of paying attention. In some embodiments, the real environment image comprises a vehicle forward environment image, the method further comprising:
and generating and sending prompt information in response to the type tag corresponding to the vehicle forward environment image accords with a preset tag range which influences driving safety.
Specifically, a preset tag range is set in advance, and a type tag corresponding to a situation affecting driving safety is classified therein. And after the type label is obtained by identifying the real environment image, judging whether the type label is in a preset label range, if so, indicating that the driving safety is influenced in the current environment, and generating and sending prompt information according to the content of the type label. For example, if the type tag obtained by recognition includes "icing", the preset tag range also includes "icing", and the icing road easily causes the vehicle to skid, so as to directly influence the driving safety, and at this time, a prompt message is generated and sent according to the type tag of "icing", for example, the driver can be reminded of "front road icing", "speed-down running", etc., so as to remind the driver of paying attention, and further improve the driving safety. For example, if the type tag obtained by recognition includes "ponding", the range of the preset tag also includes "ponding", and the ponding road may cause flameout of the vehicle, at this time, a prompt message is generated and sent according to the type tag of "ponding", for example, the "front ponding", "uniform speed passing" and the like may be reminded. In the running process of the vehicle, prompt information is generated and sent in real time, the function of predicting the running direction condition is achieved, and a user can timely change and adjust driving parameters according to the prompt information, so that driving safety is further improved.
The hint information may be further fused in the three-dimensional panoramic image. In some embodiments, the generating and issuing the hint information includes:
And generating corresponding early warning information and/or driving advice according to the type tag, and fusing the early warning information and/or the driving advice with the three-dimensional panoramic image to obtain the three-dimensional panoramic image containing the front environment prompt information.
Specifically, the prompt information may include early warning information and travel advice. When the condition affecting the driving safety exists in the environment where the vehicle is located, corresponding early warning information or driving advice is generated according to the type label, the early warning information and the driving advice can be generated at the same time, and then the early warning information and the driving advice are fused with the three-dimensional panoramic image and displayed in the three-dimensional panoramic image, so that a driver can be reminded of driving carefully, and the driving safety is improved. For example, when the type tag is "icing", corresponding early warning information, such as "front road icing", may be generated, and corresponding driving advice, such as "low-grade low-speed forward running", may be generated, and the information of "front road icing", "low-grade low-speed forward running" may be fused with the three-dimensional panoramic image, so that the "front road icing", "low-grade low-speed forward running" may be displayed in the three-dimensional panoramic image, and a user may observe surrounding conditions in time through the three-dimensional panoramic image, and adjust driving parameters in time according to the early warning information and the driving advice, thereby improving driving safety. Optionally, the early warning information and the driving advice can be displayed at the position of the content corresponding to the type tag in the three-dimensional panoramic image, taking the type tag as an example of icing, and the information corresponding to icing of a front road, low-speed forward running and the like can be directly displayed at the position with icing in the three-dimensional panoramic image, so that drivers are better reminded of avoiding or driving, and driving safety is better improved.
The navigation information may be further fused in the three-dimensional panoramic image. In some embodiments, navigation information of a current vehicle is obtained;
generating driving direction guide information based on the navigation information and the real environment image;
and fusing the driving direction guide information with the three-dimensional panoramic image to obtain the three-dimensional panoramic image containing driving direction guide.
Specifically, the driving direction guiding information is generated according to the navigation information and then fused with the three-dimensional panoramic image, so that the driving direction guiding information is displayed in the three-dimensional panoramic image, the driving of a user can be better guided, and the driving safety and the driving experience are improved. When the vehicle is required to travel on a lane change or overtaking, the driving direction guide information such as a lane change path, an overtaking path and the like is generated according to the navigation information, and then the driving direction guide information such as the lane change path, the overtaking path and the like is fused with the three-dimensional panoramic image, so that a driver can intuitively determine the driving direction related information such as the lane change path or the overtaking path and the like by observing the three-dimensional panoramic image, the driving efficiency is higher, and the driving experience is better.
When the method is applied to intelligent cabins or intelligent driving, the three-dimensional panoramic image is displayed in the central control large screen, and early warning information, driving advice, driving direction guide information and the like are displayed and updated along with the three-dimensional panoramic image in the central control large screen, so that the user can observe more intuitively, and driving experience is better.
According to the image processing method, the real environment image is acquired, then the environment element types in the real environment image are identified to obtain the corresponding type label, the type label can represent the real environment state, the scene element images obtained from the external database by the type label can accurately represent the real environment condition, the external database is not required to be mounted in a vehicle-mounted system, the storage limit of the vehicle-mounted system can be effectively reduced, the computing efficiency is improved, then the planar panoramic image is obtained by splicing the plurality of scene element images, finally the planar panoramic image is mapped into the spherical panoramic image by utilizing the three-dimensional engine technology, the real environment condition of the current vehicle can be effectively reflected, meanwhile, the three-dimensional panoramic image is the spherical panoramic image, a user has an omnibearing view, the user can observe omnibearing information around the vehicle conveniently, the user can observe the environment around the vehicle in real time without stopping or getting off the vehicle, and the driving efficiency, the driving safety and the driving experience are better. On the basis, the three-dimensional model, the early warning information, the driving advice, the driving direction guiding information and the like of the vehicle can be fused into the three-dimensional panoramic image, so that the driver can acquire information more intuitively, and further, the driver can be guided to drive better, and the driving safety and the driving experience are further improved.
It should be noted that, the method of the embodiment of the present application may be performed by a single device, for example, a computer or a server. The method of the embodiment can also be applied to a distributed scene, and is completed by mutually matching a plurality of devices. In the case of such a distributed scenario, one of the devices may perform only one or more steps of the method of an embodiment of the present application, the devices interacting with each other to accomplish the method.
It should be noted that the foregoing describes some embodiments of the present application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments described above and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
Based on the same inventive concept, the present application also provides an image processing apparatus, corresponding to the method of any of the above embodiments, referring to fig. 2, the image processing apparatus comprising:
The acquisition module 201 is used for acquiring an image of the environment where the current vehicle is located to obtain a real environment image;
the element identification module 202 is configured to identify a type of an environmental element in the real environmental image, and obtain at least one type tag;
The image obtaining module 203 is configured to obtain scene images corresponding to each type tag from an external database, so as to obtain a plurality of scene element images;
The generating module 204 is configured to construct a three-dimensional panoramic image simulating the current vehicle surroundings based on the plurality of scene element images.
In some embodiments, the generation module 204 is further to:
Splicing the plurality of scene element images into a planar panoramic image by using an image splicing technology;
And mapping the planar panoramic image into a spherical panoramic image by utilizing a three-dimensional engine technology to obtain the three-dimensional panoramic image.
In some embodiments, before the mapping the planar panoramic image into the spherical panoramic image using the three-dimensional engine technique, the method further comprises:
And removing objects below a preset height threshold from the ground in the planar panoramic image by using an image removing technology.
In some embodiments, further comprising:
The first acquisition module is used for acquiring three-dimensional model data of the current vehicle;
The first fusion module is used for fusing the three-dimensional model data of the current vehicle with the three-dimensional panoramic image to obtain a three-dimensional panoramic image containing the three-dimensional model of the current vehicle.
In some embodiments, the real environment image comprises a vehicle forward environment image, the apparatus further comprising:
and the reminding module is used for generating and sending out prompt information in response to the type tag corresponding to the vehicle forward environment image accords with a preset tag range which influences driving safety.
In some implementations, the generating and issuing the hint information includes:
And generating corresponding early warning information and/or driving advice according to the type tag, and fusing the early warning information and/or the driving advice with the three-dimensional panoramic image to obtain the three-dimensional panoramic image containing the front environment prompt information.
In some embodiments, further comprising:
the second acquisition module is used for acquiring navigation information of the current vehicle;
the information generation module is used for generating driving direction guide information based on the navigation information and the real environment image;
And the second fusion module is used for fusing the driving direction guiding information with the three-dimensional panoramic image to obtain the three-dimensional panoramic image containing the driving direction guiding.
For convenience of description, the above devices are described as being functionally divided into various modules, respectively. Of course, the functions of each module may be implemented in the same piece or pieces of software and/or hardware when implementing the present application.
The device of the foregoing embodiment is configured to implement the corresponding image processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which is not described herein.
Based on the same inventive concept, the application also provides an electronic device corresponding to the method of any embodiment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor implements the image processing method of any embodiment when executing the program.
Fig. 3 shows a more specific hardware architecture of an electronic device provided by the present embodiment, which may include a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 implement communication connections therebetween within the device via a bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit ), a microprocessor, an Application SPECIFIC INTEGRATED Circuit (ASIC), or one or more integrated circuits, etc. for executing related programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory ), static storage, dynamic storage, etc. Memory 1020 may store an operating system and other application programs, and when the embodiments of the present specification are implemented in software or firmware, the associated program code is stored in memory 1020 and executed by processor 1010.
The input/output interface 1030 is used to connect with an input/output module for inputting and outputting information. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.
Communication interface 1040 is used to connect communication modules (not shown) to enable communication interactions of the present device with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).
Bus 1050 includes a path for transferring information between components of the device (e.g., processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).
It should be noted that although the above-described device only shows processor 1010, memory 1020, input/output interface 1030, communication interface 1040, and bus 1050, in an implementation, the device may include other components necessary to achieve proper operation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.
The electronic device of the foregoing embodiment is configured to implement the corresponding image processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which is not described herein.
Based on the same inventive concept, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the image processing method according to any of the above embodiments, corresponding to the method according to any of the above embodiments.
The computer readable media of the present embodiments, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.
The storage medium of the above embodiment stores computer instructions for causing the computer to perform the image processing method according to any one of the above embodiments, and has the advantages of the corresponding method embodiments, which are not described herein.
Based on the same conception, the present application also provides a computer program product corresponding to the method of any embodiment, which includes computer program instructions, when the computer program instructions run on a computer, cause the computer to execute the image processing method according to any embodiment, and the method has the advantages of the corresponding method embodiment, which are not repeated herein.
It will be appreciated that before using the technical solutions of the various embodiments in the disclosure, the user may be informed of the type of personal information involved, the range of use, the use scenario, etc. in an appropriate manner, and obtain the authorization of the user.
For example, in response to receiving an active request from a user, a prompt is sent to the user to explicitly prompt the user that the operation it is requesting to perform will require personal information to be obtained and used with the user. Therefore, the user can select whether to provide personal information to the software or hardware such as the electronic equipment, the application program, the server or the storage medium for executing the operation of the technical scheme according to the prompt information.
As an alternative but non-limiting implementation, in response to receiving an active request from a user, the manner in which the prompt information is sent to the user may be, for example, a popup, in which the prompt information may be presented in a text manner. In addition, a selection control for the user to select to provide personal information to the electronic device in a 'consent' or 'disagreement' manner can be carried in the popup window.
It will be appreciated that the above-described notification and user authorization process is merely illustrative, and not limiting of the implementations of the present disclosure, and that other ways of satisfying relevant legal regulations may be applied to the implementations of the present disclosure.
It will be appreciated by persons skilled in the art that the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the application is limited to these examples, that combinations of technical features in the above embodiments or in different embodiments may also be implemented in any order, and that many other variations of the different aspects of the embodiments of the application as described above exist within the spirit of the application, which are not provided in detail for the sake of brevity.
Additionally, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures, in order to simplify the illustration and discussion, and so as not to obscure the embodiments of the present application. Furthermore, the devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present application, and also in view of the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the embodiments of the present application are to be implemented (i.e., such specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the application, it should be apparent to one skilled in the art that embodiments of the application can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative in nature and not as restrictive.
While the application has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of those embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
The present embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the claimed application. Therefore, any omissions, modifications, equivalent substitutions, improvements, and the like, which are within the spirit and principles of the embodiments of the application, are intended to be included within the scope of the application.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411988235.6A CN119904841A (en) | 2024-12-31 | 2024-12-31 | Image processing method, device, electronic equipment and vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411988235.6A CN119904841A (en) | 2024-12-31 | 2024-12-31 | Image processing method, device, electronic equipment and vehicle |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN119904841A true CN119904841A (en) | 2025-04-29 |
Family
ID=95465956
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411988235.6A Pending CN119904841A (en) | 2024-12-31 | 2024-12-31 | Image processing method, device, electronic equipment and vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119904841A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN121280504A (en) * | 2025-12-09 | 2026-01-06 | 天津云圣智能科技有限责任公司 | Depth data automatic acquisition method and device, electronic equipment and storage medium |
-
2024
- 2024-12-31 CN CN202411988235.6A patent/CN119904841A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN121280504A (en) * | 2025-12-09 | 2026-01-06 | 天津云圣智能科技有限责任公司 | Depth data automatic acquisition method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11282164B2 (en) | Depth-guided video inpainting for autonomous driving | |
| CN110758243B (en) | Surrounding environment display method and system in vehicle running process | |
| US12394313B1 (en) | Displaying sensor data and supplemental data as a mask for autonomous vehicles | |
| CN111144211B (en) | Point cloud display method and device | |
| CN111918053A (en) | Vehicle image verification | |
| CN110471058A (en) | The system and method detected automatically for trailer attribute | |
| CN116420096A (en) | Method and system for labeling LIDAR point cloud data | |
| US12190463B2 (en) | Synthesizing three-dimensional visualizations from perspectives of onboard sensors of autonomous vehicles | |
| CN112184844A (en) | Vehicle image generation | |
| CN114842455B (en) | Obstacle detection method, device, equipment, medium, chip and vehicle | |
| US11568653B2 (en) | Vehicle environment modeling with a camera | |
| US20210150226A1 (en) | Way to generate tight 2d bounding boxes for autonomous driving labeling | |
| CN119904841A (en) | Image processing method, device, electronic equipment and vehicle | |
| CN118941626A (en) | Roadside visual positioning method, device, equipment and medium for traffic participants | |
| CN117470258A (en) | A map construction method, device, equipment and medium | |
| CN119590408B (en) | Visitor parking method, device and equipment based on binocular camera and storage medium | |
| US12542875B2 (en) | Hybrid architecture of birds-eye view features and pixels for autonomous driving perception | |
| CN119152456A (en) | Method, device, equipment and storage medium for sensing environment around vehicle | |
| US20240223915A1 (en) | Systems and methods for downsampling images | |
| CN119723502B (en) | A method, device and medium for generating a vehicle virtual model based on a large model | |
| US20260017875A1 (en) | Simulation of viewpoint capture from environment rendered with ground truth heuristics | |
| CN121962464A (en) | Methods, devices and equipment for three-dimensional reconstruction of real-time vehicle environment | |
| CN119290006A (en) | Vehicle navigation method, device, computer equipment, storage medium and program product | |
| CN120198886A (en) | Method, device, medium and equipment for displaying vehicle driving environment | |
| CN121482324A (en) | Proximity-based surface texture generation for simulating environmental systems and applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |