CN110929606A - Vehicle blind area pedestrian monitoring method and device - Google Patents
Vehicle blind area pedestrian monitoring method and device Download PDFInfo
- Publication number
- CN110929606A CN110929606A CN201911096956.5A CN201911096956A CN110929606A CN 110929606 A CN110929606 A CN 110929606A CN 201911096956 A CN201911096956 A CN 201911096956A CN 110929606 A CN110929606 A CN 110929606A
- Authority
- CN
- China
- Prior art keywords
- camera
- pedestrian
- coordinate system
- coordinates
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
- G06V10/247—Aligning, centring, orientation detection or correction of the image by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Traffic Control Systems (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The embodiment of the invention provides a method and a device for detecting pedestrians in vehicle blind areas, wherein the method comprises the following steps: acquiring a vehicle blind area image, and detecting the vehicle blind area image through an image detection technology to obtain frame information of pedestrians; determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera; and determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance. Therefore, the calculated amount in the vision-based vehicle blind area pedestrian monitoring method is reduced, and the real-time performance of pedestrian monitoring is ensured.
Description
Technical Field
The invention relates to the technical field of pedestrian monitoring, in particular to a method and a device for monitoring pedestrians in vehicle blind areas.
Background
With the development of the world economy and the improvement of the living standard of people, the automobile is undoubtedly a necessity for the life of a plurality of people. Although automobile equipment brings considerable life convenience, economic benefits and social prosperity to human beings, the potential safety hazard is greatly increased. Statistical data show that accidents caused by automobile blind areas account for about 30% in China and about 20% in the United states of America among all traffic accidents. This is caused and caused by human eye physiological structure, automobile design and other factors. The data show that the detection and early warning device for the blind area of the automobile is researched, the device can play an obvious control role in reducing the incidence rate of similar traffic accidents, and is expected to reduce casualties in the traffic accidents in the most thorough mode. Therefore, from the perspective of traffic safety, in order to improve the safety of the automobile, a feasible technical system capable of detecting the blind area of the automobile is designed and developed, which particularly shows practical value and significance.
Currently, commonly used visual vehicle detection methods can be classified into the following two categories: the first kind of method is based on vehicle radar monitoring, and is characterized by that two 24GHz radar sensors are mounted in the rear bumper of the automobile, and when the running speed of the automobile is greater than a certain value, they can be automatically started, and can give out detection microwave signal toward right in real time, and the system can analyze and process the reflected microwave signal, so that the information of distance, speed and moving direction of pedestrian on right side can be known, and the fixed object and remote object can be removed by means of system algorithm, and the whole running process can be uninterruptedly detected and reminded so as to prevent the traffic safety accident resulted from the potential dangers of bad weather, driver's negligence, rear-view mirror blind area and new hand's road-going. The method has the advantages of small calculation amount and good real-time performance. But has poor environmental adaptability, low robustness and excessive false detection. The second method is a blind area pedestrian distance measurement algorithm based on vision, and compared with the existing radar sensor, the vision camera has higher cost performance and is widely applied to the field of intelligent vehicles. However, the existing blind area pedestrian distance measurement algorithm based on vision is complex in algorithm and large in calculation amount, so that the requirement on the calculation capability of the vehicle-mounted embedded platform is high, and meanwhile, the real-time performance of pedestrian monitoring cannot be guaranteed.
Therefore, how to reduce the amount of calculation in the vision-based vehicle blind area pedestrian monitoring method and ensure the real-time performance of pedestrian monitoring is still a problem to be solved by those skilled in the art.
Disclosure of Invention
The embodiment of the invention provides a vehicle blind area pedestrian monitoring method and device, which are used for solving the problems that in the prior art, the vehicle blind area pedestrian monitoring method based on vision is large in calculation amount and cannot ensure the real-time performance of pedestrian monitoring.
In a first aspect, an embodiment of the present invention provides a vehicle blind area pedestrian monitoring method, including:
acquiring a vehicle blind area image, and detecting the vehicle blind area image through an image detection technology to obtain frame information of pedestrians;
determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera;
and determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance.
Preferably, the vehicle blind area pedestrian monitoring method further includes:
and displaying the distance, and triggering an alarm of a level corresponding to the distance according to the distance between the pedestrian and the entity camera.
Preferably, the detecting the image by an image detection technology specifically includes: and detecting the image by an image detection technology based on Fast-SCNN algorithm.
Preferably, the performing single-point distortion removal operation on the position point to obtain the coordinates of the position point in the image coordinate system of the ideal camera specifically includes:
based on internal parameters obtained by calibrating the entity camera, carrying out distortion removal operation on coordinates of the position point in an image coordinate system of the entity camera to obtain coordinates of the position point in the image coordinate system of the ideal camera;
the entity camera is a monocular camera, and the internal parameters comprise a transverse axis focal length, a longitudinal axis focal length, principal point coordinates and a distortion coefficient of the entity camera.
Preferably, the obtaining of the coordinates of the position point in the image coordinate system of the ideal camera by performing distortion removal operation on the coordinates of the position point in the image coordinate system of the actual camera based on the internal parameters obtained by calibrating the vehicle-mounted monocular camera specifically includes:
after the entity camera is calibrated, the obtained internal parameters comprise a horizontal axis focal length fx and a vertical axis focal length f of the camerayPrincipal point coordinates (c)x,cy) And distortion coefficient [ k ]1,k2,p1,p2,k3]；
The coordinates of the position point in the image coordinate system in the actual camera are (u ', v'), u 'and v' are both constant, the coordinates of the position point in the image coordinate system in the ideal camera are (u, v),
according to the following formula,
u'＝x×(1+k1×r2+k2×r4+k3×r6)+2×p1×x×y+p2×(r2+2x2)
v'＝y×(1+k1×r2+k2×r4+k3×r6)+2×p2×x×y+p2×(r2+2y2)
the coordinates (u, v) of the location point in the image coordinate system in the ideal camera are determined.
Preferably, the determining the physical coordinates of the position point in the physical coordinate system according to the coordinates of the position point in the image coordinate system in the ideal camera specifically includes:
the coordinates of the position point in an ideal camera are (u, v), u and v are all constants, the physical coordinates of the position point in a physical coordinate system are (x, y), dx and dy respectively represent the length of a pixel in the directions of the horizontal axis and the vertical axis, the unit is mm/pixel, gamma is a distortion factor, the value is 0,
according to the following formula,
the physical coordinates (x, y) of the location point in the physical coordinate system are determined.
Preferably, the determining the distance between the pedestrian and the camera specifically includes:
the distance between the pedestrian and the camera is D, the longitudinal distance between the pedestrian and the camera is VD, the transverse distance between the pedestrian and the camera is HD, the physical coordinates of the position point in a physical coordinate system are (x, y), x and y are constants, H is the height of the camera, H is a constant, f is the focal length of the camera, f is a constant, and the physical coordinates of the main point in the physical coordinate system are (c'x，c'y)，
According to the following formula,
a distance D between the pedestrian and the camera is determined.
In a second aspect, an embodiment of the present invention provides a vehicle blind area pedestrian monitoring apparatus, including:
the frame detection unit is used for collecting the vehicle blind area images and detecting the vehicle blind area images through an image detection technology to obtain frame information of pedestrians;
the distortion removing unit is used for determining the position point of the pedestrian according to the frame information of the pedestrian and carrying out single-point distortion removing operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera;
and the monitoring unit is used for determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance.
In a third aspect, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and being executable on the processor, and is characterized in that the processor implements the steps of the vehicle blind area pedestrian monitoring method according to the first aspect when executing the program.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the vehicle blind area pedestrian monitoring method as provided in the first aspect.
The embodiment of the invention provides a method and a device for monitoring pedestrians in vehicle blind areas, which are characterized in that vehicle blind area images are collected, and the vehicle blind area images are detected through an image detection technology to obtain frame information of the pedestrians; determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera; and determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance. Through carrying out the mapping that the distortion is removed to the single pixel of position point when the distortion is removed rather than carrying out the distortion removal to all pixel of whole picture, greatly reduced the calculated amount for vehicle blind area pedestrian's control's real-time has obtained the assurance.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief description will be given below of the drawings required for the embodiments or the technical solutions in the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic flow chart of a vehicle blind area pedestrian monitoring method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a vehicle blind area pedestrian monitoring device according to an embodiment of the present invention;
fig. 3 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
The existing blind area pedestrian distance measurement algorithm based on vision has high requirement on the computing capability of a vehicle-mounted embedded platform due to complex algorithm and large calculation amount, and meanwhile, the real-time performance of pedestrian monitoring cannot be guaranteed. Therefore, the embodiment of the invention provides a pedestrian monitoring method for a vehicle blind area. Fig. 1 is a schematic flow chart of a vehicle blind area pedestrian monitoring method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
and 110, collecting a vehicle blind area image, and detecting the vehicle blind area image through an image detection technology to obtain frame information of pedestrians.
Specifically, the blind area of the vehicle is the direction far from the driver seat, and for the domestic driver, the right side of the driver is the blind area of the driver. The camera for collecting the blind area image is placed on the vehicle body in the position of the blind area part of the vehicle, namely, the vehicle body in the position far away from the driver seat side, the shot image is the image of the area outside the vehicle on the side far away from the driver seat, for example, a domestic truck, the camera for collecting the blind area image is usually placed on the right side of the vehicle head and has a certain height from the ground, and the shot image comes from the area in the right front of the truck. At present, a vehicle-mounted camera is used for collecting a blind area image, and the vehicle blind area image is detected through an image detection technology, which can be implemented by a plurality of image target detection algorithms, for example, a full volumetric single-Stage (FCOS) based image target detection algorithm, a Fast-scan Segmentation Network (Fast Semantic Segmentation Network) based image target detection algorithm, and the like, and the image detection technology is not limited specifically here. Through the image detection technology, the frame information of the pedestrian on the image is output, namely a rectangular frame which just frames the pedestrian is displayed through the coordinates of four corners of the output frame.
And 120, determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system of the ideal camera.
Specifically, the bottom side of the frame is determined according to the frame information of the pedestrian, the bottom side is used as an intersection line of the pedestrian and the ground, and the left end point of the bottom side is used as a position point of the pedestrian. Due to the non-ideality of optical devices in the camera, for example, the distortion of a camera lens, the image acquired by the camera has certain distortion, and the coordinates of the position point in an image coordinate system in an ideal camera can be obtained by performing distortion removal operation on the position point of the pedestrian.
And step 130, determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance.
Specifically, when the camera shoots an object in the real world, the image of the object in the real world in the camera can be analyzed into two transformations, the object in the real world is firstly considered to enter a pinhole model which is imaged by the camera, and the pinhole model is considered to be the pinhole model given by the ideal camera at the moment, namely, the object is converted into an image coordinate system in the ideal camera from a physical coordinate system; then, considering the problem of lens distortion of the physical camera, the transformation of the object through pinhole imaging needs to be further transformed through lens distortion, so that the coordinates of the object in the image coordinate system of the physical camera are obtained by the distortion of the coordinates of the object in the image coordinate system of the ideal camera. Here, the physical coordinates of the position point in the real world can be derived from the image coordinates of the position point in the ideal camera, and the distance between the pedestrian and the physical camera can be determined according to the physical coordinates in the physical coordinate system under the condition that the installation height of the camera is known.
The embodiment of the invention provides a vehicle blind area pedestrian monitoring method, which comprises the steps of collecting vehicle blind area images, detecting the vehicle blind area images through an image detection technology, and obtaining frame information of pedestrians; determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in the ideal camera; and determining the coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance. Through carrying out the mapping that the distortion is removed to the single pixel of position point when the distortion is removed rather than carrying out the distortion removal to all pixel of whole picture, greatly reduced the calculated amount for vehicle blind area pedestrian's control's real-time has obtained the assurance.
Based on the above embodiment, the method further includes:
and displaying the distance and triggering an alarm of a level corresponding to the distance according to the distance between the pedestrian and the entity camera.
Specifically, in the running process of the vehicle, the condition of pedestrians outside the vehicle is monitored through the camera outside the vehicle, and when the situation that the pedestrians are close to the vehicle and potential dangers occur is found, the vehicle-mounted monitoring system can analyze the behaviors in real time and trigger auditory and/or visual alarm to remind a driver to take measures; and simultaneously, triggering auditory and/or visual alarm to remind pedestrians outside the automobile to pay attention to safety. Usually, the alarm incident also can be uploaded to the high in the clouds, realizes the long-range supervision of motorcade to vehicle and driver, promotes motorcade driving safety. When the distance between a pedestrian and a camera is displayed on the vehicle-mounted display screen, an alarm area is set on an image displayed on the vehicle-mounted display screen, and three lines of green lines, yellow lines and red lines are respectively drawn on the vehicle-mounted display screen to be respectively used as different alarm areas; the green line represents a three-meter line, and pedestrians between 2 meters and 3 meters are framed by a green frame to alarm; the yellow line represents two meters, and pedestrians between 1 meter and 2 meters adopt a yellow frame to frame the pedestrians to alarm; the red line represents a one-meter line, and pedestrians between 0 meter and 1 meter adopt a red frame to frame the pedestrians to alarm; no alarm is given in the area outside the green line; the alarming modes comprise three modes of visual alarming in the vehicle, auditory alarming in the vehicle and early warning outside the vehicle. The visual alarm mode in the vehicle is as follows: if the alarm is detected, a display screen picture is automatically popped up, and the potential danger personnel are identified and framed in real time according to the alarm, potential danger and blind spot detection) algorithm of different areas. The in-vehicle audible alarm mode is as follows: if only the green line area alarms, the beep sound is sent out; if only the yellow line area alarms, the beep sound is given; if only the red line area alarms, the sound of the ticker is given out; if there is alarm area mixing, according to the grade of red > yellow > green, sound with high priority is sent out. The early warning mode outside the vehicle is as follows: the green line area does not alarm; the yellow line area and/or the red line area trigger an audible and visual alarm to give out a sound of 'the bus is dangerous and please leave', and a red alarm flashing light is given out. The alarm elimination mode specifically comprises the following steps: the right front camera detects no pedestrian condition in the 'alarm area' through a BSD (Blind spot detection) algorithm. Recording an alarm: when an alarm is generated, the terminal sends alarm information to the platform and uploads the alarm information to the platform, and the content is recorded: alarm time (date and hour/minute/second), place (GPS longitude and latitude), alarm time photo (photo shot by camera), alarm time small video (small video shot by camera, time interval needs to be selected according to set time, such as the first 5 seconds and the last 5 seconds of alarm time), and alarm category (BSD alarm).
Based on any one of the embodiments, in the method, the detecting the image by using an image detection technology specifically includes:
and detecting the image by an image detection technology based on Fast-SCNN algorithm.
Specifically, Fast-SCNN algorithm is selected to detect the image, target detection and segmentation algorithm can be combined to predict each pixel point, and the pedestrian detection effect is better.
Specifically, the details of the Fast-SCNN algorithm are as follows: Fast-SCNN can be used for a real-time semantic segmentation task on a high-resolution image; a shallow learning down-sampling module is used for fast and efficient multi-branch low-level feature extraction by using a popular skip connection (residual connection) method in an offline DCNN (Deep Convolutional Neural Network); a low-capacity network is specially designed, and the effect of running more iteration training on the network structure is proved to be as successful as the effect of pre-training by using ImageNet (open source pre-training model) or training by using an additional refined training data set through verification; by subsampling the input data with Fast-SCNN, the performance of SOT (single object tracking) can be achieved without redesigning the network.
The semantic segmentation algorithm can be summarized into two types, namely a double-branch network and a multi-branch network, and Fast-SCNN is inspired by a double-branch method. There are four common methods in improving the efficiency of DCNN: deep separable convolution, designing more efficient network structure, network quantization and network compression, because the shallow layer of DCNN extracts low-level features such as edges and corners, so that a two-branch structure with independent computation is not needed, Fast-SCNN introduces learning to downsampling, and feature computation is shared on the bottom-layer branch and the upper-layer branch in the shallow network.
The Fast-SCNN comprises a learning downsampling module, a refined global feature extraction module, a feature fusion module and a standard classifier:
(1) a downsampling learning module: the method comprises three convolution layers, wherein the first layer adopts common convolution calculation because an input picture is three channels, and the other two layers are depth separable convolutions;
(2) the global feature extraction module uses efficient bottleneck residual blocks in a MobileNet V2 (open source classical convolutional neural network), the convolutions are changed into depth separable convolution layers, and finally a pyramid pooling module is added to aggregate context information based on different regions;
(3) a feature fusion module: table 1 shows a feature fusion module, and as shown in table 1, the specific design of the module includes: a first part, High resolution (one High resolution module), Conv2D 1/1(2D convolution); the second part, X time lowerresolution (X low fraction modules), Upsample × X (X upsampling), dwconv (scaling X)3/1, f (Depth Wise connected Depth separation convolution (dilation operation), followed by a non-linear function), Conv2D 1/1(2D convolution);
add, f represents: adding the outputs of the first and second portions and then passing through a non-linear function.
TABLE 1 feature fusion Module
(4) A classifier: two depth separable convolutional layers plus one point-by-point convolution.
Since the network uses the float (floating point) data type to predict the input picture pixel by pixel, while floating point calculation requires more calculation cost than integer and binary operation, the model running time can be further shortened by quantization skill.
The output of the network is that on a gray scale, masks with different RGB (red, green, blue) values are predicted for different categories, on the basis, holes in the masks are removed through a series of operations such as expansion, corrosion and the like on the masks predicted by pedestrians, and finally the minimum circumscribed rectangle on a mask connected domain is obtained to obtain a pedestrian detection frame, and then the pedestrian detection frame is drawn on an original image.
Based on any of the above embodiments, in the method, the performing single-point distortion removal operation on the position point to obtain a coordinate of the position point in an image coordinate system of an ideal camera specifically includes:
based on internal parameters obtained by calibrating the entity camera, carrying out distortion removal operation on coordinates of the position point in an image coordinate system of the entity camera to obtain coordinates of the position point in the image coordinate system of the ideal camera;
the entity camera is a monocular camera, and the internal parameters comprise a transverse axis focal length, a longitudinal axis focal length, principal point coordinates and a distortion coefficient of the entity camera.
Specifically, the entity camera is a monocular camera, and is calibrated in advance to obtain internal parameters of the camera, wherein the internal parameters include a camera transverse axis focal length, a camera longitudinal axis focal length, principal point coordinates and a distortion coefficient, and the principal point coordinates are coordinates of a central point of a principal point in an image coordinate system of the entity camera. And for the coordinates of the position point of the pedestrian acquired each time in the image coordinate system in the physical camera, the coordinates of the position point in the image coordinate system in the ideal camera are obtained through distortion removal operation.
Based on any of the embodiments, in the method, the obtaining, by performing distortion removal operation on coordinates of the position point in an image coordinate system of the actual camera based on the internal parameters obtained by calibrating the vehicle-mounted monocular camera, coordinates of the position point in the image coordinate system of the ideal camera specifically includes:
after the entity camera is calibrated, the obtained internal parameters comprise a transverse axis focal length fx and a longitudinal axis focal length f of the entity camerayPrincipal point coordinates (c)x,cy) And distortion coefficient [ k ]1,k2,p1,p2,k3](ii) a fx and fyThe focal length of the horizontal axis and the focal length of the vertical axis of the physical camera under the pixel coordinate system;
knowing the coordinates of the position point in an image coordinate system in an actual camera as (u ', v'), and setting the coordinates of the position point in the image coordinate system in an ideal camera as (u, v), wherein u and v are unknown numbers;
the position point in the ideal camera is obtained by the following formula of the coordinate of the distorted position point of the physical camera in the image coordinate system of the physical camera:
u'＝x×(1+k1×r2+k2×r4+k3×r6)+2×p1×x×y+p2×(r2+2x2)
v'＝y×(1+k1×r2+k2×r4+k3×r6)+2×p2×x×y+p2×(r2+2y2)
through the inverse process of the process, the coordinates in the image coordinate system of the corresponding ideal camera can be deduced from the coordinates in the image coordinate system of the distorted entity camera, and the purposes of distortion removal of the position point single point and distance measurement error correction are achieved.
Based on any of the above embodiments, in the method, determining the physical coordinates of the position point in the physical coordinate system according to the coordinates of the position point in the image coordinate system of the ideal camera specifically includes:
the coordinates of the position point in an image coordinate system in an ideal camera are (u, v), u and v are all constants, the physical coordinates of the position point in a physical coordinate system are (x, y), dx and dy respectively represent the length of a pixel in the directions of a horizontal axis and a vertical axis, the unit is mm/pixel, gamma is a distortion factor, the value is 0,
according to the following formula,
the physical coordinates (x, y) of the location point in the physical coordinate system are determined.
Specifically, obtaining the coordinates of the position point in the image coordinate system of the ideal camera cannot directly calculate the distance between the pedestrian and the physical camera, and the coordinates in the coordinate system of the ideal camera need to be converted into the physical coordinates in the physical coordinate system so as to obtain the distance between the pedestrian and the camera in the real world in the following.
The transformation of the coordinates of the location point in the physical coordinate system to the coordinates in the image coordinate system in an ideal camera is as follows:
the coordinates of a known position point in an image coordinate system in an ideal camera are (u, v), u and v are constants, the physical coordinates of the position point in a physical coordinate system are (x, y), x and y are unknown numbers, dx and dy respectively represent the length of a pixel in the directions of a horizontal axis and a vertical axis, the unit is mm/pixel, gamma is a distortion factor, and the numerical value is 0. According to the above formula, it can be deduced that the abscissa and ordinate of the position point in the physical coordinate system are:
based on any one of the embodiments, in the method, the determining the distance between the pedestrian and the camera specifically includes:
the distance between the pedestrian and the camera is D, the longitudinal distance between the pedestrian and the camera is VD, the transverse distance between the pedestrian and the camera is HD, the physical coordinates of the position point in a physical coordinate system are (x, y), x and y are constants, H is the height of the camera, H is a constant, f is the focal length of the camera, f is a constant, and the physical coordinates of the main point in the physical coordinate system are (c'x，c'y)，c'xAnd c'yIn mm, according to the following formula,
a distance D between the pedestrian and the camera is determined.
Specifically, the distance between the pedestrian and the camera is calculated, and the distance is composed of two parts: the longitudinal distance VD and the transverse distance HD are, in popular terms, the longitudinal distance is the front-back distance between the pedestrian and the vehicle, and the transverse distance is the left-right distance between the pedestrian and the vehicle. The camera focal length is f, the unit of f is mm, H is the camera height, the height of H is m, and the physical coordinate of the principal point in the physical coordinate system is (c'x，c'y)，c'xAnd c'yIs in mm, theta is the pitch angle of the lens and the horizontal direction when the camera is installed,
when theta is small, the formula is simplified to
Transverse distance
The actual distance passes
And (4) calculating.
Based on any of the above embodiments, fig. 2 is a schematic structural diagram of a vehicle blind area pedestrian monitoring device according to an embodiment of the present invention. As shown in fig. 2, an embodiment of the present invention provides a vehicle blind area pedestrian monitoring apparatus including: a frame detection unit 210, a distortion removal unit 220, and a monitoring unit 230, wherein,
the frame detection unit 210 is configured to collect a vehicle blind area image, and detect the vehicle blind area image through an image detection technology to obtain frame information of a pedestrian;
the distortion removing unit 220 is configured to determine a position point of the pedestrian according to the frame information of the pedestrian, and perform single-point distortion removing operation on the position point to obtain a coordinate of the position point in an image coordinate system of the ideal camera;
the monitoring unit 230 is configured to determine a physical coordinate of the position point in the physical coordinate system according to a coordinate of the position point in the image coordinate system of the ideal camera, further determine a distance between the pedestrian and the physical camera, and monitor the pedestrian based on the distance.
The device provided by the embodiment of the invention collects the vehicle blind area image, and detects the vehicle blind area image through an image detection technology to obtain the frame information of the pedestrian; determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera; and determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance. Through carrying out the mapping that the distortion is removed to the single pixel of position point when the distortion is removed rather than carrying out the distortion removal to all pixel of whole picture, greatly reduced the calculated amount for vehicle blind area pedestrian's control's real-time has obtained the assurance.
Based on any one of the above embodiments, the apparatus further includes:
and the alarm unit is used for displaying the distance and triggering the alarm of the level corresponding to the distance according to the distance between the pedestrian and the entity camera.
Based on any one of the above embodiments, in the apparatus, the detecting the image by using an image detection technology specifically includes:
and detecting the image by an image detection technology based on Fast-SCNN algorithm.
Based on any of the above embodiments, in the apparatus, the performing single-point distortion removal operation on the position point to obtain the coordinates of the position point in the image coordinate system of the ideal camera specifically includes:
based on internal parameters obtained by calibrating the entity camera, carrying out distortion removal operation on coordinates of the position point in an image coordinate system of the entity camera to obtain coordinates of the position point in the image coordinate system of the ideal camera;
the entity camera is a monocular camera, and the internal parameters comprise a transverse axis focal length, a longitudinal axis focal length, principal point coordinates and a distortion coefficient of the entity camera.
Based on any of the embodiments described above, in the apparatus, the obtaining, by performing a distortion removal operation on coordinates of the position point in an image coordinate system of the actual camera based on the internal parameters obtained by calibrating the vehicle-mounted monocular camera, coordinates of the position point in the image coordinate system of the ideal camera specifically includes:
after the entity camera is calibrated, the obtained internal parameters comprise a horizontal axis focal length fx and a vertical axis focal length f of the camerayPrincipal point coordinates (c)x,cy) And distortion coefficient [ k ]1,k2,p1,p2,k3]；
The coordinates of the position point in the image coordinate system in the actual camera are (u ', v'), u 'and v' are both constant, the coordinates of the position point in the image coordinate system in the ideal camera are (u, v),
according to the following formula,
u'＝x×(1+k1×r2+k2×r4+k3×r6)+2×p1×x×y+p2×(r2+2x2)
v'＝y×(1+k1×r2+k2×r4+k3×r6)+2×p2×x×y+p2×(r2+2y2)
the coordinates (u, v) of the location point in the image coordinate system in the ideal camera are determined.
Based on any embodiment, in the apparatus, the determining physical coordinates of the position point in the physical coordinate system according to the coordinates of the position point in the image coordinate system in the ideal camera specifically includes:
the coordinates of the position point in an image coordinate system in an ideal camera are (u, v), u and v are all constants, the physical coordinates of the position point in a physical coordinate system are (x, y), dx and dy respectively represent the length of a pixel in the directions of a horizontal axis and a vertical axis, the unit is mm/pixel, gamma is a distortion factor, the value is 0,
according to the following formula,
the physical coordinates (x, y) of the location point in the physical coordinate system are determined.
Based on any one of the above embodiments, in the apparatus, the determining a distance between the pedestrian and the camera specifically includes:
the distance between pedestrian and the camera is D, and the longitudinal distance between pedestrian and the camera is VD, and the transverse distance between pedestrian and the camera is HD, the physical coordinate of position point in the physical coordinate system is (x, y), and x and y are all constants, and H is the height of camera, H is the constant, fIs the focal length of the camera, f is a constant, and the physical coordinate of the principal point in the physical coordinate system is (c'x，c'y)，c'xAnd c'yThe unit of (a) is mm,
according to the following formula,
a distance D between the pedestrian and the camera is determined.
Fig. 3 is a schematic entity structure diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 3, the electronic device may include: a processor (processor)301, a communication Interface (communication Interface)302, a memory (memory)303 and a communication bus 304, wherein the processor 301, the communication Interface 302 and the memory 303 complete communication with each other through the communication bus 304. The processor 301 may call a computer program stored on the memory 303 and operable on the processor 301 to perform the vehicle blind spot pedestrian monitoring method provided by the above embodiments, for example, including: acquiring a vehicle blind area image, and detecting the vehicle blind area image through an image detection technology to obtain frame information of pedestrians; determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera; and determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance.
In addition, the logic instructions in the memory 303 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or make a contribution to the prior art, or may be implemented in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to, when executed by a processor, perform the method for monitoring pedestrians in blind areas of a vehicle, which includes: acquiring a vehicle blind area image, and detecting the vehicle blind area image through an image detection technology to obtain frame information of pedestrians; determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera; and determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A vehicle blind area pedestrian monitoring method is characterized by comprising the following steps:
acquiring a vehicle blind area image, and detecting the vehicle blind area image through an image detection technology to obtain frame information of pedestrians;
determining the position point of the pedestrian according to the frame information of the pedestrian, and performing single-point distortion removal operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera;
and determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance.
2. The vehicle blind spot pedestrian monitoring method of claim 1, further comprising:
and displaying the distance, and triggering an alarm of a level corresponding to the distance according to the distance between the pedestrian and the entity camera.
3. The method for monitoring pedestrians in blind areas of a vehicle according to claim 1, wherein the detecting the image by an image detection technology specifically comprises:
and detecting the image by an image detection technology based on Fast-SCNN algorithm.
4. The method for monitoring pedestrians in blind areas of vehicles according to claim 1, wherein the step of performing single-point distortion removal operation on the position points to obtain coordinates of the position points in an image coordinate system of an ideal camera includes:
based on internal parameters obtained by calibrating the entity camera, carrying out distortion removal operation on coordinates of the position point in an image coordinate system of the entity camera to obtain coordinates of the position point in the image coordinate system of the ideal camera;
the entity camera is a monocular camera, and the internal parameters comprise a transverse axis focal length, a longitudinal axis focal length, principal point coordinates and a distortion coefficient of the entity camera.
5. The vehicle blind area pedestrian monitoring method according to claim 4, wherein the obtaining of the coordinates of the position point in the image coordinate system of the ideal camera by performing a distortion removal operation on the coordinates of the position point in the image coordinate system of the actual camera based on the internal parameters obtained by calibrating the vehicle-mounted monocular camera specifically comprises:
after the entity camera is calibrated, the obtained internal parameters comprise a horizontal axis focal length fx and a vertical axis focal length f of the camerayPrincipal point coordinates (c)x,cy) And distortion coefficient [ k ]1,k2,p1,p2,k3]；
The coordinates of the position point in the image coordinate system in the actual camera are (u ', v'), u 'and v' are both constant, the coordinates of the position point in the image coordinate system in the ideal camera are (u, v),
according to the following formula,
u'＝x×(1+k1×r2+k2×r4+k3×r6)+2×p1×x×y+p2×(r2+2x2)
v'＝y×(1+k1×r2+k2×r4+k3×r6)+2×p2×x×y+p2×(r2+2y2)
the coordinates (u, v) of the location point in the image coordinate system in the ideal camera are determined.
6. The vehicle blind spot pedestrian monitoring method according to claim 5, wherein the determining the physical coordinates of the position point in the physical coordinate system according to the coordinates of the position point in the image coordinate system in the ideal camera specifically comprises:
the coordinates of the position point in an image coordinate system in an ideal camera are (u, v), u and v are all constants, the physical coordinates of the position point in a physical coordinate system are (x, y), dx and dy respectively represent the length of a pixel in the directions of a horizontal axis and a vertical axis, the unit is mm/pixel, gamma is a distortion factor, the value is 0,
according to the following formula,
the physical coordinates (x, y) of the location point in the physical coordinate system are determined.
7. The vehicle blind spot pedestrian monitoring method according to claim 5, wherein the determining a distance between the pedestrian and the camera specifically comprises:
the distance between the pedestrian and the camera is D, the longitudinal distance between the pedestrian and the camera is VD, the transverse distance between the pedestrian and the camera is HD, the physical coordinates of the position point in a physical coordinate system are (x, y), x and y are constants, H is the height of the camera, H is a constant, f is the focal length of the camera, f is a constant, and the physical coordinates of the main point in the physical coordinate system are (c'x，c'y)，
According to the following formula,
a distance D between the pedestrian and the camera is determined.
8. A vehicle blind spot pedestrian monitoring apparatus, comprising:
the frame detection unit is used for collecting the vehicle blind area images and detecting the vehicle blind area images through an image detection technology to obtain frame information of pedestrians;
the distortion removing unit is used for determining the position point of the pedestrian according to the frame information of the pedestrian and carrying out single-point distortion removing operation on the position point to obtain the coordinate of the position point in an image coordinate system in an ideal camera;
and the monitoring unit is used for determining the physical coordinates of the position points in the physical coordinate system according to the coordinates of the position points in the image coordinate system of the ideal camera, further determining the distance between the pedestrian and the entity camera, and monitoring the pedestrian based on the distance.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the program, carries out the steps of the vehicle blind spot pedestrian monitoring method as claimed in any one of claims 1 to 7.
10. A non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the vehicle blind spot pedestrian monitoring method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911096956.5A CN110929606A (en) | 2019-11-11 | 2019-11-11 | Vehicle blind area pedestrian monitoring method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911096956.5A CN110929606A (en) | 2019-11-11 | 2019-11-11 | Vehicle blind area pedestrian monitoring method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110929606A true CN110929606A (en) | 2020-03-27 |
Family
ID=69853809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911096956.5A Pending CN110929606A (en) | 2019-11-11 | 2019-11-11 | Vehicle blind area pedestrian monitoring method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110929606A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111127540A (en) * | 2019-12-25 | 2020-05-08 | 珠海市四维时代网络科技有限公司 | Automatic distance measurement method and system for three-dimensional virtual space |
| CN112633258A (en) * | 2021-03-05 | 2021-04-09 | 天津所托瑞安汽车科技有限公司 | Target determination method, device and equipment and computer readable storage medium |
| CN113239912A (en) * | 2021-07-13 | 2021-08-10 | 天津所托瑞安汽车科技有限公司 | Method, device and storage medium for determining BSD image effective area |
| CN113887457A (en) * | 2021-10-11 | 2022-01-04 | 杭州鸿泉物联网技术股份有限公司 | Blind zone pedestrian detection method and system |
| CN115205580A (en) * | 2022-06-02 | 2022-10-18 | 陕西师范大学 | Multi-branch residual convolution neural network model and image classification method thereof |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105574552A (en) * | 2014-10-09 | 2016-05-11 | 东北大学 | Vehicle ranging and collision early warning method based on monocular vision |
| CN107985200A (en) * | 2017-11-13 | 2018-05-04 | 西南交通大学 | A kind of load truck right-hand bend safety pre-warning system and method |
| CN109334563A (en) * | 2018-08-31 | 2019-02-15 | 江苏大学 | An anti-collision warning method based on pedestrians and cyclists in front of the road |
| CN110203210A (en) * | 2019-06-19 | 2019-09-06 | 厦门金龙联合汽车工业有限公司 | A kind of lane departure warning method, terminal device and storage medium |
| CN110297232A (en) * | 2019-05-24 | 2019-10-01 | 合刃科技（深圳）有限公司 | Monocular distance measuring method, device and electronic equipment based on computer vision |
| CN110378837A (en) * | 2019-05-16 | 2019-10-25 | 四川省客车制造有限责任公司 | Object detection method, device and storage medium based on fish-eye camera |
-
2019
- 2019-11-11 CN CN201911096956.5A patent/CN110929606A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105574552A (en) * | 2014-10-09 | 2016-05-11 | 东北大学 | Vehicle ranging and collision early warning method based on monocular vision |
| CN107985200A (en) * | 2017-11-13 | 2018-05-04 | 西南交通大学 | A kind of load truck right-hand bend safety pre-warning system and method |
| CN109334563A (en) * | 2018-08-31 | 2019-02-15 | 江苏大学 | An anti-collision warning method based on pedestrians and cyclists in front of the road |
| CN110378837A (en) * | 2019-05-16 | 2019-10-25 | 四川省客车制造有限责任公司 | Object detection method, device and storage medium based on fish-eye camera |
| CN110297232A (en) * | 2019-05-24 | 2019-10-01 | 合刃科技（深圳）有限公司 | Monocular distance measuring method, device and electronic equipment based on computer vision |
| CN110203210A (en) * | 2019-06-19 | 2019-09-06 | 厦门金龙联合汽车工业有限公司 | A kind of lane departure warning method, terminal device and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| KI-YEONG P ET AL: "Robust Range Estimation with a Monocular Camera for Vision-Based Forward Collision Warning System", THE SCIENTIFIC WORLD JOURNAL * |
| KI-YEONG P: "Robust Range Estimation with a Monocular Camera for Vision-Based Forward Collision Warning System", 《THE SCIENTIFIC WORLD JOURNAL》 * |
| POUDEL R ET AL: "Fast-SCNN:Fast Semantic Segmentation Network", ARXIV:1902.04502V1[CS.CV] * |
| POUDEL R: "Fast-SCNN:Fast Semantic Segmentation Network", 《ARXIV:1902.04502V1[CS.CV]》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111127540A (en) * | 2019-12-25 | 2020-05-08 | 珠海市四维时代网络科技有限公司 | Automatic distance measurement method and system for three-dimensional virtual space |
| CN112633258A (en) * | 2021-03-05 | 2021-04-09 | 天津所托瑞安汽车科技有限公司 | Target determination method, device and equipment and computer readable storage medium |
| CN112633258B (en) * | 2021-03-05 | 2021-05-25 | 天津所托瑞安汽车科技有限公司 | Target determination method and device, electronic equipment and computer readable storage medium |
| WO2022183682A1 (en) * | 2021-03-05 | 2022-09-09 | 天津所托瑞安汽车科技有限公司 | Target determination method and apparatus, electronic device, and computer-readable storage medium |
| CN113239912A (en) * | 2021-07-13 | 2021-08-10 | 天津所托瑞安汽车科技有限公司 | Method, device and storage medium for determining BSD image effective area |
| CN113887457A (en) * | 2021-10-11 | 2022-01-04 | 杭州鸿泉物联网技术股份有限公司 | Blind zone pedestrian detection method and system |
| CN113887457B (en) * | 2021-10-11 | 2025-07-29 | 杭州鸿泉物联网技术股份有限公司 | Blind area pedestrian detection method and system |
| CN115205580A (en) * | 2022-06-02 | 2022-10-18 | 陕西师范大学 | Multi-branch residual convolution neural network model and image classification method thereof |
| CN115205580B (en) * | 2022-06-02 | 2026-03-06 | 陕西师范大学 | Multi-branch residual convolutional neural network model and its image classification method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112349144B (en) | Monocular vision-based vehicle collision early warning method and system | |
| CN110929606A (en) | Vehicle blind area pedestrian monitoring method and device | |
| CN102197418B (en) | Vehicle Surrounding Monitoring Device | |
| CN107985200B (en) | A safety warning method for right-turning of a heavy truck | |
| CN101844542A (en) | Intelligent vehicle driving safety auxiliary system | |
| CN112001208B (en) | Target detection method and device for vehicle blind area and electronic equipment | |
| CN103858156A (en) | Vehicle vicinity monitoring device | |
| CN111351474B (en) | Vehicle moving target detection method, device and system | |
| CN114898325B (en) | Vehicle dangerous lane change detection method and device and electronic equipment | |
| CN111967396A (en) | Processing method, device and equipment for obstacle detection and storage medium | |
| CN116486351B (en) | Driving early warning method, device, equipment and storage medium | |
| CN110991264A (en) | Front vehicle detection method and device | |
| CN105336217A (en) | Driving safety prewarning system based on machine vision and Android platform | |
| CN105235680A (en) | Pedestrian avoiding method and system based on cloud computing | |
| CN113246859B (en) | Electronic rearview mirror with driving auxiliary system warning function | |
| JP4752158B2 (en) | Environment complexity calculation device, environment recognition degree estimation device and obstacle alarm device | |
| CN115909261B (en) | Vehicle control method, device, computer device and storage medium | |
| CN115830563B (en) | Vehicle wading state determining method, device, equipment and storage medium | |
| CN111332306A (en) | Traffic road perception auxiliary driving early warning device based on machine vision | |
| CN115761668A (en) | Camera stain recognition method, device, vehicle and storage medium | |
| CN115063757A (en) | Method and device for identifying vehicle falling into water | |
| CN111422190B (en) | Forward collision early warning method and system for rear car loader | |
| CN118597126B (en) | Vehicle lateral collision control method and device, electronic equipment and storage medium | |
| CN110556024B (en) | Anti-collision auxiliary driving method and system and computer readable storage medium | |
| CN115497070A (en) | Method, system and storage medium for identifying moving object on road at night |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230731 Address after: No.508, tianzihu Avenue, tianzihu Town, Anji County, Huzhou City, Zhejiang Province Applicant after: ZHEJIANG HONGQUAN ELECTRONIC TECHNOLOGY Co.,Ltd. Address before: 310030 4th floor, building 2, No.3 Xiyuan 6th Road, Sandun Town, Xihu District, Hangzhou City, Zhejiang Province Applicant before: ZHEJIANG HONGQUAN VEHICLE NETWORK CO.,LTD. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200327 |