CN117950399A - An automatic obstacle avoidance decision method and system based on multimodal knowledge graph - Google Patents

Abstract

The invention discloses an automatic obstacle avoidance decision method and system based on a multi-mode knowledge graph, wherein the method comprises the following steps: constructing an automatic obstacle avoidance traffic environment multi-mode knowledge graph according to the entity, entity attribute and relation among the entity of the unmanned automobile, traffic information and obstacle bodies; according to different scenes and requirements, semantic information of three aspects of obstacle states, unmanned automobile states and traffic rules is obtained, semantic information features are obtained through information processing, and the semantic information features are respectively fused into feature vectors by using a neural network; and constructing the association between the feature vector and the obstacle avoidance decision, taking the feature vector under different scenes as input and the obstacle avoidance decision suggestion as output, training an obstacle avoidance decision neural network model, and forming a decision system. According to the invention, the sensor data of the vehicle is deeply analyzed by utilizing the multi-mode knowledge graph, and the high-efficiency and accurate automatic obstacle avoidance decision is realized by combining the real-time state information of the vehicle.

Description

An automatic obstacle avoidance decision method and system based on multimodal knowledge graph

Technical Field

The present invention relates to the technical field of driverless vehicles, and in particular to an automatic obstacle avoidance decision method and system based on a multimodal knowledge graph.

Background technique

The development and application of driverless car technology has brought new solutions to modern transportation. However, how to ensure the safety and reliability of driverless cars during driving, especially in the face of complex and changing traffic environments, is still a difficult problem to be solved. The automatic obstacle avoidance technology of driverless cars refers to enabling the car to timely identify, predict and avoid static or dynamic obstacles that may hinder its passage based on the environmental information obtained by the sensor during driving, so as to safely reach the destination. The automatic obstacle avoidance technology of driverless cars mainly includes three processes: moving obstacle detection, moving obstacle collision trajectory prediction and moving obstacle avoidance. Among them, moving obstacle detection refers to the detection of moving obstacles in the environment during the movement process, which is mainly completed by the on-board environmental perception system. Commonly used sensors include laser radar, millimeter wave radar, stereo vision, infrared sensor, ultrasonic sensor, etc. Different sensors have their own advantages and disadvantages, and generally need to be used in combination or in conjunction with other sensors to improve the accuracy and robustness of detection. Moving obstacle collision trajectory prediction refers to the possibility rating and prediction of obstacles that may be encountered during the movement, and judging the collision relationship with the driverless vehicle. Commonly used methods include probability model-based, data-driven, and deep learning-based. This step needs to consider many factors, such as the type, speed, direction, acceleration, etc. of obstacles, as well as the state of the vehicle itself and changes in the surrounding environment. Moving obstacle avoidance refers to the safe obstacle avoidance of unmanned vehicles through intelligent decision-making and path planning. It is executed by the vehicle path decision system. Commonly used methods include potential field method, fuzzy logic method, neural network method, occupancy grid method, spatial search method, etc. This step requires generating a collision-free path from the starting state to the target state under the constraints of safety, feasibility and optimality.

At present, the automatic obstacle avoidance technology of driverless cars still has the following problems: (1) Sensor data has interference factors such as noise, occlusion, and distortion, which leads to inaccurate detection of moving obstacles; (2) Data analysis and model prediction have uncertainties and errors, resulting in unreliable prediction of the collision trajectory of moving obstacles; (3) Intelligent decision-making and path planning have complexity and limitations, resulting in inefficient obstacle avoidance.

The quality of automatic obstacle avoidance technology directly affects the safety performance of driverless cars. Therefore, it is of great significance to develop a fast, accurate and efficient automatic obstacle avoidance decision method. Existing automatic obstacle avoidance technology mainly relies on sensor data, such as radar, camera and laser sensor. These sensors can provide real-time road information, but there is also the possibility of incomplete information acquisition and misjudgment. In addition, traditional obstacle avoidance decision methods mainly rely on preset rules or algorithms and cannot make adaptive decisions based on actual conditions. In the application of multimodal knowledge graphs in the automatic obstacle avoidance technology of driverless cars, there are still the following problems: (1) How to extract, integrate and represent multimodal knowledge from multiple data sources, how to ensure the quality and consistency of multimodal knowledge, how to effectively store and manage multimodal knowledge graphs, etc. (2) How to use multimodal knowledge graphs for multimodal semantic understanding, reasoning and query, how to evaluate the effect and value of multimodal knowledge graphs, how to solve the security and privacy issues of multimodal knowledge graphs, etc. (3) How to effectively integrate multimodal knowledge graphs with sensor data such as lidar, vision, IMU, GPS, etc., how to use multimodal knowledge graphs to provide rich semantic information, multi-source data fusion and intelligent decision support, how to use deep learning and other methods to achieve end-to-end automatic obstacle avoidance decisions, etc. Therefore, it is urgent to propose a new automatic obstacle avoidance decision method based on multimodal knowledge graphs, which is suitable for driverless cars and effectively solves the problems of inaccurate sensor detection, erroneous data analysis, and inefficient obstacle avoidance decision-making in existing technologies.

Summary of the invention

In view of this, it is necessary to provide an automatic obstacle avoidance decision-making method and system based on multimodal knowledge graph to address the above-mentioned problems, use multimodal knowledge graph to conduct in-depth analysis of vehicle sensor data, and combine the vehicle's real-time status information to achieve efficient and accurate automatic obstacle avoidance decisions.

To achieve the above object, the present invention is implemented according to the following technical solutions:

An automatic obstacle avoidance decision method based on a multimodal knowledge graph comprises the following steps:

Step S1: construct a multimodal knowledge graph of the automatic obstacle avoidance traffic environment based on the entities, entity attributes and relationships between the three entities of driverless cars, traffic information and obstacles;

Step S2: According to different scenarios and requirements, the semantic information of obstacle status, driverless car status and traffic rules is obtained, and semantic information features are obtained through information processing. The above semantic information features are respectively fused into feature vectors by using a neural network;

Step S3: Construct the association between the feature vector and the obstacle avoidance decision, take the feature vectors in different scenarios as input and the obstacle avoidance decision suggestions as output, train the obstacle avoidance decision neural network model, and form a decision system.

Furthermore, in step S1, the unmanned vehicle body takes itself as an entity; physical parameters describing the motion state of the entity are used as entity attributes, including but not limited to position, speed, direction, and acceleration; the relationship between entities includes but is not limited to relative position, relative speed, relative direction, and relative acceleration.

Furthermore, in step S1, the entities of the traffic information entity include but are not limited to roads, traffic signs, traffic lights and other entities; the attributes of the road entity include but are not limited to type, size, number of lanes, and road surface material; the attributes of the traffic sign entity include but are not limited to type, location, size, and meaning; the attributes of the traffic light entity include but are not limited to type, color, value, and direction.

Furthermore, in step S1, the obstacle body includes static entities and dynamic entities; static entities include but are not limited to roadblocks, stones, branches and other stationary objects, and their attributes include but are not limited to type, size, position, direction, and mass; dynamic entities include but are not limited to pedestrians, animals, vehicles and other moving objects, and their attributes include but are not limited to type, size, position, speed, direction, mass, and acceleration.

Furthermore, in step S2, the specific steps of obtaining semantic information include: for the monitored obstacles, according to their type, whether static or dynamic, extracting relevant attribute features in the knowledge graph as semantic information of the obstacle state; collecting data through sensors to obtain semantic information of the state of the driverless car, including but not limited to position, speed, direction, and acceleration; identifying traffic rules for the road section where the obstacle appears, and forming semantic information of traffic rules, including but not limited to traffic signs, traffic lights, and road conditions.

Furthermore, in step S2, the semantic information includes but is not limited to text data, sensor signal data, and video image data.

Furthermore, in step S2, the specific steps of the information processing include: encoding text data and sensor signal data, performing feature recognition and information extraction on image data, and obtaining semantic information features of three aspects: obstacle status, driverless car status, and traffic rules.

Furthermore, in step S2, the specific steps of fusing into a feature vector include:

Step S21: Perform entity extraction and relationship extraction on the text data through natural language processing technology to obtain entities of specific objects, semantic relationships between them, and attributes of entities;

Step S22: Encode/decode the sensor signal data to obtain a value corresponding to a specific entity and use it as the attribute value of the entity attribute;

Step S23: extracting entities and corresponding semantic relationships in the video/image using image processing technology for the video image data;

Step S24: The entities, relationships, and attribute values obtained from steps S21 to S23 are fused into feature vectors using a neural network to obtain a driverless car feature vector, an obstacle feature vector, and a traffic environment feature vector.

Furthermore, in step S3, the feature vectors in different scenarios include but are not limited to the driverless car feature vector, the obstacle feature vector, and the traffic environment feature vector; the obstacle avoidance decision suggestions include but are not limited to the turning angle and the braking force.

In addition, the present invention also provides an automatic obstacle avoidance decision system based on a multimodal knowledge graph, which is used to execute the automatic obstacle avoidance decision method based on a multimodal knowledge graph.

The present invention improves the existing automatic obstacle avoidance technology, uses a multimodal knowledge graph to conduct in-depth analysis of the vehicle's sensor data, and combines it with the vehicle's real-time status information to achieve efficient and accurate automatic obstacle avoidance decisions, solving the problems of inaccurate sensor detection, erroneous data analysis, and inefficient obstacle avoidance decisions in the prior art.

Compared with the prior art, the present invention has at least the following advantages and positive effects:

1. A new method for constructing a multimodal knowledge graph is proposed. It extracts, integrates and represents multimodal knowledge from multiple data sources. It uses preprocessing, analysis, feature extraction, entity extraction, attribute extraction, relationship extraction and other technologies to generate a directed graph or multi-directed graph consisting of entities, attributes and relationships. Entities and attributes can be represented by text or images, and relationships can be represented by directions or values. The multimodal knowledge graph provides rich semantic information, multi-source data fusion and intelligent decision support, thereby improving the accuracy and robustness of moving obstacle detection, collision trajectory prediction and obstacle avoidance path planning.

2. Extract knowledge and features related to driverless cars and moving obstacles from the multimodal knowledge graph using semantic understanding, reasoning, query, retrieval, feature extraction and representation techniques, and input them into a deep neural network. Use deep neural networks to achieve end-to-end automatic obstacle avoidance decisions, and improve the efficiency and flexibility of moving obstacle detection, collision trajectory prediction, and obstacle avoidance path planning.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic diagram of the workflow of the automatic obstacle avoidance decision-making method of the present invention;

FIG2 is a schematic diagram of a multimodal knowledge graph of an automatic obstacle avoidance traffic environment in Example 1 of the present invention;

FIG3 is a schematic diagram of a main body model of an unmanned vehicle in Embodiment 1 of the present invention;

FIG4 is a schematic diagram of a traffic information ontology model in Embodiment 1 of the present invention;

FIG5 is a schematic diagram of an obstacle body model in Example 1 of the present invention;

FIG6 is a schematic diagram of the data collection and knowledge graph construction process in Example 1 of the present invention;

FIG7 is a schematic diagram of a process of knowledge acquisition and feature extraction based on a scene-based knowledge graph in Example 1 of the present invention;

FIG8 is a schematic diagram of an automatic obstacle avoidance decision model based on a multimodal knowledge graph in Example 1 of the present invention.

Detailed ways

In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be pointed out that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Example 1

This embodiment is a specific description of the automatic obstacle avoidance decision method based on the multimodal knowledge graph of the present invention. It should be noted that the specific description described in this embodiment is only a set of possible or preferred combinations used in this embodiment, but it cannot be understood as a limitation on the scope of the patent of the present invention; it should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, several variations and improvements can be made, which all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be based on the attached claims.

FIG1 shows a schematic diagram of a workflow of an automatic obstacle avoidance decision method based on a multimodal knowledge graph. As shown in FIG1 , the present invention provides an automatic obstacle avoidance decision method based on a multimodal knowledge graph, comprising the following steps:

Step S1: construct a multimodal knowledge graph of the automatic obstacle avoidance traffic environment based on the entities, entity attributes and relationships between the three entities of driverless cars, traffic information and obstacles;

Step S2: According to different scenarios and requirements, the semantic information of obstacle status, driverless car status and traffic rules is obtained, and semantic information features are obtained through information processing. The above semantic information features are respectively fused into feature vectors by using a neural network;

Step S3: Construct the association between the feature vector and the obstacle avoidance decision, take the feature vectors in different scenarios as input and the obstacle avoidance decision suggestions as output, train the obstacle avoidance decision neural network model, and form a decision system.

FIG2 shows a schematic diagram of the multimodal knowledge graph of the automatic obstacle avoidance traffic environment. It can be seen that in step S1, the mode layer of the knowledge graph includes three interrelated ontologies, namely, driverless cars, traffic information, and obstacles. These three ontologies have their own entities, attributes, and relationships, forming an interrelated directed graph. In this embodiment, the "directed" can be understood as: the specific relationship between different entities determines the direction of the arrow in the knowledge graph; if the entity has attributes, the direction at this time is that the entity points to the attribute.

FIG3 shows a schematic diagram of the driverless car body model. As shown in FIG3 , in step S1, the driverless car body itself is an entity; physical parameters describing the motion state of the entity are used as entity attributes, including but not limited to position, speed, direction, and acceleration; the relationship between entities includes but is not limited to relative position, relative speed, relative direction, and relative acceleration.

Figure 4 shows a schematic diagram of the traffic information ontology model. As shown in Figure 4, in step S1, the entities of the traffic information ontology include but are not limited to roads, traffic signs, traffic lights and other entities; the attributes of the road entity include but are not limited to type, size, number of lanes, and road surface material; the attributes of the traffic sign entity include but are not limited to type, location, size, and meaning; the attributes of the traffic light entity include but are not limited to type, color, value, and direction.

FIG5 shows a schematic diagram of the obstacle entity model. As shown in FIG5 , in step S1, the obstacle entity includes a static entity and a dynamic entity; the static entity includes but is not limited to roadblocks, stones, branches and other stationary objects, and its attributes include but are not limited to type, size, position, direction, and mass; the dynamic entity includes but is not limited to pedestrians, animals, vehicles and other moving objects, and its attributes include but are not limited to type, size, position, speed, direction, mass, and acceleration.

The ontology model described in Figures 2 to 5 is the mode layer of the multimodal knowledge graph of the automatic obstacle avoidance traffic environment, in which the specific values and graphic information of the attributes are obtained by sensors such as cameras, laser radars, millimeter wave radars, GPS (global positioning system), and IMU (inertial measurement unit). The laser radar data can provide three-dimensional point cloud information in the environment, the visual data can provide two-dimensional image information in the environment, the GPS data can provide the location information of the driverless car, and the IMU data can provide the posture information of the driverless car. Among them, the picture or video is associated with the attribute value of the driverless car entity in a linked manner.

Figure 6 shows a possible data collection and knowledge graph construction process diagram. As shown in Figure 6, we can extract the attributes of entities such as driverless cars, obstacles, traffic information, and the relationship between them from various sensor data of driverless cars to build a multimodal knowledge graph. Since knowledge graphs can provide rich semantic information, multi-source data fusion, and intelligent decision support to assist in the formulation of obstacle avoidance decisions and improve the accuracy and robustness of obstacle detection, collision trajectory prediction, and obstacle avoidance path planning, this way of building knowledge graphs can directly utilize the richness and diversity of multimodal data, but it also needs to solve the fusion and representation problems between multimodal data.

FIG7 shows a possible schematic diagram of the knowledge acquisition and feature extraction process of a scenario-based knowledge graph. As shown in FIG7 , in step S2, according to different scenarios and requirements, we can obtain semantic information on obstacle status, driverless car status, and traffic rules, obtain semantic information features through information processing, and use neural networks to fuse the above semantic information features into feature vectors.

The specific steps of obtaining semantic information include: for monitored obstacles, according to their types, whether they are static or dynamic, extracting relevant attribute features in the knowledge graph as semantic information of the obstacle state; collecting data through sensors to obtain semantic information of the state of the driverless car, including but not limited to position, speed, direction, and acceleration; identifying traffic rules on the road section where the obstacle appears, and forming semantic information of traffic rules, including but not limited to traffic signs, traffic lights, and road conditions.

The sources of semantic information in these three aspects include text data, sensor signal data, video image data, etc. By encoding text data and sensor signal data, and performing feature recognition and information extraction on video image data, semantic information features of obstacle status, driverless car status, and traffic rules are obtained. Then, neural networks are used to fuse these three features into feature vectors as the basis for the decision-making system. Among them, text data uses natural language processing technology to extract entities and semantic relationships in sentences; the sensor returns numerical signals, which are converted into understandable numerical values after encoding, as the attribute values of the corresponding entities.

Specifically, the steps of data processing and feature vector fusion in step S2 include:

For text data, we use natural language processing technology to extract entities and relationships, and obtain the entities of specific objects, their semantic relationships, and the attributes of entities. Specific objects include obstacles, vehicles, pedestrians, etc.

The sensor signal data is encoded/decoded to obtain the value corresponding to the specific entity, and used as the attribute value of the entity attribute. The aforementioned specific object can be used as an entity, and the specific entity here includes an entity that can obtain data using a sensor, and the data measured by the sensor, such as speed, acceleration, position, etc., is used as the attribute value of the entity.

For video image data, image processing technology is used to extract entities in the video/image and their corresponding semantic relationships.

The entities, relationships, and attribute values obtained in the above steps are fused into feature vectors using a neural network to obtain the driverless car feature vector, obstacle feature vector, and traffic environment feature vector.

After obtaining the fused feature vector, we can start to build an automatic obstacle avoidance decision system. First, we build the association between the feature vector and the obstacle avoidance decision, and then train the obstacle avoidance decision model to form a decision system.

Figure 8 shows a possible schematic diagram of an automatic obstacle avoidance decision model based on a multimodal knowledge graph. As shown in Figure 8, the feature vectors under different obstacle states, including the state of the obstacle, the state of the driverless car, and the traffic environment, are used as inputs; obstacle avoidance decision suggestions, such as turning angle and braking force, are used as outputs to train the obstacle avoidance decision deep neural network. By utilizing the rich semantic information provided by the multimodal knowledge graph, the feature vectors of multi-source data fusion, and the obstacle avoidance decision scheme, the vehicle adjusts its own driving state according to the characteristics of different scenarios, avoids various static or dynamic obstacles encountered during the movement, and realizes end-to-end automatic obstacle avoidance decision.

In summary, based on the above steps S1 to S3, the present invention improves the existing automatic obstacle avoidance technology, uses a multimodal knowledge graph to perform in-depth analysis of the vehicle's sensor data, and combines the vehicle's real-time status information to achieve efficient and accurate automatic obstacle avoidance decisions, solving the problems of inaccurate sensor detection, erroneous data analysis, and inefficient obstacle avoidance decision-making in the prior art.

Example 2

This embodiment is an explanation of the automatic obstacle avoidance decision system based on the multimodal knowledge graph of the present invention.

Correspondingly, based on the above-mentioned automatic obstacle avoidance decision method based on multimodal knowledge graph, this embodiment also provides an automatic obstacle avoidance decision system based on multimodal knowledge graph, which is used to execute the automatic obstacle avoidance decision method based on multimodal knowledge graph described in Example 1.

Compared with the prior art, the present invention has at least the following advantages and positive effects:

1. A new method for constructing a multimodal knowledge graph is proposed. It extracts, integrates and represents multimodal knowledge from multiple data sources. It uses preprocessing, analysis, feature extraction, entity extraction, attribute extraction, relationship extraction and other technologies to generate a directed graph or multi-directed graph consisting of entities, attributes and relationships. Entities and attributes can be represented by text or images, and relationships can be represented by directions or values. The multimodal knowledge graph provides rich semantic information, multi-source data fusion and intelligent decision support, thereby improving the accuracy and robustness of moving obstacle detection, collision trajectory prediction and obstacle avoidance path planning.

2. Extract knowledge and features related to driverless cars and moving obstacles from the multimodal knowledge graph using semantic understanding, reasoning, query, retrieval, feature extraction and representation techniques, and input them into a deep neural network. Use deep neural networks to achieve end-to-end automatic obstacle avoidance decisions, and improve the efficiency and flexibility of moving obstacle detection, collision trajectory prediction, and obstacle avoidance path planning.

The above-mentioned embodiments only express one or several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims (10)

1. An automatic obstacle avoidance decision method based on a multi-mode knowledge graph is characterized by comprising the following steps:

step S1: constructing an automatic obstacle avoidance traffic environment multi-mode knowledge graph according to the entity, entity attribute and relation among the entity of the unmanned automobile, traffic information and obstacle bodies;

Step S2: according to different scenes and requirements, semantic information of three aspects of obstacle states, unmanned automobile states and traffic rules is obtained, semantic information features are obtained through information processing, and the semantic information features are respectively fused into feature vectors by using a neural network;

Step S3: and constructing the association between the feature vector and the obstacle avoidance decision, taking the feature vector under different scenes as input and the obstacle avoidance decision suggestion as output, training an obstacle avoidance decision neural network model, and forming a decision system.

2. The method for automatically avoiding obstacle decision based on multi-modal knowledge graph according to claim 1, wherein in step S1, the unmanned automobile body takes itself as an entity; physical parameters describing the physical motion state of the entity are taken as physical attributes, including but not limited to position, speed, direction and acceleration; relationships between entities include, but are not limited to, relative position, relative velocity, relative direction, relative acceleration.

3. The method of claim 1, wherein in step S1, the traffic information ontology entities include, but are not limited to, roads, traffic signs, traffic lights, and other entities; attributes of road entities include, but are not limited to, type, size, number of lanes, road surface texture; attributes of traffic sign entities include, but are not limited to, type, location, size, meaning; attributes of traffic light entities include, but are not limited to, type, color, value, direction.

4. The method for automatically avoiding obstacle decision based on multi-modal knowledge-graph according to claim 1, wherein in step S1, the obstacle body includes a static entity and a dynamic entity; static entities include, but are not limited to, barriers, stones, branches, and other stationary objects, the attributes of which include, but are not limited to, category, size, location, orientation, mass; dynamic entities include, but are not limited to, pedestrians, animals, vehicles, and other moving objects, whose attributes include, but are not limited to, category, size, position, speed, direction, mass, acceleration.

5. The method for automatically avoiding obstacle decision based on multi-modal knowledge-graph according to claim 1, wherein in step S2, the specific step of obtaining semantic information includes: for the monitored obstacle, extracting relevant attribute features from the knowledge graph according to the static or dynamic type of the obstacle, and taking the relevant attribute features as semantic information of the obstacle state; collecting data through a sensor to obtain semantic information of the unmanned automobile state, including but not limited to position, speed, direction and acceleration; traffic rules of road sections where obstacles appear are identified, and traffic rule semantic information is formed, including but not limited to traffic signs, traffic lights, and road conditions.

6. The automated obstacle avoidance decision method based upon multimodal knowledge-graph of claim 1 wherein in step S2, the semantic information includes, but is not limited to, text data, sensor signal data, video image data.

7. The method for automatically avoiding obstacle decision based on multi-modal knowledge-graph according to claim 6, wherein in step S2, the specific steps of information processing include: and encoding the text data and the sensor signal data, and carrying out feature recognition and information extraction on the image data to obtain semantic information features of three aspects of obstacle states, unmanned automobile states and traffic rules.

8. The method for automatically avoiding obstacle decision based on multi-modal knowledge-graph according to claim 6, wherein in step S2, the specific step of merging into feature vectors comprises:

step S21: extracting the entity and the relation of the text data by a natural language processing technology to obtain the entity, the semantic relation among the entities and the attribute of the entity of the specific object;

Step S22: encoding/decoding the sensor signal data to obtain a numerical value corresponding to a specific entity, and taking the numerical value as an attribute value of the attribute of the entity;

Step S23: extracting entities in the video/image and corresponding semantic relations by an image processing technology aiming at video image data;

Step S24: and (3) fusing the entity, the relation and the attribute values obtained in the steps S21 to S23 into feature vectors by utilizing a neural network to obtain unmanned automobile feature vectors, barrier feature vectors and traffic environment feature vectors.

9. The method for automatically avoiding obstacle decision based on multi-modal knowledge-graph according to claim 1, wherein in step S3, the feature vectors in different scenes include, but are not limited to, unmanned car feature vectors, obstacle feature vectors, traffic environment feature vectors; obstacle avoidance decision advice includes, but is not limited to, angle of rotation, braking effort.

10. An automatic obstacle avoidance decision system based on a multimodal knowledge graph, characterized by being configured to perform the automatic obstacle avoidance decision method based on a multimodal knowledge graph according to any of claims 1 to 9.