JP2025136197A - Information processing device, information processing method, and information processing system - Google Patents

Abstract

To reduce the burden on autonomous vehicle monitors.SOLUTION: An information processing device 1 according to an embodiment comprises: a video receiving unit 151 which receives real-time video obtained from a camera mounted on an autonomously drivable vehicle; an obstacle position receiving unit 152 which receives obstacle position information used to identify the position of an obstacle in a world coordinate system at a first time point at which the vehicle detected the obstacle; a vehicle position receiving unit 153 which receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time point later than the first time point; a position identification unit 155 which identifies the obstacle projection position, which is the projected position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system and the camera coordinate system; and a display processing unit 159 which causes the real-time video indicative of the obstacle projection position indicated to be displayed on a display unit 13.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing device, information processing method, and information processing system for displaying real-time video showing the location of obstacles.

Technology is known that makes it easier to determine whether to resume operation when an autonomous vehicle makes an emergency stop (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2023-145366

After an autonomous vehicle makes an emergency stop, a supervisor remotely monitors the vehicle and determines whether it is OK to resume operation by looking at real-time footage obtained from a camera mounted on the vehicle. However, one supervisor often monitors multiple autonomous vehicles, which creates a problem of a heavy burden on the supervisor.

The present invention was developed in consideration of these points, and aims to reduce the burden on monitors of autonomous vehicles.

An information processing device according to a first aspect of the present invention includes an image receiving unit that receives real-time image data obtained from a camera mounted on an autonomously driven vehicle; an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle; a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time; a position identifying unit that identifies an obstacle projection position, which is the projection position of the obstacle on the real-time image, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and the camera coordinate system, which is the coordinate system of the camera; and a display processing unit that displays the real-time image in which the obstacle projection position is indicated on a display unit.

The video receiving unit may receive an image at the first time from the camera, and the display processing unit may display the image at the first time and the real-time video showing the obstacle projection position on the display unit.

The position identification unit may determine whether the obstacle projection position is included in the imaging range of the real-time video, and the display processing unit may display the real-time video on the display unit in a different manner depending on whether the position identification unit determines that the obstacle projection position is included in the imaging range or not.

When the position identification unit determines that the obstacle projection position is included in the shooting range of the real-time image, the display processing unit may display the real-time image on the display unit, with the obstacle highlighted.

The obstacle position receiving unit may receive feature information indicating features of the obstacle detected at the first time, and the information processing device may further include a determination unit that determines whether the obstacle detected at the first time is a moving object or a stationary object based on the feature information, and the display processing unit may display the real-time video on the display unit in a manner that makes it possible to distinguish whether the obstacle is a moving object or a stationary object.

The obstacle position receiving unit may receive error information indicating an error in the physical position of the obstacle at the first time, and the display processing unit may cause the display unit to display the real-time video in which the obstacle is shown in a different manner depending on the error indicated by the error information.

The error information may be at least one of the accuracy of the vehicle's self-position estimation, the degree of shaking of the vehicle, the distance from the vehicle to the obstacle, and the reliability of the obstacle detection.

The position identification unit may use the vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system at the second time corresponding to the obstacle position indicated by the obstacle position information in the world coordinate system at the first time; identify a second obstacle position, which is a position in the camera coordinate system at the second time corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system; and identify the obstacle projection position by converting the second obstacle position into a position in an image coordinate system corresponding to the camera coordinate system.

The obstacle position receiving unit may receive a plurality of pieces of obstacle position information and a plurality of pieces of accuracy information indicating the detection accuracy of the obstacle at a plurality of times after the first time, and the position identifying unit may select the obstacle position information to be used to identify the obstacle projection position from the plurality of pieces of obstacle position information based on the plurality of pieces of accuracy information at the plurality of times.

The position identification unit may select the obstacle position information at the time when the detection accuracy indicated by the accuracy information is highest.

The obstacle position receiving unit may receive a plurality of pieces of obstacle position information and a plurality of pieces of accuracy information indicating the detection accuracy of the obstacle at a plurality of times after the first time, and the position identifying unit may identify the obstacle projection position on the real-time video based on a weighted average of a plurality of the physical positions corresponding to each of the plurality of pieces of obstacle position information so that the obstacle position information with higher detection accuracy indicated by the accuracy information is reflected to a higher extent in the identification of the obstacle projection position on the real-time video.

The information processing device may further include an input receiving unit that receives input of a driving restart instruction, which is an instruction to resume driving of the vehicle, and a signal transmitting unit that, when the input receiving unit receives input of the driving restart instruction, transmits to the vehicle a driving restart signal, which is a signal to resume driving of the vehicle.

An information processing method according to a second aspect of the present invention is executed by a computer and includes: an image receiving unit that receives real-time image data obtained from a camera mounted on an autonomously driven vehicle; an obstacle receiving step that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle; a vehicle receiving step that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time; a position identifying step that identifies an obstacle projection position, which is the projection position of the obstacle on the real-time image, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and the camera coordinate system, which is the coordinate system of the camera; and a display processing step that displays the real-time image in which the obstacle projection position is indicated on a display unit.

An information processing system according to a third aspect of the present invention comprises an information processing device and a vehicle terminal mounted on a vehicle capable of communicating with the information processing device, wherein the information processing device includes a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle, an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle, a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time after the first time, and a vehicle position receiving unit that receives the obstacle position information, the vehicle position information, and a vehicle position information used to define the position and orientation of the vehicle indicated by the vehicle position information. and a display processing unit that displays the real-time image showing the obstacle projection position on a display unit, using the relative positional relationship between a vehicle coordinate system that is used to project the obstacle and a camera coordinate system that is the coordinate system of the camera. The vehicle terminal has a position information generation unit that generates the vehicle position information, an obstacle detection unit that detects the obstacle and generates the obstacle position information, a transmission unit that transmits the real-time image, the obstacle position information, and the vehicle position information, and a vehicle control unit that stops the vehicle in response to the obstacle detection unit detecting the obstacle, and starts the vehicle in response to receiving a travel resume signal from the information processing device.

The present invention has the effect of reducing the burden on monitors of autonomous vehicles.

FIG. 10 is a diagram showing an example of a real-time image showing an obstacle projection position. FIG. 1 is a diagram showing a configuration of an information processing system S. FIG. 1 is a diagram illustrating an outline of the configuration of an information processing device 1. FIG. 2 is a diagram illustrating an example of the configuration of a vehicle terminal 2. 1 is a diagram showing a configuration of an information processing device 1 according to a first embodiment. 4 is a flowchart showing a processing flow in the information processing device 1 according to the first embodiment. FIG. 10 is a diagram illustrating a configuration of an information processing device 1 according to a second embodiment. 10 is a flowchart showing a processing flow in the information processing device 1 according to the second embodiment. FIG. 10 is a diagram illustrating a configuration of an information processing device 1 according to a third embodiment. 10 is a flowchart showing a processing flow in the information processing device 1 according to the third embodiment.

[Outline of Information Processing System S]
An overview of an information processing system S according to this embodiment will be described using Figures 1 and 2. The information processing system S includes an information processing device 1 and a vehicle terminal 2. The information processing system S may also include other devices such as a server and a terminal. The information processing system S is a system for reducing the burden on a monitor who remotely monitors a vehicle when determining whether the vehicle may resume driving after the vehicle detects an obstacle and makes an emergency stop. The vehicle is, for example, an autonomous vehicle, but may also be a vehicle that is capable of both manual driving by a driver and autonomous driving. The following description will exemplify a case in which the vehicle is an autonomous vehicle.

The information processing device 1 is a computer such as a server that receives real-time video footage obtained from a camera mounted on an autonomous vehicle and displays the received real-time video on a display unit. After an autonomous vehicle makes an emergency stop, a supervisor remotely monitoring the autonomous vehicle determines whether it is OK to resume operation by viewing the real-time video footage obtained from the camera mounted on the autonomous vehicle. However, one supervisor often monitors multiple autonomous vehicles, which creates the problem of a heavy burden on the supervisor.

Therefore, the information processing device 1 receives real-time images obtained from a camera mounted on the autonomous driving vehicle, and displays the real-time images exemplified in (1) to (3) below on the display unit.
(1) Real-time video showing an obstacle projection position, which is the projection position of the obstacle on the real-time video, identified using the position of the obstacle at the time the vehicle detected the obstacle. (2) Real-time video showing the position of an object that has the characteristics of the obstacle detected by the vehicle, among multiple objects present in the real-time video. (3) Real-time video showing, among multiple real-time videos obtained from multiple cameras mounted on the vehicle, images with a relatively large obstacle projection degree, which indicates the degree of projection of the obstacle in the real-time video, in a manner that makes it easy for the observer to identify them.

Figure 1 shows an example of real-time video in which the projection position of an obstacle is displayed. Figure 1 shows an example of real-time video obtained from a camera mounted on a traveling vehicle when the vehicle detects an obstacle and makes an emergency stop on the shoulder of the road. As shown in Figure 1, the cardboard box, which is the obstacle, is shown surrounded by a dotted rectangular frame. In this way, in the information processing system S according to this embodiment, the obstacle in the real-time video is displayed in a manner that is easy for the observer to identify.

The information processing device 1 is connected to multiple vehicle terminals 2 via a communication network such as the Internet. The vehicle terminals 2 are information terminals installed in autonomous vehicles. Note that "autonomous driving" as used in this specification is a concept that encompasses both fully autonomous driving and partial autonomous driving (i.e., a concept that encompasses all autonomous driving levels from Level 1 to 5), but in this embodiment, we are primarily considering Level 4 autonomous driving (driving in which the autonomous driving system performs all driving on behalf of the driver under certain conditions).

An overview of the processing performed by the information processing system S will be explained with reference to Figure 2. Figure 2 is a diagram showing the configuration of the information processing system S. The vehicle V periodically transmits real-time video obtained from a camera mounted on the vehicle V to the information processing device 1.

In this embodiment, when identifying the position of an obstacle in real-time video, a process of coordinate conversion between different coordinate systems is required. For this reason, we will first explain the definitions of each coordinate system used in this specification. The world coordinate system is a coordinate system that defines the entire three-dimensional space, and can also be said to be a coordinate system linked to a 3D map constructed in advance so that the autonomous driving system can define and estimate its own position. The world coordinate system serves as an immutable reference because its origin does not move and its coordinate axes do not rotate.

The vehicle coordinate system is a coordinate system used to define the position and orientation of vehicle V indicated by vehicle position information. Since the vehicle coordinate system is a three-dimensional coordinate system based on an origin within each vehicle V, the origin differs depending on the vehicle V. Vehicle V can estimate the position of obstacles and its own position using LiDAR (Light Detection and Ranging) installed on the vehicle V; in this case, the vehicle coordinate system is the LiDAR coordinate system.

The camera coordinate system is the coordinate system of the camera mounted on vehicle V. The camera coordinate system is a three-dimensional coordinate system based on the camera's origin.

The image coordinate system is a coordinate system based on the origin of the display screen on which the monitor views real-time video. While the world coordinate system, vehicle coordinate system (LiDAR coordinate system), and camera coordinate system described above are three-dimensional coordinate systems, the image coordinate system is a two-dimensional coordinate system with its origin at, for example, the upper left corner of the display screen. In this specification, the image coordinate system is sometimes referred to as the pixel coordinate system.

Returning to the explanation of the processing, vehicle V travels while constantly monitoring its own position and surrounding conditions using the LiDAR installed on vehicle V. Using LiDAR SLAM (Simultaneous Localization and Mapping) technology, for example, vehicle V generates vehicle position information indicating the position and orientation of vehicle V in the world coordinate system by comparing point cloud data indicating the shapes of objects around vehicle V detected by the LiDAR with map data in a world coordinate system created in advance. Point cloud data is data including coordinate data for each of multiple points output by LiDAR.

Using LiDAR SLAM technology, for example, vehicle V compares point cloud data detected by LiDAR with pre-created map data in a world coordinate system to generate vehicle position information indicating vehicle V's position in the world coordinate system. When vehicle V detects an obstacle and makes an emergency stop on the shoulder of the road, for example, vehicle V transmits the generated vehicle position information to information processing device 1.

Furthermore, when vehicle V detects an obstacle, vehicle V uses LiDAR SLAM technology to compare the point cloud data detected by LiDAR with map data in a world coordinate system created in advance. If the point cloud data and map data match by a standard value or more, but some of the point cloud data do not match, vehicle V determines the position of the mismatched point cloud data to be the position of the obstacle. Vehicle V generates obstacle position information that indicates the center position of the area specified by the point cloud data or multiple positions on the contour of the area. Furthermore, when vehicle V detects an obstacle, vehicle V generates feature information that indicates the features of the obstacle detected by vehicle V. Vehicle V transmits the generated obstacle position information and feature information to information processing device 1.

The information processing device 1 receives real-time video periodically transmitted from the vehicle V. The information processing device 1 also receives obstacle position information, vehicle position information, and feature information transmitted from the vehicle V.

When displaying the real-time images (1) and (3) described above, the information processing device 1 converts coordinates indicating the obstacle position in the world coordinate system into coordinates in a two-dimensional image coordinate system as follows. First, the information processing device 1 converts the coordinates of the obstacle position in the world coordinate system into coordinates in the vehicle coordinate system based on the vehicle position information. Next, the information processing device 1 converts the coordinates in the vehicle coordinate system into coordinates in the camera coordinate system. Finally, the information processing device 1 converts the coordinates in the camera coordinate system into coordinates in a two-dimensional image coordinate system. The coordinates in the two-dimensional image coordinate system are the projection position of the obstacle on the real-time image.

Alternatively, when displaying the real-time video of (2) above, the information processing device 1 performs image analysis of the real-time video to detect, from among multiple objects present in the real-time video, an object that has the characteristics indicated by the characteristic information. The position of the detected object is displayed on the real-time video.

The information processing device 1 then displays real-time video so that the observer M can easily identify the location of the obstacle. Once the observer M confirms that the obstacle in the real-time video has been removed and determines that it is okay for the vehicle V to resume traveling, the observer M inputs a traveling resume instruction to the information processing device 1, which is an instruction to cause the vehicle V to resume traveling. Upon receiving the input of the traveling resume instruction, the information processing device 1 transmits a traveling resume signal to the vehicle V, which is a signal to cause the vehicle V to resume traveling. As a result, the vehicle V resumes traveling from the position where it was stopped.

In this way, the information processing device 1 displays the real-time video in a manner that makes it easy for the observer M to identify obstacles in the real-time video, thereby reducing the monitoring burden on the observer M of the vehicle V. As a result, when the vehicle V detects an obstacle and makes an emergency stop, it can quickly resume traveling as soon as it is confirmed that the surrounding conditions of the vehicle V are safe.
The configurations and operations of the information processing device 1 and the vehicle terminal 2 will be described below.

[Configuration and Operation of Information Processing Device 1]
Next, a description will be given of the configuration and operation of the information processing device 1. Fig. 3 is a diagram showing an outline of the configuration of the information processing device 1. The information processing device 1 includes a device communication unit 11, a storage unit 12, a display unit 13, an input unit 14, and a control unit 15.

The device communication unit 11 is a communication interface for communicating with the vehicle terminal 2 via a communication network such as the Internet. The device communication unit 11 receives real-time video, obstacle position information, vehicle position information, and feature information. The device communication unit 11 transmits a travel resume signal.

The memory unit 12 is a storage medium including ROM (Read Only Memory) and RAM (Random Access Memory). The memory unit 12 stores programs executed by the control unit 15.

The display unit 13 is a device for displaying real-time video, such as a display. The input unit 14 is an input device for receiving input from the observer M of an instruction to resume driving, such as a touch screen, keyboard, or mouse.

The control unit 15 is, for example, a CPU (Central Processing Unit). The control unit 15 executes an information processing program stored in the storage unit 12. Details of the processing executed by the control unit will be described later.

[Configuration and Operation of Vehicle Terminal 2]
Next, a description will be given of the configuration and operation of the vehicle terminal 2. Fig. 4 is a diagram showing an example of the configuration of the vehicle terminal 2. The vehicle terminal 2 includes a vehicle communication unit 21, a storage unit 22, a LiDAR 23, a camera 24, and a control unit 25.

The vehicle communication unit 21 is a communication interface for communicating with the information processing device 1 via a communication network such as the Internet. The vehicle communication unit 21 receives a travel resumption signal.

The memory unit 22 is a storage medium including ROM, RAM, etc. The memory unit 22 stores programs executed by the control unit 25. For example, the memory unit 22 stores programs that cause the control unit 25 to function as a position information generation unit 251, an obstacle detection unit 252, a transmission unit 253, and a vehicle control unit 254.

The LiDAR 23 is mounted on the vehicle V. The vehicle V constantly monitors its own position and surrounding conditions by using the LiDAR 23. The vehicle V can also detect obstacles by using the LiDAR 23. The camera 24 is also mounted on the vehicle V. The camera 24 acquires real-time images. In this embodiment, in principle, the LiDAR 23 detects obstacles, but the camera 24 may also detect obstacles.

The control unit 25 is, for example, a CPU. By executing programs stored in the memory unit 22, the control unit 25 functions as a position information generation unit 251, an obstacle detection unit 252, a transmission unit 253, and a vehicle control unit 254.

The position information generation unit 251 generates vehicle position information used to identify the position and orientation of the vehicle V in the world coordinate system. The position information generation unit 251 generates vehicle position information using, for example, LiDAR SLAM technology.

Specifically, the position information generation unit 251 uses LiDAR SLAM technology to constantly generate vehicle position information indicating the position and orientation of vehicle V in the world coordinate system by comparing point cloud data indicating the shapes of objects around vehicle V detected by LiDAR 23 with map data in a world coordinate system created in advance.

The position information generation unit 251 may constantly generate point cloud data indicating the shapes of objects in the vicinity of the vehicle V detected by the LiDAR 23 as vehicle position information. In this case, the information processing device 1 generates information indicating the position and orientation of the vehicle V in the world coordinate system by comparing the point cloud data indicating the shapes of objects in the vicinity of the vehicle V detected by the LiDAR 23 with map data in a world coordinate system created in advance.

The obstacle detection unit 252 detects an obstacle and generates obstacle position information that is used to identify the position of the obstacle in the world coordinate system at the first time when the vehicle V detects the obstacle. The obstacle detection unit 252 generates the obstacle position information using, for example, LiDAR technology.

Specifically, when the vehicle V detects an obstacle, the obstacle detection unit 252 uses LiDAR SLAM technology to compare point cloud data indicating the shape of objects around the obstacle detected by the LiDAR 23 with map data in a world coordinate system created in advance, thereby generating obstacle position information indicating the position of the obstacle in the world coordinate system at the first time when the vehicle V detected the obstacle. The position of the obstacle in the world coordinate system is, for example, the center point of the obstacle or multiple points indicating the obstacle.

The obstacle detection unit 252 may generate, as obstacle position information, point cloud data indicating the shapes of objects in the vicinity of the obstacle detected by the LiDAR 23. In this case, the information processing device 1 compares the point cloud data indicating the shapes of objects in the vicinity of the obstacle detected by the LiDAR 23 with map data in a world coordinate system created in advance, thereby generating information indicating the position of the obstacle in the world coordinate system at the first time.

The obstacle detection unit 252 may detect obstacles and generate feature information. The obstacle detection unit 252 may generate feature information, for example, by performing image analysis of obstacles in real-time video.

Feature information includes, for example, image data of the obstacle, feature vectors of the image of the obstacle, information indicating the shape, size, outline, color, texture, weight, or condition of the obstacle (hereinafter, these may be collectively referred to as "shape, etc."), information indicating the type of obstacle (cardboard, animal, etc.), or information for identifying whether the obstacle is a moving or stationary object.

In addition, instead of the LiDAR SLAM described above, vehicle position information and obstacle position information may be generated using, for example, Visual SLAM using a camera or Depth SLAM using a ToF sensor.

The transmitter 253 transmits real-time video, vehicle position information, obstacle position information, and feature information to the information processing device 1 via the vehicle communication unit 21. The transmitter 253 transmits the vehicle position information, for example, at a second time that is later than the first time (for example, the time when the vehicle V makes an emergency stop).

The vehicle control unit 254 changes the driving state of the vehicle V by controlling the drive system for driving the vehicle V. The vehicle control unit 254 stops the vehicle V in response to the obstacle detection unit 252 detecting an obstacle, and starts the vehicle V driving in response to receiving a driving resumption signal from the information processing device 1.

The following provides a detailed explanation of the various functions provided by the information processing system S.

First Embodiment
As a first embodiment, we will explain a process in which the obstacle projection position on the real-time image is identified using the position of the obstacle at the first time when the vehicle V detects the obstacle, and the real-time image showing the identified obstacle projection position is displayed.

[Processing Overview]
The information processing device 1 periodically receives real-time video images obtained from a camera mounted on an autonomously driven vehicle V. As will be described below, the vehicle V uses LiDAR SLAM technology to acquire information indicating the position of an obstacle in a world coordinate system at the time the obstacle is detected, and information indicating the position and orientation of the vehicle V in the world coordinate system at the time the vehicle V makes an emergency stop.

When vehicle V detects an obstacle, it compares point cloud data indicating the shape of objects around the obstacle detected by the LiDAR mounted on vehicle V with pre-created world coordinate system map data to generate obstacle position information indicating the position of the obstacle in the world coordinate system at the first time the obstacle was detected. The world coordinate system map data includes point cloud data corresponding to objects in the three-dimensional space of the world coordinate system and is stored, for example, in memory unit 22. Vehicle V transmits the generated obstacle position information to information processing device 1.

Vehicle V also uses LiDAR SLAM technology to generate vehicle position information indicating the position and orientation of vehicle V in the world coordinate system by comparing point cloud data indicating the shapes of objects around vehicle V detected by LiDAR with pre-created map data in the world coordinate system. When vehicle V makes an emergency stop after detecting an obstacle, vehicle V transmits the generated vehicle position information to information processing device 1.

The information processing device 1 receives obstacle position information and vehicle position information from the vehicle V. The information processing device 1 converts the position of the obstacle in the world coordinate system at a first time when the vehicle V detects the obstacle into a projected position on the real-time video. The projected position of the obstacle on the real-time video changes from moment to moment depending on the position and orientation of the vehicle V in the world coordinate system, and therefore changes depending on the time when the vehicle V transmits the vehicle position information to the information processing device 1. Below, we will explain how to identify the projected position of the obstacle on the real-time video, assuming a situation in which the vehicle V transmits the generated vehicle position information to the information processing device 1 at a second time after the first time when the obstacle was detected (for example, the time when the vehicle V makes an emergency stop).

First, the information processing device 1 converts the obstacle position indicated by the obstacle position information in the world coordinate system at the first time into a first obstacle position, which is a position in the vehicle coordinate system (LiDAR coordinate system) at the second time, by calculating the difference between the coordinates indicated by the obstacle position at the first time and the coordinates indicated by the vehicle position information at the second time. Next, the information processing device 1 converts the first obstacle position in the vehicle coordinate system into a second obstacle position, which is a position in the camera coordinate system, using a conversion formula described below.

The world coordinate system, vehicle coordinate system, and camera coordinate system described so far are three-dimensional coordinate systems, but because real-time video is two-dimensional information, a process is required to convert the three-dimensional position information into two-dimensional position information. Therefore, the information processing device 1 converts the second obstacle position into a position in the two-dimensional image coordinate system, thereby identifying the obstacle projection position, which is the projection position of the obstacle on the real-time video. In this way, the information processing device 1 converts the position of the obstacle in the three-dimensional world coordinate system into a projection position on the two-dimensional real-time video.

The information processing device 1 displays real-time video showing the identified obstacle projection position on the display unit 13. This allows the observer M to easily identify obstacles in the real-time video, thereby reducing the observation burden on the observer M.

[Processing performed by the control unit]
As the processing executed by the control unit in the first embodiment, a description will be given of the processing executed by the video receiving unit 151, the obstacle position receiving unit 152, the vehicle position receiving unit 153, the position identifying unit 155, the determining unit 156, the display processing unit 159, the input accepting unit 160, and the signal transmitting unit 161. Fig. 5 is a diagram showing the configuration of the information processing device 1 in the first embodiment.

The video receiving unit 151 receives real-time video obtained from a camera mounted on the autonomously driven vehicle V. The video receiving unit 151 periodically receives real-time video, for example, via the device communication unit 11.

The video receiving unit 151 may receive an image from a camera mounted on the vehicle V at a first time when the vehicle V detects an obstacle. This allows the display processing unit 159 to display the image at the first time on the display unit 13 together with the real-time video, as will be described in detail below. Note that the concept of an image includes both moving images and still images.

The obstacle position receiving unit 152 receives obstacle position information used to identify the position of the obstacle in the world coordinate system at the first time when the vehicle V detects the obstacle. The obstacle position receiving unit 152 receives the obstacle position information after the first time, for example, via the device communication unit 11.

The obstacle position information may be information indicating the position of the obstacle in the world coordinate system at the first time when the vehicle V detects the obstacle, or it may be point cloud data indicating the shape of objects around the obstacle detected at the first time by the LiDAR mounted on the vehicle V. In the latter case, the obstacle position receiving unit 152 uses LiDAR SLAM technology to acquire information indicating the position of the obstacle in the world coordinate system at the first time.

The obstacle position receiving unit 152 may receive feature information indicating the features of the obstacle detected at the first time when the vehicle V detected the obstacle. The obstacle position receiving unit 152 receives the feature information along with the obstacle position information, for example, via the device communication unit 11. This allows the display processing unit 159 to display real-time video on the display unit 13 in a manner that makes it possible to distinguish whether the obstacle is a moving or stationary object, as will be described in detail below.

Feature information may be, for example, image data of the obstacle, a feature vector of the obstacle image, information indicating the shape of the obstacle, or information indicating the type of obstacle (cardboard, animal, etc.).

The obstacle position receiving unit 152 may receive error information indicating an error in the physical position of the obstacle at the first time when the vehicle V detects the obstacle. The error information is, for example, at least one of the accuracy of the self-position estimation performed by the vehicle V, the degree of shaking of the vehicle V, the distance from the vehicle V to the obstacle, and the reliability of the obstacle detection. The obstacle position receiving unit 152 receives the error information together with the obstacle position information, for example, via the device communication unit 11. This allows the display processing unit 159 to display a real-time image on the display unit 13 in which obstacles with large errors are highlighted, as will be described in detail below.

The obstacle position receiving unit 152 may receive multiple pieces of obstacle position information and multiple pieces of accuracy information indicating the detection accuracy of the obstacle at multiple times after the first time when the vehicle V detected the obstacle. The detection accuracy is, for example, an object recognition score, which is the result of performing object recognition processing on the obstacle. The detection accuracy is, for example, a reference value when determining which piece of obstacle position information at multiple times should be weighted or which piece of obstacle position information at multiple times should be used. The obstacle position receiving unit 152 receives, for example, the obstacle position information and accuracy information at the first time and the obstacle position information and accuracy information at times between the first and second times via the device communication unit 11. In this way, by the obstacle position receiving unit 152 receiving obstacle position information at multiple times, the display processing unit 159 can display real-time video with high accuracy of the obstacle projection position on the display unit 13, as will be described in detail below.

The detection accuracy indicated by the accuracy information will be higher, for example, the closer the obstacle detected by the LiDAR mounted on vehicle V is to the center of the detection area, or the larger the area it occupies within the detection area.

The vehicle position receiving unit 153 receives vehicle position information used to identify the position and orientation of the vehicle V in the world coordinate system at a second time, which is later than the first time. The vehicle position receiving unit 153 receives the vehicle position information, for example, via the device communication unit 11, at the second time, which is the time when the vehicle V makes an emergency stop.

The vehicle position information may be information indicating the position and orientation of vehicle V in the world coordinate system at the second time, or point cloud data indicating the shapes of objects around vehicle V detected at the second time by LiDAR mounted on vehicle V. In the latter case, the vehicle position receiving unit 153 uses LiDAR SLAM technology to acquire information indicating the position and orientation of vehicle V in the world coordinate system at the second time.

The position identification unit 155 identifies the obstacle projection position, which is the projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the camera's coordinate system. For example, the position identification unit 155 converts coordinates indicating the obstacle position in the world coordinate system at the first time into coordinates in the vehicle coordinate system (LiDAR coordinate system) and coordinates in the camera coordinate system, and finally converts them into coordinates in a two-dimensional image coordinate system (pixel coordinate system). The converted coordinates in the pixel coordinate system are the obstacle projection position on the real-time video.

The following describes in detail the process in which the position identification unit 155 converts the coordinates ( _Xw , _Yw , _Zw ) indicating the obstacle position in the world coordinate system at the first time into coordinates (u, v) in the two-dimensional pixel coordinate system.

The position identification unit 155 uses vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system at a second time later than the first time, corresponding to the obstacle position indicated by the obstacle position information in the world coordinate system at the first time when the vehicle V detected the obstacle.

For example, the position identification unit 155 uses vehicle position information to convert coordinates (Xw, Yw, _Zw ) indicating the obstacle position in the world coordinate system at a first time into coordinates ( _XL , _YL , _ZL ) in the LiDAR coordinate system at a second time using the following equation 1, thereby identifying coordinates _{(XL , YL} , _ZL ) as the _first obstacle position. In the following equation 1, Ww _,L indicates the position and orientation of vehicle V in the world coordinate system (vehicle position information) and is a transformation matrix for converting coordinates in the world coordinate system into coordinates in the LiDAR coordinate system. Note that in the following equation, s indicates a scale coefficient.

Furthermore, for example, W _w,c is calculated by the following equation 2 from the correspondence between coordinates (X _w , Y _w , Z _w ) in the world coordinate system and coordinates (u, v) in the pixel coordinate system. In equation 2 below, W _w,c indicates the position and orientation in the world coordinate system of the camera mounted on vehicle V. In addition, in the equation below, A indicates a known internal parameter (3×3 matrix) of the camera. In addition, in the equation below, if a prime (') is added to W, W is a 3×4 matrix, and if a prime (') is not added to W, W is a 4×4 matrix.

The position identification unit 155 acquires, for example, W _L,C , which is a transformation matrix for transforming coordinates in the LiDAR coordinate system into coordinates in the camera coordinate system, calculated in advance using Equation 3 shown below.

The position identification unit 155 identifies a second obstacle position, which is a position in the camera coordinate system at the second time corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. For example, the position identification unit 155 uses the acquired W _L,C to convert the coordinates (X _L ,Y _L ,Z _L ) in the LiDAR coordinate system at the second time, which are the first obstacle position, into coordinates (X _C ,Y _C ,Z _C ) in the camera coordinate system at the second time, thereby identifying the coordinates (X _C ,Y _C ,Z _C ) as the second obstacle position.

Finally, the position identification unit 155 identifies the obstacle projection position in the image coordinate system at the second time by converting the second obstacle position in the camera coordinate system into a position in the image coordinate system corresponding to the camera coordinate system.

The position identification unit 155 converts the coordinates ( _XC , _YC , _ZC ) in the camera coordinate system at the second time into coordinates (u, v) in the two-dimensional pixel coordinate system using, for example, Equation 4 shown below, and thereby identifies the coordinates (u, v) as the obstacle projection position.

In this way, the position identification unit 155 converts each of the coordinates ( _Xw , _Yw , _Zw ) indicating the obstacle position in the world coordinate system at the first time into coordinates (u, v) in the two-dimensional pixel coordinate system using the above-mentioned conversion.

The position identification unit 155 may determine whether the obstacle projection position is included in the capture range of the real-time video. For example, the position identification unit 155 may determine whether the obstacle projection position is included in the capture range of a camera mounted on the vehicle V. More specifically, the position identification unit 155 determines, for example, whether the coordinates of the second obstacle position in the camera coordinate system are included in the capture area of the camera coordinate system, or whether the converted coordinates after converting the second obstacle position into coordinates in the pixel coordinate system are included in the capture area of the pixel coordinate system. This allows the display processing unit 159 to display the real-time video on the display unit 13 in different ways depending on whether the obstacle projection position is included in the capture range of the real-time video or not, as will be described in detail below.

Incidentally, when the obstacle position receiving unit 152 receives multiple pieces of obstacle position information at multiple times, the position identifying unit 155 may identify the obstacle projection position based on the obstacle position information at a time closer to the second time, which is the emergency stop time of the vehicle V. This allows the position identifying unit 155 to identify the obstacle projection position with higher accuracy than when the position identifying unit 155 identifies the obstacle projection position based on obstacle position information at a time farther from the second time (e.g., the first time).

Furthermore, as described above, if the obstacle position receiving unit 152 receives multiple pieces of obstacle position information and multiple pieces of accuracy information at multiple times, the position identifying unit 155 may select obstacle position information to be used to identify the obstacle projection position from the multiple pieces of obstacle position information based on the multiple pieces of accuracy information at the multiple times. The position identifying unit 155 may, for example, select the obstacle position information at the time when the detection accuracy indicated by the accuracy information is the highest.

Alternatively, the position identification unit 155 may identify the obstacle projection position on the real-time video based on a weighted average of multiple physical positions corresponding to each of multiple pieces of obstacle position information so that obstacle position information with higher detection accuracy indicated by the accuracy information is reflected to a higher degree in the identification of the obstacle projection position on the real-time video. For example, when the multiple physical positions in the world coordinate system corresponding to each piece of obstacle position information are ( _X1 , _Y1 , _Z1 ), ( _X2 , _Y2 , _Z2 ), ... ( _XN , _YN , _ZN ) and the detection accuracy values are _m1 , _m2 , ... _mN , respectively, the position identification unit 155 may identify the coordinates in the world coordinate system of _the weighted average position _as (( _m1X1 + _m2X2 + _{... mNXN} )/( _{m1 + m2} +... _{mN )} , ( _{m1Y1 + m2Y2 +} ... _mNYN )/( _{m1 + m2} + _{... mN} ), (m1Z1+ _m2Z2 +... _mNZN )/( _m1 + _m2 +... _{mN )} ).

In this way, the position identification unit 155 identifies the obstacle projection position using obstacle position information at the time when the obstacle detection accuracy was highest, or by identifying the obstacle projection position using a physical position calculated by taking a weighted average according to the obstacle detection accuracy, the accuracy of the obstacle projection position displayed on the real-time video is improved. As a result, the reliability of the information processing system S according to this embodiment is improved.

The determination unit 156 determines whether the obstacle detected at the first time is a moving or stationary object based on the feature information received by the obstacle position receiving unit 152. A moving object is, for example, a person, an animal, a motorcycle, or a car. A stationary object is, for example, a stationary object such as cardboard, garbage, a road sign, or a guardrail.

The determination unit 156 determines whether an obstacle detected by the vehicle V is moving or stationary, for example, using a machine learning model trained by machine learning using, as training data, feature information and a correct answer label indicating whether an object having the feature indicated by the feature information is moving or stationary. Specifically, when the feature information received by the obstacle position receiving unit 152 is input to the machine learning model, the machine learning model outputs information indicating whether the obstacle detected by the vehicle V is moving or stationary.

The display processing unit 159 causes the display unit 13 to display real-time video in which the obstacle projection position is highlighted. For example, the display processing unit 159 causes the display unit 13 to display real-time video in which the obstacle projection position is marked. The display processing unit 159 may also cause the display unit 13 to display real-time video in which the obstacle projection position is surrounded by a box.

In this way, by having the display processing unit 159 display real-time video on the display unit 13 with the obstacle projection position highlighted, the monitor M can easily identify obstacles in the real-time video, thereby reducing the monitoring burden on the monitor M. As a result, when the vehicle V detects an obstacle and makes an emergency stop, it can quickly resume driving as soon as it is confirmed that the surrounding conditions of the vehicle V are safe.

The display processing unit 159 may display on the display unit 13 obstacle projection positions corresponding to relatively large obstacles in a more emphasized manner than obstacle projection positions corresponding to other obstacles. This allows the observer M to prioritize checking relatively large obstacles in the real-time video.

The display processing unit 159 may display, on the display unit 13, an image at a first time when the vehicle V detected an obstacle and real-time video showing the obstacle projection position. For example, the display processing unit 159 may display the image at the first time and the real-time video side by side on the same screen, or may display them separately on multiple screens.

In this way, by having the display processing unit 159 display on the display unit 13 the image at the first time when the vehicle V detected the obstacle, along with the real-time video, the monitor M can also check the image at the time when the vehicle V detected the obstacle. As a result, even in situations where a more careful decision needs to be made as to whether the vehicle V can resume traveling (for example, when there is heavy traffic on the road the vehicle V is traveling on or the road the vehicle V is traveling on is a highway), the monitor M can make an appropriate decision, thereby reducing the probability of a traffic accident occurring when the vehicle V resumes traveling.

The display processing unit 159 may display the real-time video on the display unit 13 in different modes depending on whether the position identification unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video or whether the position identification unit 155 determines that the obstacle projection position is not included in the shooting range of the real-time video. For example, if the position identification unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video, the display processing unit 159 displays the real-time video in which the obstacle projection position is surrounded by a box on the display unit 13. On the other hand, for example, if the position identification unit 155 determines that the obstacle projection position is not included in the shooting range of the real-time video, the display processing unit 159 displays the real-time video including a string indicating this (for example, the string "No obstacle detected").

Furthermore, when the position identification unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video, the display processing unit 159 may cause the display unit 13 to display the real-time video in which the obstacle is highlighted. For example, the display processing unit 159 may cause the display unit 13 to display real-time video in which the color of the box surrounding the obstacle projection position is different from the other colors in the real-time video.

In this way, the display processing unit 159 displays the real-time video on the display unit 13 in different ways depending on whether the obstacle projection position is included in the shooting range of the real-time video or not. This allows the monitor M to eliminate or shorten the time it takes to check real-time video in which no obstacles are projected. As a result, the monitoring burden on the monitor M can be reduced. Furthermore, the monitor M can concentrate on checking real-time video in which obstacles are projected, improving the accuracy of monitoring.

The display processing unit 159 may display real-time video on the display unit 13 in a manner that makes it possible to distinguish whether an obstacle is a moving object or a stationary object. For example, the display processing unit 159 may display an obstacle projection position corresponding to a moving object on the display unit 13 in a more emphasized manner than an obstacle projection position corresponding to a stationary object. Furthermore, the display processing unit 159 may also display, for example, the type of moving object (person, animal, motorcycle, automobile, etc.) at the obstacle projection position corresponding to the moving object.

In this way, the display processing unit 159 displays real-time video on the display unit 13 in a manner that makes it possible to distinguish whether the obstacle is a moving or stationary object. This allows the monitor M to more carefully consider giving an instruction to resume operation of the vehicle V when a moving object is present around the vehicle V. As a result, the probability of a traffic accident occurring when the vehicle V resumes operation can be reduced.

The display processing unit 159 may cause the display unit 13 to display real-time video in which the obstacle is shown in a different manner depending on the error indicated by the error information indicating the error in the physical position of the obstacle at the first time when the vehicle V detected the obstacle. For example, the display processing unit 159 may cause the display unit 13 to display an obstacle projection position corresponding to an obstacle with a relatively large error in a more emphasized manner than obstacle projection positions corresponding to other obstacles. For example, the display processing unit 159 may display a box enclosing an obstacle projection position corresponding to an obstacle with a relatively large error larger than boxes enclosing obstacle projection positions corresponding to other obstacles.

In this way, the display processing unit 159 causes the display unit 13 to display real-time video in which the obstacle is shown in different ways depending on the error indicated by the error information. This prevents the actual position of the obstacle from being included in the obstacle projection position, even when, for example, the vehicle V is shaking significantly or the distance from the vehicle V to the obstacle is large. As a result, the reliability of the information processing system S according to this embodiment is improved.

The input receiving unit 160 receives an input of a restart driving instruction, which is an instruction to resume driving of the vehicle V. When the observer M monitoring the real-time video confirms that the obstacle has been removed and determines that it is okay to resume driving of the vehicle V, the observer M inputs a restart driving instruction to the input receiving unit 160 via the input unit 14. In this way, the input receiving unit 160 receives an input of a restart driving instruction from the observer M via the input unit 14.

When the input receiving unit 160 receives an input of a driving restart instruction, the signal transmitting unit 161 transmits a driving restart signal to the vehicle V, which is a signal for causing the vehicle V to resume driving. The signal transmitting unit 161 transmits the driving restart signal to the vehicle V, for example, via the device communication unit 11. Upon receiving the driving restart signal, the vehicle V resumes driving.

[Processing flow in information processing device 1]
A description will be given of the flow of processing in the first embodiment of the information processing device 1. Fig. 6 is a flowchart showing the flow of processing in the first embodiment of the information processing device 1.

The video receiving unit 151 periodically receives real-time video via the device communication unit 11 (S1). The obstacle position receiving unit 152 receives obstacle position information via the device communication unit 11, which is used to identify the position of the obstacle in the world coordinate system at a first time when the vehicle V detects the obstacle (S2). The vehicle position receiving unit 153 receives vehicle position information, which is used to identify the position and orientation of the vehicle V in the world coordinate system at a second time after the first time (S3).

The position identification unit 155 identifies the obstacle projection position, which is the projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the camera's coordinate system (S4).

The display processing unit 159 displays real-time video showing the obstacle projection position on the display unit 13 (S5). This allows the observer M to check the real-time video and determine whether or not it is appropriate to resume driving the vehicle V.

If the monitor M determines that it is OK to allow the vehicle V to resume traveling, the input receiving unit 160 receives input of a traveling restart instruction from the monitor M, which is an instruction to resume traveling of the vehicle V (S6). When the input receiving unit 160 receives input of the traveling restart instruction, the signal transmitting unit 161 transmits a traveling restart signal, which is a signal to resume traveling of the vehicle V, to the vehicle V (S7). Upon receiving the traveling restart signal, the vehicle V resumes traveling.

Second Embodiment
As a second embodiment, we will explain a process in which real-time video is analyzed using the characteristics of an obstacle detected by a vehicle V to detect the obstacle, and real-time video showing the position of the detected obstacle is displayed.

[Processing Overview]
The processing in the second embodiment is the same as that in the first embodiment in that the position of an obstacle is displayed on the real-time video, but the method for achieving this is different. In the first embodiment, the obstacle projection position on the real-time video is identified using the position of the obstacle at the first time when the vehicle V detected the obstacle, whereas in the second embodiment, the obstacle is detected from among multiple objects present in the real-time video by performing image analysis on the real-time video.

An overview of the processing performed by the information processing device 1 will now be explained. The information processing device 1 periodically receives real-time video images obtained from a camera mounted on an autonomously driven vehicle V. When the vehicle V detects an obstacle, it makes an emergency stop and generates feature information indicating the characteristics of the detected obstacle. The feature information may be, for example, image data of the obstacle or the shape of the obstacle. The vehicle V transmits the generated feature information to the information processing device 1.

The information processing device 1 receives characteristic information from the vehicle V. Then, the information processing device 1 detects, from among multiple objects present in the real-time video, an object that has the characteristic indicated by the characteristic information. The information processing device 1 displays the real-time video, showing the position of the detected object, on the display unit 13. This allows the monitor M to easily identify obstacles in the real-time video, thereby reducing the monitoring burden on the monitor M.

[Processing performed by the control unit]
As the processing executed by the control unit in the second embodiment, a description will be given of the processing executed by the video receiving unit 151, the feature receiving unit 154, the position identifying unit 155, the detecting unit 157, the display processing unit 159, the input accepting unit 160, and the signal transmitting unit 161. Fig. 7 is a diagram showing the configuration of the information processing device 1 in the second embodiment.

The video receiving unit 151 receives real-time video obtained from a camera mounted on the autonomously driven vehicle V. The video receiving unit 151 periodically receives real-time video, for example, via the device communication unit 11.

The feature receiving unit 154 receives feature information indicating the features of an obstacle detected by the vehicle V. The feature receiving unit 154 receives feature information generated by the obstacle detection unit 252 of the vehicle V, for example, via the device communication unit 11. That is, the feature receiving unit 154 receives feature information generated on the vehicle V side, such as image data of the obstacle, a feature vector of the obstacle image, information indicating the shape of the obstacle, information indicating the type of obstacle, or information for identifying whether the obstacle is a moving or stationary object.

In this way, the feature receiving unit 154 may receive feature information generated on the vehicle V side, but it may also receive image data of the obstacle from the vehicle V and, by analyzing the received image data, identify the feature vector of the obstacle image, information indicating the shape of the obstacle, information indicating the type of obstacle, or information for identifying whether the obstacle is a moving or stationary object.

In this case, the feature receiving unit 154 may identify the feature information of the image data received by the feature receiving unit 154 using a machine learning model that has been trained using the image data and feature information of the object indicated by the image data as training data. Specifically, when the image data received by the feature receiving unit 154 is input to the machine learning model, the machine learning model outputs the feature information of the object indicated by the input image data.

The feature receiving unit 154 may receive obstacle position information indicating the physical location of the obstacle at the time the vehicle V detected the obstacle. This allows the position identifying unit 155 to identify the obstacle projection position, as will be described in detail below. As a result, the display processing unit 159 can display real-time video on the display unit 13, which also indicates the obstacle projection position.

The feature receiving unit 154 may receive error information indicating the error in the physical position of the obstacle at the time the vehicle V detected the obstacle. This allows the display processing unit 159 to display real-time video on the display unit 13 in a manner that does not display the obstacle projection position on the display unit 13 if the error indicated by the error information is large, as will be described in detail below.

The position identification unit 155 identifies the obstacle projection position, which is the position of the obstacle on the real-time video, based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting the obstacle in the vehicle V and the camera mounted on the vehicle V. The specific details of this processing are similar to the processing performed by the position identification unit 155 in the first embodiment, so a detailed description will be omitted.

The detection unit 157 detects, from among multiple objects present in the real-time video, objects that have the characteristics indicated by the characteristic information. For example, the detection unit 157 targets the real-time video received by the video receiving unit 151 and detects, from among multiple objects present in this real-time video, objects that have the characteristic information received or identified by the characteristic receiving unit 154.

The detection unit 157 may identify the features of multiple objects present in the real-time video and detect, from among the multiple objects, objects whose features have a similarity to the features indicated by the feature information that is equal to or greater than a threshold. For example, the detection unit 157 may identify the features of each object in the real-time video received by the video receiving unit 151. The features referred to here include image data of the object, a feature vector of the object's image, information indicating the shape of the object, information indicating the type of object, or information for identifying whether the object is moving or stationary.

As an example of detection performed by the detection unit 157, when the feature information is image data, the detection unit 157 detects, from the image data corresponding to each object in the real-time video, objects whose similarity to the image data indicated by the feature information is equal to or greater than a threshold value.

The detection unit 157 may detect an object using, for example, a single piece of feature information (e.g., image data of an obstacle), or may detect an object using a combination of feature information (e.g., image data and shape of an obstacle). When the detection unit 157 uses a combination of feature information, the detection unit 157 detects, for example, an object from among multiple objects present in real-time video that has a feature whose similarity to the feature indicated by the first feature information is equal to or greater than a threshold, and that has a feature whose similarity to the feature indicated by the second feature information is equal to or greater than a threshold. In this way, by the detection unit 157 using a combination of feature information, the detection unit 157 can perform object detection with higher accuracy.

The display processing unit 159 causes the display unit 13 to display real-time video showing the position of the detected object detected by the detection unit 157. For example, the display processing unit 159 causes the display unit 13 to display real-time video in which the position of the detected object is marked. The display processing unit 159 may also cause the display unit 13 to display real-time video in which the position of the detected object is surrounded by a box.

In this way, by having the display processing unit 159 display real-time video showing the position of the detected object on the display unit 13, the monitor M can easily identify obstacles in the real-time video, thereby reducing the monitoring burden on the monitor M. As a result, when the vehicle V detects an obstacle and makes an emergency stop, it can quickly resume driving as soon as it is confirmed that the surrounding conditions of the vehicle V are safe.

The display processing unit 159 may cause the display unit 13 to display real-time video that further indicates the obstacle projection position. That is, the display processing unit 159 may cause the display unit 13 to display real-time video that indicates the obstacle projection position identified by the position identification unit 155 and the position of the detected object detected by the detection unit 157. In this way, by displaying the position of an obstacle identified using multiple different means in the real-time video, it is possible to reduce the probability of an obstacle not being displayed in the real-time video even though it actually exists.

However, there are cases where the observer M wants to check the real-time video after distinguishing whether the position of an obstacle displayed on the real-time video is the obstacle projection position or the position of a detected object. Therefore, the display processing unit 159 may display the real-time video on the display unit 13 in a manner that allows the obstacle projection position and the position of the detected object to be distinguished. For example, the display processing unit 159 may display the real-time video on the display unit 13 in which a box surrounding the obstacle projection position and a box surrounding the position of the detected object are displayed in different colors or with different types of lines.

In this way, the display processing unit 159 displays real-time video on the display unit 13 in a manner that allows the obstacle projection position and the position of the detected object to be distinguished, allowing the observer M to check the real-time video while distinguishing between the obstacle projection position and the position of the detected object. As a result, the efficiency of the observer M's monitoring is improved.

The method of displaying the obstacle projection position and the method of displaying the position of the detected object each have their advantages and disadvantages. The method of displaying the obstacle projection position has the advantage that the obstacle projection position is identified based on the position of the obstacle detected by the vehicle V, so there is a low probability of an object present at the obstacle projection position in the real-time video being mistakenly displayed as not actually being an obstacle. Another advantage is that there is no need to send obstacle characteristic information to the information processing device 1, which reduces communication traffic. Another advantage is that there is no need for the information processing device 1 to detect objects, which reduces the processing time required to display the real-time video and reduces the specifications required for the information processing device 1.

On the other hand, the method of displaying the projected position of an obstacle has the disadvantage that if there is a large error in the physical position of the obstacle (for example, if the distance from the vehicle V to the obstacle is large), the position of the obstacle detected by the vehicle V will likely be inaccurate, and as a result, the position of the obstacle displayed on the real-time video image will likely be inaccurate. Another disadvantage is that if the obstacle moves, the displayed position of the obstacle will likely be inaccurate.

The method of displaying the position of a detected object has the advantage that, even if the obstacle is moving and moves between the time vehicle V detects the obstacle and the time vehicle V makes an emergency stop, the displayed position of the obstacle is highly likely to be accurate, because image analysis is performed on real-time video.

On the other hand, the method of displaying the position of a detected object has the disadvantage that if image analysis of real-time video fails, the position of the obstacle may not be displayed even if it is present. Also, if the real-time video shows multiple obstacles, including objects other than the obstacle that caused vehicle V to stop, when vehicle V is stopped, object detection has the disadvantage that the object that caused vehicle V to stop cannot be identified.

As described above, the display processing unit 159 may determine whether to display the obstacle projection position or the detected object position on the display unit 13, taking into account the current situation and conditions. Four patterns are explained below.

In the first pattern, if the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a stationary object, the detection unit 157 may not detect the object, and the display processing unit 159 may display the obstacle projection position on the display unit 13. In other words, if the obstacle detected by the vehicle V is a stationary object, the display processing unit 159 may display the obstacle projection position according to the first embodiment on the display unit 13. On the other hand, the position of the detected object according to the second embodiment is not displayed on the real-time video because the detection unit 157 does not detect the object in the first place.

As a second pattern, if the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a moving object, the display processing unit 159 may display the position of the detected object on the display unit 13 without displaying the obstacle projection position on the display unit 13.

As a third pattern, if the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a moving object, and the distance between the obstacle projection position and the position of the detected object is equal to or less than a predetermined threshold, the display processing unit 159 may display the obstacle projection position on the display unit 13, but may not display the position of the detected object on the display unit 13. In other words, if the obstacle detected by the vehicle V is a moving object, but the obstacle does not move between the time the vehicle V detects the obstacle and the time the vehicle V makes an emergency stop, the display processing unit 159 may treat the moving obstacle as equivalent to a stationary object, and may display the obstacle projection position according to the first embodiment on the display unit 13, but may not display the position of the detected object according to the second embodiment on the display unit 13.

As a fourth pattern, if the error in the physical position of the obstacle indicated by the error information is equal to or greater than a threshold, the display processing unit 159 may cause the display unit 13 to display the position of the detected object, but may not cause the display unit 13 to display the obstacle projection position. In other words, for example, if the distance from the vehicle V to the obstacle is large, there is a high probability that the position of the obstacle detected by the vehicle V is inaccurate, and therefore there is a high probability that the obstacle projection position cannot be accurately identified. Therefore, the display processing unit 159 may cause the display unit 13 to display the position of the detected object according to the second embodiment, but may not cause the display unit 13 to display the obstacle projection position according to the first embodiment.

In this way, the display processing unit 159 takes into account the situation and conditions at the time and displays the appropriate obstacle projection position or the detected object position on the display unit 13, thereby improving the efficiency of monitoring by the monitor M.

The processing performed by the input reception unit 160 and the signal transmission unit 161 is similar to that described in the first embodiment, so further description will be omitted.

[Processing flow in information processing device 1]
A description will be given of the flow of processing in the second embodiment of the information processing device 1. Fig. 8 is a flowchart showing the flow of processing in the second embodiment of the information processing device 1.

The video receiving unit 151 periodically receives real-time video via the device communication unit 11 (S1). The feature receiving unit 154 receives feature information generated by the obstacle detection unit 252 of the vehicle V via the device communication unit 11 (S2). The detection unit 157 detects an object that has the feature indicated by the feature information from among multiple objects present in the real-time video (S3).

The position identification unit 155 identifies the obstacle projection position, which is the position of the obstacle on the real-time image, based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting the obstacle in the vehicle V and the camera mounted on the vehicle V (S4).

The display processing unit 159 displays the real-time video on the display unit 13 in a manner that allows the obstacle projection position and the position of the detected object detected by the detection unit 157 in S3 to be distinguished (S5). This allows the observer M to check the real-time video and determine whether or not it is appropriate to resume driving the vehicle V.

If the monitor M determines that it is OK to allow the vehicle V to resume traveling, the input receiving unit 160 receives input of a traveling restart instruction from the monitor M, which is an instruction to resume traveling of the vehicle V (S6). When the input receiving unit 160 receives input of the traveling restart instruction, the signal transmitting unit 161 transmits a traveling restart signal, which is a signal to resume traveling of the vehicle V, to the vehicle V (S7). Upon receiving the traveling restart signal, the vehicle V resumes traveling.

Third Embodiment
As a third embodiment, a process for controlling the display of multiple real-time images obtained from multiple cameras mounted on a vehicle V will be described.

[Processing Overview]
If a vehicle V is equipped with multiple cameras, multiple real-time images will be available for each vehicle V, equal in number to the number of cameras. However, the real-time image that actually captures an obstacle may only be a portion of the multiple real-time images. Furthermore, even if an obstacle is captured in multiple real-time images, only a portion of the real-time images may capture the obstacle clearly enough for the observer M to see.

In such cases, the monitor M must first select the real-time video that they should focus on checking from among multiple real-time videos, which increases the workload of the monitor M. Therefore, the information processing device 1 controls the display of multiple real-time videos based on the obstacle projection degree, which indicates the degree of projection of the obstacle, allowing the monitor M to instantly know which real-time video they should check.

An overview of the processing performed by information processing device 1 will now be described. Information processing device 1 periodically receives multiple real-time images obtained from multiple cameras mounted on an autonomously driven vehicle V. As described below, vehicle V uses LiDAR SLAM technology to acquire information indicating the position of detected obstacles in a world coordinate system, as well as information indicating the position and orientation of vehicle V in a world coordinate system.

When vehicle V detects an obstacle, it compares point cloud data indicating the shape of objects around the obstacle detected by the LiDAR installed in vehicle V with pre-created map data in a world coordinate system to generate obstacle position information indicating the position of the obstacle in the world coordinate system. Vehicle V transmits the generated obstacle position information to information processing device 1.

Vehicle V also uses LiDAR SLAM technology to generate vehicle position information indicating the position and orientation of vehicle V in the world coordinate system by comparing point cloud data indicating the shapes of objects around vehicle V detected by LiDAR with pre-created map data in the world coordinate system. If vehicle V makes an emergency stop after detecting an obstacle, vehicle V transmits the generated vehicle position information to information processing device 1 at a second time that is after the first time the obstacle was detected.

The information processing device 1 receives obstacle position information and vehicle position information from the vehicle V. The information processing device 1 converts the position of the obstacle in the world coordinate system into a projection position on the real-time video using a method similar to that described in the processing overview of the first embodiment. Then, if the converted projection position has three or more coordinates, the information processing device 1 determines, as the obstacle projection degree indicating the degree of projection of the obstacle, the area of a figure including the coordinates of three or more points; if the converted projection position has two coordinates, the length of the line segment connecting the coordinates of the two points. The information processing device 1 determines the obstacle projection degree for each of the multiple real-time videos.

The information processing device 1 then controls the display of multiple real-time images based on the obstacle projection degree. For example, the information processing device 1 displays on the display unit 13 the real-time image with the highest obstacle projection degree among the multiple real-time images. Alternatively, when the information processing device 1 displays multiple real-time images on the display unit 13, it displays on the display unit 13 the real-time image with a relatively high obstacle projection degree in a more emphasized manner than the other real-time images.

In this way, the information processing device 1 controls the display of multiple real-time images based on the obstacle projection degree, allowing the observer M to instantly know which real-time image to check. This reduces the monitoring burden on the observer M.

[Processing performed by the control unit]
As the processing executed by the control unit in the third embodiment, a description will be given of the processing executed by the video receiving unit 151, the obstacle position receiving unit 152, the vehicle position receiving unit 153, the projection degree determining unit 158, the display processing unit 159, the input accepting unit 160, and the signal transmitting unit 161. Fig. 9 is a diagram showing the configuration of the information processing device 1 in the third embodiment.

The video receiving unit 151 receives multiple real-time images obtained from multiple cameras mounted on the autonomously driven vehicle V. The video receiving unit 151 periodically receives real-time images, for example, via the device communication unit 11. For example, for each vehicle V, the video receiving unit 151 receives real-time images equal to the number of cameras mounted on the vehicle V.

The video receiving unit 151 may receive multiple real-time images obtained from multiple cameras mounted on another vehicle V separate from the vehicle V. This allows the projection degree determination unit 158 to determine the obstacle projection degree in the multiple real-time images obtained from the multiple cameras mounted on the other vehicle V, as will be described in detail below.

The obstacle position receiving unit 152 receives obstacle position information used to identify the position in the world coordinate system of an obstacle detected by the vehicle V. The obstacle position receiving unit 152 receives, for example, via the device communication unit 11, obstacle position information used to identify the position in the world coordinate system of an obstacle at a first time when the vehicle V detects the obstacle.

The obstacle position information may be information indicating the position of the obstacle in a world coordinate system, or point cloud data indicating the shape of objects around the obstacle detected by LiDAR mounted on the vehicle V. In the latter case, the obstacle position receiving unit 152 uses LiDAR SLAM technology to acquire point cloud data of coordinates indicating the position of the obstacle in the world coordinate system.

The vehicle position receiving unit 153 receives vehicle position information used to determine the position and orientation of the vehicle V in the world coordinate system. The vehicle position receiving unit 153 receives vehicle position information used to determine the position and orientation of the vehicle V in the world coordinate system via the device communication unit 11, for example, at a second time that is later than the first time.

The vehicle position information may be information indicating the position and orientation of vehicle V in the world coordinate system, or point cloud data indicating the shapes of objects around vehicle V detected by LiDAR mounted on vehicle V. In the latter case, the vehicle position receiving unit 153 uses LiDAR SLAM technology to acquire information indicating the position and orientation of vehicle V in the world coordinate system.

The projection degree determination unit 158 identifies the projection area of the obstacle in each of the multiple real-time images corresponding to the obstacle position information using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the camera's coordinate system. The projection degree determination unit 158, for example, uses the vehicle position information to convert the obstacle position indicated by the obstacle position information in the world coordinate system into a position in the vehicle coordinate system. For example, the projection degree determination unit 158 converts the converted position in the vehicle coordinate system into a position in the camera coordinate system. For example, the projection degree determination unit 158 converts the converted position in the camera coordinate system into a position in the image coordinate system. For example, the projection degree determination unit 158 identifies the area indicating the converted position in the image coordinate system as the projection area of the obstacle on the real-time image.

The projection degree determination unit 158 then determines the obstacle projection degree, which indicates the degree of projection of the obstacle, based on the shape of the identified projection area. The projection degree determination unit 158 determines the obstacle projection degree, for example, based on the size of the projection area. For example, the projection degree determination unit 158 may use the value indicating the size of the projection area as the obstacle projection degree, or may calculate the obstacle projection degree by substituting the value indicating the size of the projection area into a predetermined calculation formula.

For example, when the coordinates of a point indicating the position of an obstacle in the world coordinate system are converted into coordinates in the image coordinate system, the size of the projection area is defined by the area or length of a line segment based on the converted coordinates in the image coordinate system. Below, we will explain in detail the process by which the projection degree determination unit 158 determines this area or length of a line segment.

First, we will explain in detail the process by which the projection degree determination unit 158 determines the area based on the transformed coordinates in the image coordinate system. The obstacle position information is, for example, point cloud data that indicates the position of the obstacle in the world coordinate system.

The projection degree determination unit 158 uses vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system corresponding to the position indicated by the obstacle position information in the world coordinate system. The projection degree determination unit 158 identifies a second obstacle position, which is a position in the camera coordinate system corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. The projection degree determination unit 158 converts the second obstacle position into a position in the image coordinate system corresponding to the camera coordinate system. The specific content of the processing described in this paragraph is similar to the content of the processing performed by the position identification unit 155 in the first embodiment, so further description will be omitted.

The projection degree determination unit 158 determines the area based on the positions in the transformed image coordinate system as the size of the projection region. For example, the projection degree determination unit 158 determines the area of a plane figure that includes at least 80% of the points, and preferably all of the points, among the coordinates that indicate the positions in the transformed image coordinate system as the size of the projection region. The type of this plane figure is not particularly limited, and may be, for example, a polygon, a circle, an ellipse, a sector, etc.

Alternatively, the projection degree determination unit 158 may determine the size of the projection region to be the area of a plane figure formed when connecting coordinates that exist on the outside among the coordinates that indicate positions in the transformed image coordinate system. For example, if there are N coordinates on the outside among the coordinates that indicate positions in the transformed image coordinate system, the projection degree determination unit 158 may determine the area of an N-angle system formed when connecting these N points as the size of the projection region.

Next, we will explain in detail the process by which the projection degree determination unit 158 determines the length of a line segment based on transformed coordinates in the image coordinate system. Obstacle position information is, for example, the coordinates of two points at both ends of point cloud data on a plane at a specific height in the world coordinate system, among point cloud data indicating the position of an obstacle in the world coordinate system. Specifically, if point cloud data exists in the world coordinate system that is included in a plane at a specific height from the ground, when the point cloud data is viewed from a viewpoint at the same height as this plane (directly to the side of this plane), several points in the point cloud will appear to gather together to form a straight line. The points located at both ends of this line are the two points at both ends mentioned above.

The projection degree determination unit 158 uses the vehicle position information to identify the coordinates of two points in the vehicle coordinate system that correspond to the coordinates of two points indicated by the obstacle position information in the world coordinate system, based on the relative positional relationship between the world coordinate system and the vehicle coordinate system. The projection degree determination unit 158 identifies the coordinates of two points in the camera coordinate system that correspond to the identified coordinates of the two points in the vehicle coordinate system, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. The projection degree determination unit 158 converts the identified coordinates of the two points in the camera coordinate system into coordinates of two points in the image coordinate system that correspond to the camera coordinate system. The specific content of the processing described in this paragraph is similar to the content of the processing performed by the position identification unit 155 in the first embodiment, so further description will be omitted.

The projection degree determination unit 158 determines the length of the line segment connecting the coordinates of the two converted points as the size of the projection area. In this way, when the projection degree determination unit 158 determines the size of the projection area based on the coordinates of two points in the world coordinate system, the vehicle V only needs to transmit data indicating the coordinates of two points in the world coordinate system to the information processing device 1, rather than point cloud data indicating a large number of points in the world coordinate system. In other words, when the projection degree determination unit 158 determines the size of the projection area based on the coordinates of two points in the world coordinate system, the amount of data transmitted by the vehicle V to the information processing device 1 is smaller than when the projection degree determination unit 158 determines the size of the projection area based on the coordinates of two points in the world coordinate system, compared to when the size of the projection area is determined based on a large number of point cloud data in the world coordinate system. As a result, the convenience of using the information processing system S according to this embodiment is improved.

The display processing unit 159 controls the display of multiple real-time images based on the obstacle projection degree. For example, the display processing unit 159 displays specific real-time images based on the obstacle projection degree, or displays specific real-time images with more emphasis than other real-time images. Specific examples of display control performed by the display processing unit 159 are described below.

The display processing unit 159, for example, causes the display unit 13 to display one or more real-time images in which the obstacle projection degree is relatively high among the multiple real-time images. The display processing unit 159, for example, compares the obstacle projection degree among the real-time images from the number of cameras mounted on a certain vehicle V, and causes the display unit 13 to display the real-time image with the highest obstacle projection degree.

The display processing unit 159 may display multiple real-time images received by the image receiving unit 151 on the display unit 13, and may highlight a real-time image with a relatively high obstacle projection degree among the multiple real-time images over the other real-time images. The display processing unit 159 may, for example, display as many real-time images as there are cameras mounted on a certain vehicle V on the display unit 13. In this case, the display processing unit 159 may, for example, compare the real-time images for the number of cameras and highlight the real-time image with the highest obstacle projection degree by displaying the real-time image with the highest obstacle projection degree on the display unit 13 with an outer frame line surrounding the real-time image in a different color from the outer frame lines surrounding the other real-time images, or with the outer frame line surrounding the real-time image being thicker than the outer frame lines surrounding the other real-time images.

The display processing unit 159 may cause the display unit 13 to display real-time images in which the obstacle projection degree determined by the projection degree determination unit 158 is equal to or greater than a threshold, and may highlight a real-time image in which the obstacle projection degree is relatively high among the multiple real-time images more than the other real-time images. For example, the display processing unit 159 may cause the display unit 13 to display real-time images in which the obstacle projection degree is equal to or greater than a threshold, from among the real-time images from the number of cameras mounted on a certain vehicle V. In this case, the display processing unit 159 may, for example, compare the real-time images displayed on the display unit 13, and highlight the real-time image with the highest obstacle projection degree in a manner similar to that described in the previous paragraph.

In this way, the display processing unit 159 can display specific real-time images based on the obstacle projection degree, or can highlight specific real-time images more than other real-time images, allowing the observer M to instantly know which real-time images to check. As a result, the monitoring burden on the observer M can be reduced.

However, if the obstacle projection degree in all of the multiple real-time images obtained from the cameras mounted on a certain vehicle V is low, the observer M will not be able to properly check the real-time images. In such a case, the projection degree determination unit 158 may determine the obstacle projection degree in the multiple real-time images obtained from the multiple cameras mounted on another vehicle V other than the vehicle V. This allows the observer M to check the surrounding situation after an emergency stop of the vehicle V by checking the multiple real-time images obtained from the multiple cameras mounted on the other vehicle V. Below, the process of determining the obstacle projection degree in the multiple real-time images obtained from the multiple cameras mounted on the other vehicle V will be described.

When the obstacle projection degree in each of the multiple real-time images obtained from the multiple cameras mounted on the vehicle V falls below a threshold, the projection degree determination unit 158 uses the obstacle position information, the separate vehicle position information, and the relative positional relationship between the separate vehicle coordinate system and the separate camera coordinate system to identify the projection area of the obstacle in each of the multiple real-time images obtained from the multiple cameras mounted on the separate vehicle V that corresponds to the obstacle position information. The separate vehicle position information is information used to identify the position and orientation of the separate vehicle V in the world coordinate system. The separate vehicle coordinate system is a coordinate system used to define the position and orientation of the separate vehicle V indicated by the separate vehicle position information. The separate camera coordinate system is the coordinate system of the camera of the separate vehicle V. The projection degree determination unit 158 determines the obstacle projection degree, which indicates the projection degree of the obstacle, based on the shape of the identified projection area. The specific content of the processing described in this paragraph is similar to the content of the processing performed by the projection degree determination unit 158 described above, and therefore will not be described again.

The processing described in the previous paragraph may be performed using multiple fixed cameras installed on or near the road, instead of a camera mounted on another vehicle V. This allows the projection degree determination unit 158 to identify the obstacle projection degrees in the multiple real-time images obtained from multiple fixed cameras installed on or near the road, even in a situation where the obstacle projection degrees are low in all of the multiple real-time images obtained from the cameras mounted on vehicle V that has made an emergency stop, and there are no other vehicles around vehicle V. As a result, observer M can properly check the real-time images.

Alternatively, as another method for dealing with cases where the obstacle projection degree is low in all of the multiple real-time images obtained from the cameras mounted on a certain vehicle V, the obstacle projection degree may be improved by controlling the orientation of the cameras mounted on the vehicle V. Below, we will explain the process for improving the obstacle projection degree in the multiple real-time images obtained from the multiple cameras mounted on the vehicle V.

When the obstacle projection degree in each of the multiple real-time images obtained from the camera mounted on the vehicle V falls below a threshold, the signal transmission unit 161 transmits to the vehicle V a camera control signal, which is a signal for controlling the orientation of the camera so that the obstacle projection degree in each of the multiple real-time images is equal to or greater than the threshold.

The signal transmission unit 161, for example, determines whether all of the obstacle projection indices in multiple real-time images obtained from cameras mounted on a certain vehicle V are below a threshold. If the signal transmission unit 161 determines that all of the obstacle projection indices are below the threshold, for example, it calculates the camera orientation at which the obstacle projection indices in each of the multiple real-time images are equal to or greater than the threshold. Then, the signal transmission unit 161 transmits to the vehicle V a camera control signal, which is a signal for moving the camera mounted on the certain vehicle V left and right (pan) or up and down (tilt) so that the orientation of the camera matches the calculated camera orientation.

In this way, even if the obstacle projection degree of the real-time video obtained from vehicle V is low, the projection degree determination unit 158 can identify the obstacle projection degree in multiple real-time videos obtained from other vehicles V, and the signal transmission unit 161 can improve the obstacle projection degree by transmitting to vehicle V a signal for controlling the orientation of the camera mounted on vehicle V. As a result, even if the obstacle projection degree of the real-time video obtained from vehicle V is low, the monitor M can check the surrounding situation after an emergency stop of vehicle V by taking either of the above measures. As a result, the convenience of using the information processing system S according to this embodiment is improved.

The processing performed by the input reception unit 160 and the processing for transmitting the travel resumption signal performed by the signal transmission unit 161 are the same as those described in the first embodiment, and therefore will not be described again.

[Processing flow in information processing device 1]
A description will be given of the flow of processing in the third embodiment of the information processing device 1. Fig. 10 is a flowchart showing the flow of processing in the third embodiment of the information processing device 1.

The video receiving unit 151 periodically receives multiple real-time images obtained from multiple cameras via the device communication unit 11 (S1). The obstacle position receiving unit 152 receives obstacle position information used to identify the position of an obstacle in the world coordinate system via the device communication unit 11 (S2). The vehicle position receiving unit 153 receives vehicle position information used to identify the position and orientation of the vehicle V in the world coordinate system (S3).

The projection degree determination unit 158 uses the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the camera's coordinate system, to identify the projection area of the obstacle in each of the multiple real-time images corresponding to the obstacle position information, and determines the obstacle projection degree, which indicates the degree of projection of the obstacle, based on the shape of the identified projection area (S4).

The display processing unit 159 controls the display of multiple real-time images based on the obstacle projection degree (S5). This allows the observer M to check the real-time images and determine whether or not it is appropriate to resume driving the vehicle V.

If the monitor M determines that it is OK to allow the vehicle V to resume traveling, the input receiving unit 160 receives input of a traveling restart instruction from the monitor M, which is an instruction to resume traveling of the vehicle V (S6). When the input receiving unit 160 receives input of the traveling restart instruction, the signal transmitting unit 161 transmits a traveling restart signal, which is a signal to resume traveling of the vehicle V, to the vehicle V (S7). Upon receiving the traveling restart signal, the vehicle V resumes traveling.

Furthermore, this invention will contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and alterations are possible within the spirit of the invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of new embodiments resulting from the combination will also have the effects of the original embodiments.

DESCRIPTION OF SYMBOLS 1 Information processing device 11 Device communication unit 12 Memory unit 13 Display unit 14 Input unit 15 Control unit 151 Video receiving unit 152 Obstacle position receiving unit 153 Vehicle position receiving unit 154 Feature receiving unit 155 Position identification unit 156 Determination unit 157 Detection unit 158 Projection degree determination unit 159 Display processing unit 160 Input reception unit 161 Signal transmission unit 2 Vehicle terminal 21 Vehicle communication unit 22 Memory unit 23 LiDAR
24 Camera 25 Control unit 251 Position information generation unit 252 Obstacle detection unit 253 Transmission unit 254 Vehicle control unit S Information processing system V Vehicle M Observer

Claims (14)

a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle;
an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle;
a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time;
a position identification unit that identifies an obstacle projection position, which is a projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and a relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system that is a coordinate system of the camera;
a display processing unit that displays the real-time image showing the obstacle projection position on a display unit;
An information processing device having the above. the video receiving unit receives an image at the first time from the camera;
the display processing unit causes the display unit to display the image at the first time and the real-time video in which the obstacle projection position is indicated.
The information processing device according to claim 1 . the position identification unit determines whether the obstacle projection position is included in a capturing range of the real-time image;
the display processing unit displays the real-time image on the display unit in a manner different from a case where the position specifying unit determines that the obstacle projection position is included in the shooting range and a case where the position specifying unit determines that the obstacle projection position is not included in the shooting range.
The information processing device according to claim 1 . the display processing unit, when the position identification unit determines that the obstacle projection position is included in the shooting range of the real-time image, causes the display unit to display the real-time image in which the obstacle is highlighted.
The information processing device according to claim 3 . the obstacle position receiving unit receives characteristic information indicating a characteristic of the obstacle detected at the first time;
the information processing device further includes a determination unit that determines whether the obstacle detected at the first time is a moving object or a stationary object based on the feature information;
the display processing unit displays the real-time video on the display unit in a manner that enables the user to distinguish whether the obstacle is a moving object or a stationary object.
The information processing device according to claim 1 . the obstacle position receiving unit receives error information indicating an error in the physical position of the obstacle at the first time;
the display processing unit causes the display unit to display the real-time video in which the obstacle is shown in a different manner depending on the error indicated by the error information.
The information processing device according to claim 1 . the error information is at least one of accuracy of the self-position estimation performed by the vehicle, a degree of shaking of the vehicle, a distance from the vehicle to the obstacle, and a reliability of detection of the obstacle;
The information processing device according to claim 6 . the position identification unit uses the vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system at the second time corresponding to an obstacle position indicated by the obstacle position information in the world coordinate system at the first time; identifies a second obstacle position, which is a position in the camera coordinate system at the second time corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system; and identifies the obstacle projection position by converting the second obstacle position into a position in an image coordinate system corresponding to the camera coordinate system.
The information processing device according to claim 1 . the obstacle position receiving unit receives a plurality of pieces of obstacle position information and a plurality of pieces of accuracy information indicating detection accuracy of the obstacle at a plurality of times after the first time,
the position specifying unit selects the obstacle position information to be used for specifying the obstacle projection position from the plurality of pieces of obstacle position information based on the plurality of pieces of accuracy information at the plurality of times.
The information processing device according to claim 1 . the position identification unit selects the obstacle position information at the time when the detection accuracy indicated by the accuracy information is the highest.
The information processing device according to claim 9 . the obstacle position receiving unit receives a plurality of pieces of obstacle position information and a plurality of pieces of accuracy information indicating detection accuracy of the obstacle at a plurality of times after the first time,
the position identification unit identifies the obstacle projection position on the real-time image based on a weighted average of a plurality of the physical positions corresponding to the plurality of pieces of obstacle position information so that the obstacle position information having a higher detection accuracy indicated by the accuracy information is reflected at a high rate in the identification of the obstacle projection position on the real-time image;
The information processing device according to claim 1 . The information processing device includes:
an input receiving unit that receives an input of a travel restart instruction that is an instruction to restart the travel of the vehicle;
a signal transmitting unit that transmits, when the input receiving unit receives the input of the travel restart instruction, a travel restart signal to the vehicle, the travel restart signal being a signal for causing the vehicle to resume traveling;
The information processing device according to claim 1 , further comprising: The computer executes
a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle;
an obstacle receiving step of receiving obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle;
a vehicle receiving step of receiving vehicle position information used to identify a position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time;
a position specifying step of specifying an obstacle projection position, which is a projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and a relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system, which is a coordinate system of the camera;
a display processing step of displaying the real-time image showing the obstacle projection position on a display unit;
An information processing method comprising: An information processing device and a vehicle terminal mounted on a vehicle and capable of communicating with the information processing device,
The information processing device includes:
a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle;
an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle;
a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time;
a position identification unit that identifies an obstacle projection position, which is a projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and a relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system that is a coordinate system of the camera;
a display processing unit that displays the real-time image showing the obstacle projection position on a display unit;
and
The vehicle terminal
a position information generating unit that generates the vehicle position information;
an obstacle detection unit that detects the obstacle and generates the obstacle position information;
a transmitter that transmits the real-time video, the obstacle position information, and the vehicle position information;
a vehicle control unit that stops the vehicle in response to the obstacle detection unit detecting the obstacle, and starts the vehicle traveling in response to receiving a traveling restart signal from the information processing device;
having
Information processing system.