KR102936865B1 - Method and Apparatus for Identifying Damage Information Based on Environmental Conditions at Disaster Sites - Google Patents

Abstract

The present invention relates to a method for identifying damage information at a disaster site. This method receives image data captured at a disaster site, analyzes the environmental conditions contained in the image, and classifies multiple facility classes within the image using a segmentation model trained based on the analyzed conditions. Next, for each segmented facility class, a damage assessment model appropriate for the environmental conditions is applied to identify damage information, and the method includes outputting facility information corresponding to the identified damage information. This method can improve the accuracy of damage identification under various disaster site conditions.

Description

Method and Apparatus for Identifying Damage Information Based on Environmental Conditions at Disaster Sites

One embodiment of the present invention relates to disaster site analysis technology. More specifically, the present invention relates to a method and device for identifying damage information based on environmental conditions at a disaster site. This method segments facilities based on environmental conditions contained in image data acquired at the disaster site, and applies a damage assessment model appropriate to the environmental conditions to each facility. This method enhances damage identification performance under various conditions.

Existing disaster response technologies have tended to identify structural damage based on a single illumination level or fixed shooting conditions. This has led to significant variations in damage identification accuracy depending on environmental factors such as lighting, distance, angle, and humidity. In particular, disaster sites often present damaged or deformed structures, making it difficult to identify damage based solely on standard image conditions. While some technologies attempt damage analysis based on object detection or image segmentation, a structural approach that actively improves damage identification accuracy across diverse environmental conditions has been lacking. Furthermore, even for the same damage type, appropriate assessment criteria should vary depending on the structure's shape, location, and surrounding environment, yet adaptive assessment models have been inadequate.

Korean Patent No. 10-2555009 (July 10, 2023) [Korea Institute of Construction Quality]

The present invention has been devised to solve the above-mentioned problems, and its purpose is to provide a method for improving damage identification accuracy by determining environmental conditions based on image data collected from a disaster site and combining and applying a segmentation model and a damage assessment model according to the environment. In particular, the present invention analyzes various factors such as the color distribution of pixels in the image, the relationship with surrounding structures, and the location and shape of facilities, and actively adjusts the direction of illumination or the location of sensors, or preprocesses the image itself, thereby enabling damage assessment optimized for the environment. Furthermore, in the case of an image containing multiple types of damage, the present invention provides a disaster response system that simultaneously considers resource efficiency and analysis accuracy by selecting whether to process the image or perform light illumination based on the relative damage ratio and risk level.

The problem to be solved by this specification is not limited to what has been described above, and can be expanded to various matters that can be derived from the embodiments of the invention described below.

A method for identifying damage information according to environmental conditions of a disaster site according to one embodiment of the present specification is performed by a computing device, and includes the steps of: receiving image data captured at a disaster site; determining environmental conditions included in the image data; segmenting the image data into a plurality of facility classes using a segmentation model to which the environmental conditions are applied; identifying a damage class by applying a damage judgment model selected according to the environmental conditions to an image area corresponding to each of the segmented facility classes; and outputting facility class information corresponding to the identified damage class.

In one embodiment, the environmental conditions may include one or more of humidity, temperature, illuminance, shooting distance, and shooting angle.

In one embodiment, the segmentation model may be a model learned based on a learning data set that includes image data acquired at different illumination levels, angles, or shooting positions for the same damage type.

In one embodiment, the step of acquiring image data may include determining environmental conditions that provide high accuracy in identifying the damage based on the type of damage class to be identified and the shape and arrangement information of the structure where the damage is located, and acquiring the image data by adjusting the position of a light source or sensor to correspond to the determined environmental conditions.

In one embodiment, the environmental conditions may include being set by determining an optimal illuminance value and light irradiation position corresponding to color information based on color information calculated by analyzing a color combination of all pixels included in the image data.

In one embodiment, the step of identifying the damage class may include the step of applying different damage judgment models according to each facility class.

In one embodiment, the step of determining whether the damage class is included may determine a damage judgment model to be applied to the facility class based on the positional relationship between the facility class and other adjacent facility classes and damage information within the other adjacent facility classes.

In one embodiment, before applying the damage judgment model, the method may further include calculating the types of damage classes included in the image data and the pixel ratio or number of objects occupied by each damage class, and, based on the types and ratios of the calculated damage classes, operating the light irradiation unit to irradiate light of a specific intensity to re-acquire image data, or performing image processing on the current image data to correct the illuminance.

In one embodiment, the selection may include operating the light irradiation unit to obtain high-illuminance image data when the ratio of the peeling class is higher than the ratio of the crack class, omitting the operation of the light irradiation unit and performing damage determination while maintaining the image data when the ratio of the crack class is above a certain standard, or performing illuminance correction to increase the accuracy of crack identification using an image processing method.

A device for identifying damage information according to environmental conditions at a disaster site according to one embodiment of the present specification can perform the above-described method at least partially or entirely.

A recording medium according to one embodiment of the present specification may be combined with hardware and include a command or program for executing the above-described method.

The present invention enables damage identification that reflects various environmental conditions at disaster sites, thereby overcoming the limitations of existing systems, which often suffer from reduced identification accuracy depending on lighting or shooting angle. Furthermore, even for identical damage, the assessment model can be applied differently based on the location and shape of the structure and surrounding damage information, enabling more precise damage analysis.

The optimal lighting direction can be calculated based on the pixel color combination within the image, the image quality can be optimized by automatically setting the light irradiation position, and whether or not to irradiate light can be flexibly controlled by considering the relative risk and ratio of multiple damage types.

This configuration can be linked to real-time analysis systems at disaster sites to provide useful basic data for predicting structural collapse, prioritizing maintenance of risk areas, and making rapid recovery decisions.

It should be understood that the effects of this specification are not limited to the matters described above, but can be expanded to various contents that can be derived from the detailed description of the embodiments of the invention below.

FIG. 1 is a diagram illustrating an example of an operating environment of a system according to one embodiment of the present specification.
FIG. 2 is a block diagram for explaining the internal configuration of a computing device (200) in one embodiment of the present specification.
FIG. 3 is a drawing for explaining the hardware configuration of an investigation robot connected to a computing device according to an embodiment of the present invention.
FIG. 4 is a block diagram more specifically illustrating a service processing unit (250) of a computing device (200) for driving a robot (300) according to an embodiment of the present invention to output facility analysis information corresponding to indoor/outdoor space 3D analysis.
FIG. 5 is a flowchart of a damage information identification procedure using facility segmentation within a disaster site according to one embodiment of the present specification.
Figure 6 shows the operation structure of an artificial intelligence model (210) according to one embodiment of the present specification.
FIG. 7 illustrates an example of facility class division on image data (700) according to one embodiment of the present specification.
FIG. 8 is a comparative diagram for explaining the improved effect of a damage judgment processing method according to one embodiment of the present specification.
FIG. 9 is a drawing for explaining the structural positional relationship and damage judgment criteria of a beam and a column according to one embodiment of the present specification.
Figure 10 is a flowchart of a method for identifying damage information according to environmental conditions of a disaster site according to one embodiment of the present specification.
Figure 11 shows the results of crack detection by angle according to one embodiment of the present specification.
FIG. 12 is a diagram illustrating damage detection results according to illumination change conditions for a crack class according to one embodiment of the present specification.
Figure 13 shows the results of analyzing the performance of damage information detection according to changes in image illuminance according to one embodiment of the present specification.
FIG. 14 is a diagram for explaining an image data preprocessing process according to one embodiment of the present specification.

In describing the embodiments of this specification, if a detailed description of a known configuration or function is judged to obscure the gist of the embodiments of this specification, a detailed description thereof will be omitted. In addition, parts of the drawings that are not related to the description of the embodiments of this specification have been omitted, and similar parts have been designated with similar drawing reference numerals.

In the embodiments of this specification, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

In the embodiments of this specification, the terms first, second, etc. are used only for the purpose of distinguishing one component from another component, and do not limit the order or importance between components unless specifically stated otherwise. Therefore, within the scope of the embodiments of this specification, a first component in an embodiment may be referred to as a second component in another embodiment, and similarly, a second component in an embodiment may be referred to as a first component in another embodiment.

In the embodiments of this specification, distinct components are used to clearly illustrate their respective characteristics and do not necessarily imply separation. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of the embodiments of this specification.

In this specification, the term "network" may encompass both wired and wireless networks. In this case, the term "network" may refer to a communications network that enables data exchange between devices, systems, and devices, and is not limited to a specific network.

Embodiments described herein may be entirely hardware, partially hardware and partially software, or entirely software. As used herein, "unit," "device," or "system" refers to a computer-related entity such as hardware, a combination of hardware and software, or software. For example, a unit, module, device, or system as used herein may be, but is not limited to, a running process, a processor, an object, an executable, a thread of execution, a program, and/or a computer. For example, both an application running on a computer and the computer itself may correspond to a unit, module, device, or system as used herein.

Additionally, in this specification, a device may be a mobile device such as a smartphone, tablet PC, wearable device, or HMD (Head Mounted Display), as well as a fixed device such as a PC or home appliance with display functions. Furthermore, as an example, a device may be an in-vehicle cluster or an IoT (Internet of Things) device. In other words, in this specification, a device may refer to any device capable of operating an application, and is not limited to a specific type. For convenience of explanation, the device on which an application operates is referred to as a device below.

In this specification, the network communication method is not limited, and connections between each component may not be made using the same network method. The network may include not only communication methods utilizing communication networks (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.), but also short-range wireless communication between devices. For example, the network may include all communication methods that enable objects to network with each other, and is not limited to wired communication, wireless communication, 3G, 4G, 5G, or other methods. For example, wired and/or networks include Local Area Network (LAN), Metropolitan Area Network (MAN), Global System for Mobile Network (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Zigbee, Wi-Fi, VoIP (Voice over Internet Protocol), LTE Advanced, IEEE802.16m, Wireless MAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple Access (BDMA), Wi-MAX(World Interoperability for Microwave) A communication network may refer to a communication network using one or more communication methods selected from the group consisting of, but not limited to, communication utilizing access and ultrasonic communication.

The components described in various embodiments are not necessarily essential components, and some may be optional. Therefore, embodiments comprising a subset of the components described in the embodiments are also included within the scope of the embodiments of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also included within the scope of the embodiments of the present disclosure.

Hereinafter, embodiments of the present specification will be described in detail with reference to the drawings.

FIG. 1 is a diagram illustrating an example of the operating environment of a system according to one embodiment of the present specification. Referring to FIG. 1, a user device (110) and one or more servers (120, 130, 140) are connected via a network (1). FIG. 1 is merely an example for explaining the invention, and the number of user devices or servers is not limited to that shown in FIG. 1.

The user device (110) may be a fixed terminal or a mobile terminal implemented as a computer system. The user device (110) may be, for example, a smart phone, a mobile phone, a navigation device, a computer, a laptop, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (Internet of Things) device, a VR (virtual reality) device, an AR (augmented reality) device, etc. For example, in the embodiments, the user device (110) may mean one of various physical computer systems that can communicate with other servers (120 to 140) through a network (1) using a wireless or wired communication method.

Each server may be implemented as a computer device or multiple computer devices that communicate with a user device (110) via a network (1) to provide commands, codes, files, content, services, etc. For example, the server may be a system that provides each service to a user device (110) connected via a network (1). As a more specific example, the server may provide a service (e.g., provision of information, etc.) intended by an application to the user device (110) through an application as a computer program installed and run on the user device (110). As another example, the server may distribute a file for installing and running the above-described application to the user device (110) and receive user input information to provide a corresponding service.

FIG. 2 is a block diagram illustrating the internal configuration of a computing device (200) according to one embodiment of the present specification. This computing device (200) may be applied to the user device (110) or server (120-140) described above with reference to FIG. 1, and each device and server may have the same or similar internal configuration by adding or excluding some components.

Referring to FIG. 2, a computing device (200) may include a memory (210), a processor (220), a communication module (230), and a transceiver (240). The memory (210) is a non-transitory computer-readable recording medium and may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. Here, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the above-described device or server as a separate permanent storage device distinct from the memory (210). In addition, the memory (210) may store an operating system and at least one program code (for example, a browser installed and operated on a user device (110), or a code for an application installed on a user device (110) to provide a specific service). These software components may be loaded from a computer-readable recording medium separate from the memory (210). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, a memory card, etc.

In another embodiment, the software components may be loaded into the memory (210) via a communication module (230) rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory (210) based on a computer program (e.g., the application described above) that is installed by files provided by developers or a file distribution system (e.g., the server described above) that distributes the installation files of the application via a network (1).

The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor (220) by the memory (210) or the communication module (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

The communication module (230) can provide a function for the user device (110) and the server (120 - 140) to communicate with each other via the network (1), and can provide a function for each of the device (110) and/or the server (120 - 140) to communicate with other electronic devices.

The transceiver (240) may be a means for interfacing with an external input/output device (not shown). For example, external input devices may include devices such as a keyboard, mouse, microphone, camera, etc., and external output devices may include devices such as a display, speaker, haptic feedback device, etc.

As another example, the transmitter/receiver (240) may be a means for interfacing with a device that integrates input and output functions, such as a touch screen.

In addition, in other embodiments, the computing device (200) may include more components than the components of FIG. 2 depending on the nature of the device to which it is applied. For example, when the computing device (200) is applied to a user device (110), it may be implemented to include at least some of the above-described input/output devices, or may further include other components such as a transceiver, a Global Navigation Satellite System module, a camera, various sensors, a database, etc. As a more specific example, when the user device is a smartphone, it may be implemented to further include various components that are generally included in a smartphone, such as an acceleration sensor or a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and a vibrator for vibration.

Hereinafter, a computing device operating based on FIGS. 1 and 2 will be described. As an example, a user may communicate with a robot (300) based on a computing device (200). In addition, the robot (300) described below may be a device that is connected to one computing device (200) and communicates with the outside world. As a specific example, the robot (300) may communicate with another device or server via a network and may be configured with the components included in the computing device (200) of FIG. 2. In addition, the robot (300) may include a sensor for identifying a surrounding image or video, which will be described later.

Overview of Disaster Site Investigation Robots

FIG. 3 is a diagram illustrating the hardware configuration of an investigation robot connected to a computing device according to an embodiment of the present invention. Referring to FIG. 3, the robot (300) is connected to a multi-sensor (320) including at least one sensor by a header (330) capable of fine adjustment and light source formation, and the multi-sensor (320) is connected to a robot arm (310) by a separate connecting means to perform indoor structure analysis. Here, the robot arm (310) is designed to be able to rotate 360 degrees and may be provided in a form capable of folding and extending.

In addition, the robot (300) may be equipped with a control unit (340) at the bottom that controls the robot's hardware, power, and communication, and a moving unit (350) that controls the means of movement according to movement, thereby enabling movement for structural damage analysis in indoor environments such as disaster areas. For example, the moving unit (350) may be configured to perform the function of moving between indoor and outdoor spaces using wheels or other means, and is not limited to a specific form.

In this way, the robot (300) is configured to be able to move in indoor and outdoor spaces, and may be equipped with a communication function to collect various information and transmit it to another device or server. In addition, the robot (300) may acquire sensing information by combining with a multi-sensor (320). Here, the multi-sensor (320) may include a lidar sensor, an IMU sensor, a vision camera, a depth camera, a gyro sensor, and various other sensors for sensing surrounding information, and is not limited to a specific sensor.

A LiDAR sensor may be a sensor that measures the position coordinates of a reflector by emitting a laser pulse and measuring the time it takes for it to be reflected and returned. In addition, an IMU sensor may be a motion sensor that measures the direction, acceleration, and position of a device by measuring speed, direction, and magnetism through a speedometer, gyroscope, and magnetometer. In addition, a vision camera may sense image information of a surrounding image, and a depth camera may be a sensor that measures or senses a 3D image (i.e., a depth image) of surrounding objects based on a stereo method or a ToF (time of flight) method. Other sensors may include sensors for temperature, air pressure, and other surrounding sensing, and are not limited to specific embodiments.

For example, the robot (300) may include a lidar sensor (or module) for indoor positioning and mapping. Here, the lidar sensor may be manufactured in a compact form and coupled to the robot (300). In particular, the lidar sensor operates based on SLAM, generating an indoor map in real time, thereby enabling the recognition of indoor structures.

In addition, for example, a global navigation satellite system (GNSS) can be utilized for indoor and outdoor positioning at a disaster site. However, in cases where communication and radio wave connection from the outside are not smooth due to building shielding and damage at an indoor disaster site, there is a need to independently recognize the location in the indoor and outdoor space and perform mapping. For this purpose, the robot (300) may include a camera and a lidar as multiple sensors (320). In this case, as an example, the lidar sensor may be sensitive to vibration caused by external factors because it measures time based on the ToF method. Considering the above, the multiple sensors (320) may include an IMU sensor, through which the values for the movement and vibration of the robot (300) can be sensed, and position movement can be performed considering the vibration and analysis accuracy, thereby performing stable crack detection and damage information analysis.

In this way, the multi-sensor (320) may include sensors for the robot (300) to smoothly perform sensing in an indoor structure collapsed by an earthquake, and the sensing information is not limited to a specific embodiment. For convenience of explanation, the following description is based on an operation in which the multi-sensor (320) is coupled to the robot (300) and the robot (300) performs sensing through the multi-sensor (320) to analyze the indoor structure. That is, the robot (300) described below may be a robot (300) equipped with the multi-sensor (320), and may not be limited to a specific embodiment.

The multi-sensor (320) according to an embodiment of the present invention may include a camera sensor, an IMU sensor, and a lidar sensor positioned at the top. Here, each sensor module of the multi-sensor (320) is configured to be combined in hardware, and is combined around a single main axis that vertically penetrates each module, so that their three-dimensional space sensing coordinate axes can be synchronized. According to the synchronization of the coordinate axes of the multi-sensor (320), positioning and rotation information due to the movement of the robot (300) and the entry into a slope section, etc., can be processed in real time and reflected when acquiring sensing information of the multi-sensor (320). For example, the output of each sensor module acquired according to the synchronization of the coordinate axes can be mapped with position information centered on the synchronized main coordinate axes information, and image information, depth sensing information, rotation information, angle information, acceleration information, and vibration information acquired in synchronization from each sensor module can be mapped with the position information, and used for tracking the position of the robot (300) and damage detection processing of indoor and outdoor space facilities.

In addition, in order to facilitate the processing of sensing information of these multiple sensors (320), it is preferable to utilize data processing based on the known SLAM technology. SLAM is a technology that estimates a location and creates a three-dimensional map of an unknown environment using various sensors. As computer processing speeds improve and sensor technologies such as cameras and lidars develop, it is a technology that is being applied in various fields, and in particular, it is a technology used for indoor positioning and mapping in cases where there is no existing map, such as a disaster site, and constructs spatial information through lidar and camera-based scanning, data structuring, and three-dimensional indoor map visualization.

Accordingly, the robot (300) can construct a 3D indoor map by converting information sensed from multiple sensors (320) into data, and can detect estimated facilities within the constructed indoor map and the degree of damage to each facility based on crack analysis and share the same. For example, the robot (300) can be deployed to an indoor structure collapsed by an earthquake, construct spatial information based on the robot's location indoors, and analyze and output the degree of damage to each facility.

At this time, for example, an indoor structure collapsed by an earthquake may be in a rough terrain environment, and the rough terrain conditions may be classified as flat ground, speed bumps, gravel, etc., as detected by sensing average acceleration and maximum acceleration, for example. The robot (300) and multi-sensor (320) configured in this manner may be controlled by the computing device (200) according to an embodiment of the present invention.

The computing device (200) may be included in the aforementioned investigation robot (300) or exist independently thereof to control the investigation robot and receive and process information obtained from the investigation robot. For example, the robot (300) may utilize SLAM based on the aforementioned multi-sensor (320) to perform indoor positioning and mapping and damage analysis, and the control of the robot (300) may be performed by the connected computing device (200). For example, the robot (300) and the computing device (200) may exchange information through a communication connection between them. At this time, the robot (300) may perform sensing in indoor and outdoor spaces based on the control operation of the computing device (200), and the computing device (200) may control the position of the multi-sensor (320) or the movement of the moving part (350) in real time to increase the accuracy of crack detection.

FIG. 4 is a block diagram more specifically illustrating a service processing unit (250) of a computing device (200) for driving a robot (300) according to an embodiment of the present invention to output facility analysis information corresponding to indoor/outdoor space 3D analysis. Referring to FIG. 4, the computing device (200) according to an embodiment of the present invention may further include a service processing unit (250). The service processing unit (250) may be configured with processor modules that perform various service processes for crack analysis, damage detection, object identification, and robot (300) control data processing according to an embodiment of the present invention among the logic and input/output operations processed by the processor (220). The computing device (200) can provide the generated service processing result to a user device (110) or one or more servers (120, 130, 140) by configuring a service processing unit (250) within the processor (220) or performing service processing using data processed from a device equipped with a service processing unit (250) as an external processor.

According to this processing, the user device (110) or one or more servers (120, 130, 140) may receive analysis information configured by the operation of the service processing unit (250) according to an embodiment of the present invention and display it through a display device or use it to transmit and receive data through an external network.

More specifically, referring first to FIG. 4, the service processing unit (250) includes a robot driving unit (251), a sensor module driving unit (253), a crack analysis unit (255), and an analysis information output unit (257). The robot driving unit (251) generates a control signal for performing hardware driving control to control the position movement and sensing operation of the robot (300) illustrated in FIGS. 3 and 4 described above, transmits the control signal to the robot (300), and performs driving control by receiving the response. Here, the robot driving unit (251) can perform driving control of each of the multi-sensor (320), header (330), robot arm (310), control unit (340), and moving unit (350) of the robot (300) described above, and accordingly, movement of the robot (300), internal position of the multi-sensor (320), and sensing control can be performed.

In addition, the sensor module driving unit (253) acquires the sensor signals of each sensor equipped in the multi-sensor (320), configures them into sensing information synchronized for each location based on SLAM, and performs data processing to transmit the information to the crack analysis unit (255).

Here, the sensor signal may include a sensor signal of a lidar sensor, a sensor signal of an IMU sensor, and a vision sensing signal of a camera sensor. In addition, the sensor module driving unit (253) mutually maps and converts signals obtained from each sensor module of the multi-sensor (320) to configure SLAM data, and the configured SLAM data may be transmitted to the crack analysis unit (255).

Here, the SLAM data may further include information converted and processed to form 3D spatial data, in addition to sensor signals from basic lidar sensors, IMU sensors, and camera sensors.

For example, SLAM data may further include indoor location coordinate information calculated from an IMU sensor, a raw video mapping image obtained from a camera sensor, pose information of a camera and lidar calculated from the IMU sensor, and feature point cloud information obtained from the raw video image.

In addition, if the sensing accuracy of each sensor information is below a threshold, the sensor module driving unit (253) may request the robot driving unit (251) to move the position of the robot (300) or to control the precise position of the multiple sensors (320) in order to improve the accuracy.

And, the crack analysis unit (255) obtains the information converted and processed by the sensor module driving unit (253) described above, and processes the construction of a 3D indoor map and facility damage detection using the obtained sensing information. Here, the crack analysis unit (255) can identify facility segments of indoor and outdoor spaces based on the sensing information for damage detection processing, and can obtain visual crack identification information analyzed from vision image information of the sensing information corresponding to the facility segment and crack depth change information determined by depth sensing information corresponding to the visual crack identification information, and can estimate and calculate the degree of damage to the facility segment based on the presence or absence of a crack and the degree of the crack obtained based on the visual crack identification information and the crack depth change information. Accordingly, the analysis information output unit (257) can configure facility information analysis result data of the facility segment based on the degree of damage, combine it with the constructed 3D indoor map information, and output it to one or more devices. These one or more devices may be, for example, user devices (110) or may include one or more servers (120, 130, 140).

Identifying damage information using facility segmentation within a disaster site

FIG. 5 is a flowchart illustrating a damage information identification procedure using facility segmentation within a disaster site according to one embodiment of the present disclosure. Referring to FIG. 5, a computing device (200) may receive image data captured at a disaster site (S110). The image data may be collected, for example, by a camera sensor and/or a lidar sensor included in the multi-sensor (320) of the robot (300) illustrated in FIG. 3. The image data may include two-dimensional image data, and may also include three-dimensional point cloud data acquired by the lidar sensor. In other words, the image data may be in an integrated form that includes not only camera-based RGB images but also lidar-based three-dimensional spatial information including depth information or position coordinates. Such composite image data may be transmitted to the service processing unit (250) within the computing device (200) and utilized in subsequent analysis steps.

In one embodiment, the computing device (200) may apply a segmentation model based on input image data to segment multiple facility classes on a pixel-by-pixel basis within the image (S120). As described above, the segmentation model may identify different facility classes, such as beams, columns, floors, ceilings, windows, and walls, based on, for example, shape, color, texture, and distance information of the facilities within the image, and generate mask information for the corresponding classes on a pixel or point-by-pixel basis. To this end, the service processing unit (250) may assume the presence of multiple facilities for each analysis frame and perform segmentation processing.

Thereafter, in one embodiment, the computing device (200) can apply a damage judgment model to an image area or point cloud area corresponding to each segmented facility class to determine whether the damage class is included (S130). At this time, the damage judgment model can be implemented in the crack analysis unit (255) described above, and can determine whether damage such as cracks, peeling, or detachment is present in the corresponding area based on the mask range of each facility class. In addition to visual characteristics, the judgment criteria can also be comprehensively utilized such as SLAM-based 3D position information, pose information, and point cloud information described in FIG. 4.

Thereafter, in one embodiment, the computing device (200) can output facility class information corresponding to the damage class as an analysis result (S140). The analysis information output unit (257) can configure information including the type, location, and extent of damage of the facility where damage has been detected based on the damage determination result described above, and the information can be transmitted to the user device (110) or external server (120, 130, 140) of FIG. 4 and visually output or stored.

Fig. 6 illustrates the operation structure of an artificial intelligence model (210) according to one embodiment of the present specification. Referring to Fig. 6, the artificial intelligence model (210) can receive image data, point cloud data, sensor information, etc. acquired at a disaster site as input. The input may include, for example, RGB images collected from multiple sensors (320) of a robot (300), LiDAR-based 3D point cloud information, IMU-based position and attitude information, etc. Each of these inputs can go through a preprocessing process and be transmitted to an internal module of the artificial intelligence model (210).

In one example, the artificial intelligence model (210) may include a segmentation model (201) and a damage assessment model (202). The segmentation model (201) may segment multiple objects corresponding to facilities within an image using input images and point cloud data. At this time, the segmentation may be performed on a pixel or point basis, and may be classified into facility classes such as beams, columns, walls, ceilings, floors, and windows by considering the shape, material, color, location, etc. of the building structure. In addition, in one example, the damage assessment model (202) may be individually applied to each segmented facility class. That is, as illustrated in FIG. 6, after the facility classes are identified, different damage assessment models may be selectively applied depending on the type of each class. For example, a damage assessment model that focuses on vertical cracks may be applied to the column class, and a damage assessment model that focuses on whether there is peeling or detachment may be applied to the wall class. Through this configuration, the artificial intelligence model (210) can select and apply an appropriate damage assessment model based on the structural function, location, material, damage risk, etc. of each facility, rather than using a single damage assessment model. Each damage assessment model is described in more detail in the embodiments described below, and may include, for example, CNN series models of various sizes, transformer-based models, or specialized deep learning-based classifiers.

FIG. 7 illustrates an example of segmenting facility classes on image data (700) according to one embodiment of the present specification. Referring to FIG. 7, the image data (700) may be an original image captured at a disaster site, and each pixel or point within the image may be segmented into individual facility classes by a segmentation model. As illustrated in FIG. 7, 701 represents a beam, 702 represents a wall, 703 represents a window, 704 represents a column, and 705 represents a floor. Each facility class may be segmented based on, for example, the structural shape, material, location, color, contrast, etc. of the building, and the segmentation may be performed through a segmentation network based on YOLOv8.

Referring further to Fig. 7, Fig. 7 illustrates the class segmentation results in a two-dimensional planar image, but according to another embodiment of the present invention, three-dimensional segmentation of the same facility can be performed using point cloud data based on a lidar sensor. In this case, a class is assigned to each point or group (cluster) of points, and three-dimensional segmentation considering three-dimensional depth information is possible even in the same image.

According to this embodiment, even a single facility can be divided into multiple different classes based on its structural or functional characteristics. For example, a window (703) can be assigned different classes depending on whether the wall adjacent to the window is damaged or not, and different damage assessment models can be applied to each. This configuration is intended to distinguish between high-risk areas (e.g., areas adjacent to cracks) and relatively stable areas (e.g., areas other than damage) to enable more precise analysis.

Furthermore, segmentation results are not limited to the results from a single frame, but can be determined by continuously analyzing multiple frames. For example, if a facility class is not continuously detected in a specific frame and a certain number of missing identifications occur in the middle, the area may be considered noise or estimated as a potential damaged area and resegmented into a new, separate class area. For example, if a vertical crack exists in the wall (702) and the segmentation boundary is not clear, the crack area may be separated from the original wall class and assigned a separate class as a damage estimated area. This area may then be designated as a priority analysis target by a damage assessment model.

Figure 8 is a comparative diagram illustrating the improved effects of a damage assessment processing method according to one embodiment of the present disclosure. Referring to Figure 8, the image (810) on the left shows the results of applying a single damage assessment model to the entire image, in accordance with a conventional technique. In contrast, the image (820) on the right shows the results of analyzing a facility by first segmenting it and then applying a damage assessment model corresponding to each segmented facility class, in accordance with one embodiment of the present invention.

Although the image (810) contains various objects such as a wall (811), a pillar (812), and a waiting area (813), since segmentation of the facility is not performed in advance, the same damage assessment model is applied to the entire image, which makes it difficult to accurately identify the shape or location of damage to the structure. For example, the waiting area (813) is processed identically without separate distinction even though it is not a subject of analysis, which results in a waste of computational resources and a decrease in the accuracy of damage assessment.

On the other hand, image (820) shows an example in which each facility class (e.g., wall (821), floor (822), waiting space (823), etc.) is segmented by using the segmentation model according to the present invention, and damage assessment is performed only on the class corresponding to an actual building structure. Accordingly, damage assessment is applied only to structures such as the building exterior, windows, and floor, and unnecessary analysis areas such as the waiting space (823) are excluded, thereby reducing unnecessary operations and enabling more precise recognition of the structural boundary of the object. In particular, in image (820), the location, shape, and boundary of the building exterior wall (821) can be more clearly identified, and the floor surface (822) is processed as a separate class distinct from other structures, thereby establishing a foundation for independently analyzing cracks, damage, etc. of the floor. In addition, in distinguishing the boundary between the exterior surface of a building and an air space, the present invention can infer three-dimensional spatial information by estimating the vanishing point of an image (based on the floor surface) and the slope line of a structure perpendicular thereto, thereby enabling more precise judgment of the exterior surface of a building structure and its boundary.

According to one embodiment of the present invention, damage assessment to determine whether a damage class is included is performed not only based on the external appearance information of a single structure, but also by considering the positional relationship between the facility class and one or more adjacent facility classes, as well as damage information detected within the adjacent facility classes. For example, when assessing damage for a column class, a beam class connected to or adjacent to the column may also be considered. In this case, since the beam class and the column class are structurally connected to each other or share loads, the geometrical conditions of the beam, such as its inclination, deflection, and crack direction, can be important factors in assessing whether the column is damaged. In particular, if the beam is inclined at a certain angle or more with respect to the horizontal axis, there is a possibility of a load imbalance at the connection portion, which may be associated with the possibility of cracking or fracture of the column. Accordingly, the damage assessment model can receive such inclination information as input and adjust the assessment algorithm. Furthermore, if delamination damage is detected in the beam class and moisture (determined by moisture reflectance) is detected around the delamination area, the structural integrity of the beam may be presumed to have deteriorated. In this case, if the column and beam classes are structurally integrated into a masonry or reinforced concrete structure, there is a possibility that the column may have also experienced durability degradation or secondary damage due to water leakage. In such cases, the damage assessment model can be configured to integrate the damage characteristics of surrounding facility classes and environmental factors in addition to texture changes within a single image.

In another embodiment, the relative positional relationship between structures may include one or more of the following: distance between two structures, distance between centers of gravity, angle between centerlines, directionality of joints, and height difference, from which damage propagation paths, stress concentration areas, and the possibility of structural cascade effects can be inferred. Therefore, the damage assessment model may be composed of a model that learns the context between adjacent structures together, rather than a single image-based classification model, and may select different assessment paths depending on the location and damage correlation.

According to one embodiment of the present specification, the positional relationship with another facility class may include an angle between the two facility classes, a center of gravity of the two facility classes, and damage information within the adjacent other facility class may include crack shape information including the direction and depth of the crack.

FIG. 9 is a drawing illustrating the structural positional relationship and damage judgment criteria of a beam and a column according to one embodiment of the present specification. Referring to FIG. 9, a beam (911) and a column (912) have a structure that are connected to each other, and their connection portion (920) is a part with a high possibility of damage and can be used as a judgment criterion. In this embodiment, one or more of various factors may be considered to determine the positional relationship between facility classes. For example, one or more of the relative angle between two facility classes, the distance between their center positions, or the relative direction vector of the center positions may be included. For example, if a beam (911) vertically connected to a column (912) is not maintained in a horizontal direction but is tilted, whether the beam is sagging or tilting can be determined through the angle information between the two facilities. In addition, the center of gravity of each facility can be calculated based on the external characteristics, location coordinates, and facility type, and a change in the center of gravity can indirectly indicate a structural abnormality.

Additionally, in one example, damage information occurring within adjacent facility classes can be used as an auxiliary criterion for judgment. For example, if a crack that propagates vertically within a column (912) extends to a connection (920) of a beam (911), the direction and depth of the crack can be key indicators of the potential for damage propagation. Thus, crack directionality, whether it is curved, and the depth of fracture can be defined as crack shape information, and based on this information, a damage judgment model can be selected or adjusted for more precise application.

In one embodiment of the present specification, the center of gravity of two facility classes can be determined based on surface pattern information of the target facility class, shape information of the target facility class viewed from multiple angles, and building information of a building including the target facility class. For example, in a structure where a wall and a column meet, if the surface of the column includes a repetitive pattern (e.g., traces of concrete mold, crack distribution, etc.), the density and center of distribution of this pattern can be reflected in the calculation of the center of gravity. In addition, even for the same column, a more precise center position can be determined by reconstructing the three-dimensional shape based on image data captured from various angles, such as the top, side, and bottom.

Additionally, it can incorporate additional building design information, including the structural role the structure plays within the building (e.g., load-bearing, internal compartments, exterior walls, etc.). For example, in the area where beams and columns intersect, the center of gravity can be defined as the structural center where the load is concentrated, rather than simply the geometric center.

According to one embodiment of the present specification, a segmentation model may perform a segmentation function to distinguish the location and boundary of each facility object from input image data. Here, the segmentation function may mean a process of distinguishing meaningful objects (e.g., pillars, walls, windows, etc.) into different regions at the pixel or point level within the image. Specifically, the segmentation model may perform multiple convolution operations with different receptive fields in parallel to effectively identify facility classes with different sizes and shapes within the image. The receptive field refers to the range of the area where each convolution filter collects information from the input image, and a small receptive field can be used to identify detailed local information, and a large receptive field can be used to identify structural global information. The results of each convolution operation performed in parallel are then integrated in a single integration layer, so that a segmentation result indicating which facility class each pixel belongs to can be generated.

According to another embodiment of the present disclosure, a segmentation model can be trained as independent subnetworks for indoor and outdoor structures. For example, indoor lighting is constant and the background consists of walls, ceilings, and floors, whereas outdoor images differ in image characteristics due to the influence of natural light, sky, trees, and roads. To reflect these differences, one network can be trained to segment facilities that often appear in indoor environments (e.g., wallpaper, lights, ceiling structures, etc.), while another network can be trained to process structures in outdoor environments (e.g., exterior walls, streetlights, exterior pillars, etc.). When an image acquired from a disaster site is input, whether it is indoor or outdoor can be determined based on background information (e.g., presence of sky, floor type, etc.) and illuminance information (e.g., whether natural or artificial light is used).

Furthermore, in one example, a damage assessment model can be trained based on diverse image data collected from disaster sites. Specifically, training can include image data for the same damage class (e.g., cracks, delamination, detachment, etc.) under various lighting conditions, shooting angles, and resolutions. For example, even the same crack may appear as a shadow under low lighting conditions, be perceived as distorted at an oblique angle, and appear blurry at low resolutions. A model that learns how damage is represented under these diverse conditions can accurately assess damage even in situations where lighting or shooting conditions are uneven, such as at actual disaster sites.

According to one embodiment of the present disclosure, a damage assessment model can evaluate the probability of occurrence of each damage class by accumulating it over time. Specifically, if multiple frames or image data captured at multiple points in time at a single disaster site exist, the probability of damage can be continuously tracked, accumulated, and analyzed for each frame. The converted location information can be accurately mapped onto a 3D digital map or point cloud model of the site. This allows users or administrators to visually identify the spatial location of damaged facility classes. For example, if a crack develops at the top of a column, a red dot or annotation can be displayed at the top of the column on a 3D model.

A segmentation model according to one embodiment of the present invention is a neural network-based model that performs a segmentation (semantic segmentation) function to segment various facility objects (e.g., pillars, walls, windows, etc.) present in a disaster scene image into pixel units. Specifically, in this embodiment, a segmentation network based on the YOLOv8 structure can be applied. For example, the YOLOv8-based network is characterized by a lightweight backbone and a fine-grained head structure, and includes a unit structure capable of performing object detection and segmentation simultaneously. The network extracts a feature map from input image data, and then performs multiple convolution operations with different receptive fields in parallel, thereby increasing the segmentation accuracy for facility objects of various sizes and shapes.

According to one embodiment of the present disclosure, facility class information corresponding to a damage class can be mapped to corresponding locations on a 3D map of a disaster site and visually output. Specifically, damage information identified through an artificial intelligence model is processed together with mapping information indicating which facility class each damage class belongs to. At this time, the location information of the corresponding facility class (e.g., pillar, wall, ceiling, etc.) can be converted into coordinates in 3D space based on image coordinates or point cloud data, but the present invention is not limited thereto.

Advanced learning and damage information identification using environmental conditions

Fig. 10 is a flowchart of a method for identifying damage information based on environmental conditions at a disaster site according to one embodiment of the present specification. Referring to Fig. 10, the method for identifying damage information according to the present embodiment may be performed by a computing device and may include a step (S210) of receiving image data captured at a disaster site. The input image data may include, for example, RGB images acquired through a camera mounted on a disaster response robot or a fixed surveillance device, 3D point cloud information based on LiDAR, and multiple environmental information collected through illuminance and temperature/humidity sensors. Such data may be collected together with metadata for each image frame and utilized as basic information for subsequent processing.

Step S220 may involve determining environmental conditions contained in the input image data. These environmental conditions may be determined by analyzing metadata of the image frame or feature information of the image itself. For example, the overall color distribution of pixels within an image may be analyzed to estimate the illuminance level, or the shooting distance and angle may be calculated based on the degree of object distortion or sharpness. Furthermore, if information on the external temperature or humidity at the time of image collection is provided as metadata, this may be directly utilized.

In step S230, a process of segmenting multiple facility classes within image data may be performed using a segmentation model to which environmental conditions are applied. The segmentation model used at this time is a segmentation model learned under various environmental conditions, and is designed to reflect changes in appearance depending on illumination, distance, angle, etc., even for the same structure. In this embodiment, pixel-level classification is performed to determine whether each pixel within the image belongs to a specific facility class, such as a beam, column, wall, window, or floor. The segmentation model may be based on a multi-convolutional layer structure with different receptive fields, and such a structure enables the classification of objects with various sizes and resolutions.

In step S240, a damage assessment model selected based on the environmental conditions may be applied to the image area corresponding to each segmented facility class to identify the damage class. The damage assessment model may be categorized into various damage types, such as cracks, peeling, leakage, and breakage, and a model with a high recognition rate under the corresponding environmental conditions is selected and applied. For example, in a low-light environment, a damage assessment model optimized for dark images may be applied, and in an image with a wide shooting angle, a model reflecting distortion correction may be used.

In step S250, the identified damage class and the facility class information containing the damage can be output together. The output information can consist of not only coordinates on a 2D image, but also 3D location data combined with SLAM-based or GPS-based location information, allowing for a quick overview of the location and type of damage for each structure. The output information can be transmitted to a cloud server or on-site response device for further evaluation or reporting.

According to one embodiment of the present disclosure, environmental conditions may include one or more of humidity, temperature, illuminance, shooting distance, and shooting angle. Specifically, humidity can be used as an indicator of the moisture concentration in the outdoor or indoor air at the time of image capture, and may influence the identification of certain types of moisture-related damage, such as delamination or mold growth. Temperature can affect the thermal expansion of structural materials or visual distortion of damaged areas, and thus may be considered as a variable to improve analysis accuracy.

In addition, illuminance is a factor indicating the brightness of the environment in which image data is acquired, and can include not only external illuminance due to ambient light sources but also the original illuminance value of the image determined by the sensor settings of the photographing equipment. In particular, in the present embodiment, relative illuminance change values can be calculated by analyzing not only the illuminance measurement value included in the image metadata, but also the overall brightness distribution of pixels in the image or histogram-based color combination information. Such illuminance information can be used as a criterion for selecting a damage judgment model to correct the identification of a specific damage class when the identification performance varies depending on the illuminance.

Shooting distance and angle can also be considered key environmental conditions. Shooting distance affects the resolution and clarity of objects in an image, and is particularly important when identifying detailed damage, such as cracks. Because visual patterns can vary depending on the viewing angle, even for the same damage, the trained model needs to be calibrated accordingly. Therefore, this embodiment comprehensively analyzes multiple environmental conditions, allowing image data to be processed under conditions optimized for the segmentation model and damage assessment model.

Figure 11 illustrates the results of angle-based crack detection according to one embodiment of the present specification. Referring to Figure 11, the left side shows original video images taken from various angles for the same crack at a distance of 120 cm, and the right side shows the results of applying a damage detection model to images taken at different distances for the same crack.

In the left image of Fig. 11, “front” is an image captured from the front (0 degrees), and “left-30”, “left-45”, and “left-60” represent images captured at angles rotated 30 degrees, 45 degrees, and 60 degrees to the left with respect to the target crack, respectively. Similarly, “right-30”, “right-45”, and “right-60” represent images captured at the corresponding angles to the right. Although these multi-angle captured images show substantially the same damage (crack), since the visual pattern appears differently depending on the angle, there is a possibility that the image-based artificial intelligence model may misrecognize them as heterogeneous damage. Therefore, in this embodiment, we intend to improve the accuracy and consistency of detection by processing images captured from various angles for the same damage as separate learning data or utilizing them as correction factors in damage judgment. In addition, the right image of Fig. 11 shows the detection result by applying the damage judgment model based on images acquired from different shooting distances for the same crack. 30cm, 60cm, 90cm, 120cm, 150cm, 180cm, and 210cm respectively represent the distance between the image acquisition device and the surface where the crack exists, and show the difference in resolution and the change in recognition accuracy according to the change in distance. According to this embodiment, by collecting data from various angles and distances for the same damage and constructing a segmentation model and a damage judgment model that learn based on the data, there is an advantage in that the detection performance of damage information can be maintained even under various environmental conditions. In particular, since it is difficult to systematically construct learning data in disaster sites, this embodiment can provide a structure that regards image data acquired for the same object under various conditions as different learning samples or increases the identification accuracy through correction data between the same samples.

FIG. 12 is a diagram illustrating damage detection results according to illumination change conditions for a crack class according to one embodiment of the present specification. Referring to FIG. 12, based on three original images of Case 1, Case 2, and Case 3 reflecting different environments, various images with illumination reduced from -90% to -10% are generated, and the results of evaluating crack detection performance for these are presented. Under conditions where illumination is reduced by -70% or more, some cracks begin to be recognized, and as the illumination level recovers to -60% or less, the accuracy of damage detection and the box-shaped segmentation results become clearer. In this way, by including data under various illumination conditions for the same crack class and learning or evaluating performance, the sensitivity and threshold level of damage detection according to illumination change can be quantitatively identified, and a segmentation model robust to illumination change can be constructed.

Figure 13 illustrates the results of analyzing damage detection performance according to changes in image illumination according to one embodiment of the present disclosure. As illustrated in Figure 13, damage detection results were compared for the same disaster scene image when the illumination was adjusted to -80%, -50%, 0% (original), +50%, and +100% compared to the original. The red box in each image indicates the identified damage class, labeled "crack" and the detection probability (score). The analysis results show that most damage was not detected under the extremely low -80% illumination condition, while only some damage was detected under -50%. On the other hand, damage such as cracks were reliably detected within a brightness adjustment range of ±50% of the original illumination. In particular, when the illumination increased to +100%, some cracks were clearly visible, but detection performance degraded due to reflection or overexposure. Therefore, in this embodiment, we experimentally verified the impact of changes in image illumination on damage detection performance and deduced that the optimal illumination range for damage assessment was between -50% and +50%.

According to one embodiment of the present disclosure, the segmentation model may be trained based on a training data set that includes image data acquired under different illumination conditions, angles, or shooting positions for the same damage type. Specifically, by collecting multiple image data captured under different environmental conditions for a single damage, such as a crack or delamination, and organizing these data into training samples corresponding to a single damage class, the model can be trained to produce consistent segmentation results under various conditions. For example, as illustrated in Figure 11, the training data may include images of the same damage captured at a distance of 120 cm, captured at various angles ranging from 60 degrees to the left to 60 degrees to the right, or images captured under various illumination conditions from the same location with only the illumination changed. This data complements the problem of visual representations of damage varying across different image conditions and helps the model recognize the geometric and morphological characteristics of damage in a more general manner. Furthermore, a learning structure that reflects this environmental diversity can contribute to improving segmentation accuracy even in situations where images acquired at actual disaster sites are inconsistent or where illumination or shooting angles may differ significantly. For example, since the directionality of a crack may be subtle under certain illumination conditions but clearly apparent under other angles, training data from diverse conditions can contribute to the model's ability to better recognize the morphological consistency of damage.

According to one embodiment of the present specification, the step of acquiring image data may include a procedure of determining environmental conditions that can improve the accuracy of damage identification by considering the type of damage class to be identified and the shape, arrangement, positional relationship, etc. of the structure where damage has occurred, and then adjusting the image acquisition conditions to match the environmental conditions.

For example, in the case of crack damage, if the illumination is too high or too low, the crack boundary can be blurred, and in certain structures, the identification can be improved when the light is irradiated in an oblique direction. On the other hand, in the case of delamination damage, frontal or near-vertical illumination may be advantageous. Considering these conditions, a computing device or robot control unit can predict the type of damage before image acquisition or analyze the shape of the structure, and then dynamically adjust the illumination angle and intensity of the light source, the number of lights, or the shooting position and direction of the sensor. This control can be achieved, for example, through a variable light source mounted on the robot, a rotating camera, or a distance sensor. In addition, the optimal shooting position can be determined to reduce interference between structures by considering the structure layout information, and appropriate environmental conditions can be configured in real time at that position.

According to one embodiment of the present specification, environmental conditions can be determined by analyzing the color combinations of all pixels included in the image data. Specifically, the computing device analyzes the overall color distribution by aggregating the color values, such as RGB or HSV, of each pixel in the image, and when a specific color combination is included in large numbers, the corresponding optimal illuminance value can be calculated through a reference table or learning model. For example, in the case of a concrete wall mainly composed of gray tones, low illuminance may cause problems in identifying damage boundaries, so illuminance conditions with high color contrast can be automatically selected.

Additionally, the distribution of shaded or bright areas within an image can be estimated based on color combinations, which can be used to determine the location of light exposure. For example, if a specific area within an image contains a dark color combination due to shadows, a separate light exposure unit can be used to control the illumination of that area.

According to another embodiment of the present disclosure, the step of identifying damage classes may be performed by applying different damage assessment models to each structure class. Specifically, the computing device may select and apply a damage assessment model optimized for the characteristics of each structure class, such as beams, columns, walls, floors, and ceilings, segmented within the image. For example, a model sensitive to horizontal cracks or bending deformation may be used for beams, while a model that detects vertical cracks and compressive damage may be used for columns.

According to another embodiment of the present specification, the step of determining whether a damage class is included may be performed by determining a damage assessment model to be applied to the facility class based on the positional relationship between the facility class and other adjacent facility classes and damage information within the other adjacent facility classes. For example, when determining damage to a column class, the positional relationship, such as the angle, distance, and connection type, with the beam class connected to the column may be analyzed, and damage information, such as the direction and depth of cracks observed in the beam, may be additionally considered. For example, when determining whether a column class is damaged, it may be difficult to determine micro-cracks on the surface based solely on visual information of the column itself. In this case, if the analysis of the damage information of the beam class connected to the column reveals that a deep vertical crack exists in the beam and that the crack extends toward the joint where it meets the column, it may be determined that the column is also likely to have been damaged by the same load.

FIG. 14 is a diagram illustrating an image data preprocessing process according to one embodiment of the present disclosure. Referring to FIG. 14, image data (1500) contains a variety of damage types, and multiple damage classes, such as delamination (1510) and cracks (1521, 1522, 1523), may exist. For such image data (1500), a computing device can automatically analyze the types of damage classes included in the image and the pixel ratio or number of objects occupied by each damage class before applying a damage assessment model.

At this time, the computing device can determine the priority of the main damage class that requires more accurate detection in the image based on the area ratio, number of objects, or location information of each crack class and delamination class existing in the image. For example, in FIG. 14, if the image data (1500) includes multiple cracks (1521, 1522, 1523), and among them, the crack (1522) occupies the largest area and is located at a high risk location in terms of stress distribution of the structure, the computing device can determine the image acquisition strategy in a way that complements the damage detection performance centered on the crack (1522). More specifically, the computing device first analyzes the image features including the illumination, color contrast, texture change, etc. around each damage area, and can derive candidate light irradiation locations (1610, 1620, 1630, etc.) that are easy to detect damage by referring to a prediction model that has been learned in advance about how the damage is visually deformed depending on the light conditions. At this time, each candidate location can be set based on a certain distance and angle from the center of damage, and the optimal investigation location for the corresponding damage type can be derived through a dataset learned based on past similar types of cracks or peeling cases.

For example, in FIG. 14, when illumination correction is required for a crack (1522), the computing device can select a location (e.g., 1630) among the candidates for light irradiation locations that can most clearly highlight the crack shape, taking into account the directionality and surface reflection characteristics of the crack (1522). This selection may be determined not by a simple distance or angle, but by considering a specific pattern according to the damage class (e.g., a finely branched crack structure, a diffusion directionality from the center, etc.), and these criteria may be performed by a pre-learned light irradiation optimization model or a rule-based algorithm. After this processing, the computing device operates the actual light irradiation unit corresponding to the selected light irradiation location to reacquire the image, or controls the current image data to be corrected through an image processing technique (e.g., brightness correction of the ROI area, contrast enhancement, etc.). In this way, the present embodiment may include a dynamic light irradiation location selection process that takes into account the damage distribution characteristics and structural locations of the image data.

According to one embodiment of the present specification, the selection of a light control method for damage determination may be determined based on the relative distribution of damage classes included in the image data. In this case, the 'ratio' may be calculated as the ratio of the number of pixels occupied by a specific damage class to the total number of pixels in the image data, or the ratio of the number of objects belonging to a specific class to the total number of damaged objects present in the image. For example, if the entire damaged area in the image is composed of approximately 1,000 pixels, of which 700 pixels are determined to be delamination areas and the remaining 300 pixels are determined to be cracks, the pixel ratio of the delamination class may be calculated as 70% and the crack class as 30%. Alternatively, if a total of 10 damaged objects are identified, and there are 7 delaminations and 3 cracks, the delamination ratio may be determined to be high.

Based on this judgment, the computing device can control the light irradiation unit to reacquire image data with relatively high illuminance in order to create high-illuminance conditions that are advantageous for improving the accuracy of delamination identification when the ratio of the delamination class is higher than that of the crack class. On the other hand, when the crack class is dominant, there is a concern that the fine pattern of the crack may be lost in a high-illuminance environment. Therefore, damage identification can be performed based on image data that maintains the current illuminance state without light irradiation, or illuminance compensation can be performed through an image processing method (e.g., contrast enhancement for a specific ROI, local equalization, etc.). This method can improve the consistency and reliability of damage detection performance by dynamically controlling the image acquisition conditions by reflecting the correlation between the light sensitivity characteristics of each damage type and the possibility of visual identification.

The embodiments described above may be implemented at least in part as computer programs and recorded on a computer-readable recording medium. A computer-readable recording medium on which a program for implementing the embodiments is recorded includes any type of recording device that stores data that can be read by a computer. Examples of computer-readable recording media include ROMs, RAMs, CD-ROMs, magnetic tapes, and optical data storage devices. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, such that computer-readable codes are stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the embodiments will be readily understood by those skilled in the art to which the embodiments pertain.

Although the present specification has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and variations of the embodiments are possible. However, such modifications should be considered within the technical protection scope of the present specification. Therefore, the true technical protection scope of the present specification should be determined to include other implementations, other embodiments, and equivalents to the claims, based on the technical spirit of the appended claims.

Claims (15)

A method for identifying damage information according to environmental conditions at a disaster site, performed by a computing device,
Step of receiving video data taken at a disaster site;
A step of determining environmental conditions included in the above image data;
A step of segmenting multiple facility classes from the image data using a segmentation model to which environmental conditions are applied;
A step of identifying a damage class by applying a damage judgment model selected according to the environmental conditions to an image area corresponding to each of the above-described divided facility classes; and
Including a step of outputting facility class information corresponding to the identified damage class,
Before applying the above damage assessment model,
Calculate the types of damage classes included in the corresponding image data and the pixel ratio or number of objects occupied by each damage class,
A method for identifying damage information according to environmental conditions at a disaster site, further comprising a step of selecting whether to operate a light irradiation unit to irradiate light of a specific intensity to re-acquire image data or to correct illuminance by performing image processing on the current image data, based on the type and ratio of the generated damage class.
In the first paragraph,
The above environmental conditions are,
A method for identifying damage information according to environmental conditions of a disaster site, characterized in that it includes at least one of humidity, temperature, illuminance, shooting distance, and shooting angle. In the first paragraph,
The above division model is,
A model characterized by being learned based on a learning data set that includes image data acquired at different illumination, angles, or shooting positions for the same damage type.
A method for identifying damage information based on environmental conditions at a disaster site. In the first paragraph,
The step of acquiring the above image data is:
Based on the type of damage class to be identified and the shape and layout information of the structure where the damage is located,
Determine the environmental conditions that provide high accuracy in identifying the damage;
A method for identifying damage information according to environmental conditions at a disaster site, including acquiring image data by adjusting the position of a light source or sensor to correspond to the determined environmental conditions.
In the first paragraph,
The above environmental conditions are,
Based on the color information derived by analyzing the color combination of all pixels included in the image data,
A method for identifying damage information according to environmental conditions at a disaster site, including setting the optimal illuminance value and light irradiation position corresponding to the color information.
In the first paragraph,
The steps of identifying the above damage class are:
A method for identifying damage information using facility segmentation in a disaster site, characterized in that different damage assessment models are applied according to each facility class.
In the first paragraph,
The step of determining whether the above damage class is included is:
A method for identifying damage information using facility segmentation within a disaster site, characterized in that a damage judgment model to be applied to a facility class is determined based on the locational relationship between the facility class and other adjacent facility classes and damage information within other adjacent facility classes.
delete In the first paragraph,
The above selection is,
When the ratio of the peeling class is higher than the ratio of the crack class, the light irradiation unit is operated to obtain image data with high illuminance,
A method for identifying damage information according to environmental conditions at a disaster site, including performing damage judgment while maintaining image data and omitting the operation of a light irradiation unit when the ratio of crack classes is above a certain standard, or performing illuminance correction to increase the accuracy of crack identification using an image processing method.
As a device for identifying damage information according to environmental conditions at a disaster site, the device,
Step of receiving video data taken at a disaster site;
A step of determining environmental conditions included in the above image data;
A step of segmenting multiple facility classes from the image data using a segmentation model to which environmental conditions are applied;
A step of identifying a damage class by applying a damage judgment model selected according to the environmental conditions to an image area corresponding to each of the above-described divided facility classes; and
Including a step of outputting facility class information corresponding to the identified damage class,
Before applying the above damage assessment model,
Calculate the types of damage classes included in the corresponding image data and the pixel ratio or number of objects occupied by each damage class,
Based on the type and ratio of the generated damage class, the step of selecting whether to operate the light irradiation unit to irradiate light of a specific intensity to re-acquire image data or to perform image processing on the current image data to correct the illuminance is further included.
A device for identifying damage information according to environmental conditions at a disaster site.

In Article 10,
The above environmental conditions are,
A device for identifying damage information according to environmental conditions of a disaster site, characterized in that it includes at least one of humidity, temperature, illuminance, shooting distance, and shooting angle.
In Article 10,
The above division model is,
A device for identifying damage information according to environmental conditions at a disaster site, characterized in that the model is learned based on a learning data set including image data acquired at different illumination levels, angles, or shooting positions for the same damage type.
In Article 10,
The step of acquiring the above image data is:
A device for identifying damage information according to environmental conditions at a disaster site, comprising: determining environmental conditions for high accuracy in identifying damage based on the type of damage class to be identified and the shape and layout information of the structure where the damage is located; and acquiring image data by adjusting the position of a light source or sensor to correspond to the determined environmental conditions.
In Article 10,
A device for identifying damage information according to environmental conditions of a disaster site, wherein the environmental conditions are set by determining the optimal illuminance value and light irradiation location corresponding to the color information based on the color information calculated by analyzing the color combination of all pixels included in the image data.
A computer program stored in a computer-readable medium for executing the method described in any one of claims 1 to 7 and 9 on a computer.