KR102848807B1 - Apparatus and method for reducing the weight of large capacity data - Google Patents

Abstract

A large-capacity data lightweighting device and method are disclosed, and a large-capacity data lightweighting device according to one embodiment of the present invention may include a data collection unit that collects large-capacity data from a plurality of servers, a data preprocessing unit that preprocesses the collected large-capacity data, and a data conversion unit that lightweights the preprocessed large-capacity data based on an artificial intelligence model.

Description

Apparatus and Method for Reducing the Weight of Large Capacity Data

The present invention relates to a device and method for reducing the weight of large-capacity data.

The size and complexity of large-scale 3D data is rapidly increasing due to the increasing need for high-resolution models in fields such as architectural design, medical imaging, and game development. To efficiently handle large amounts of data, various lightweighting techniques are being developed. Large-scale data lightweighting is a technology that reduces the original data size while retaining its core information and functionality. This technology saves data storage space, improves transmission speed, enhances processing efficiency, and enhances usability in real-time applications.

However, lightweighting techniques can degrade data quality and have limitations in applications that require precise shape and detail retention. For example, in medical imaging, accuracy is crucial to retain fine structures, but there is a risk of losing detailed information during the lightweighting process. Furthermore, the lack of level-of-detail (LOD) standards when integrating Building Information Modeling (BIM) and Geographic Information Systems (GIS) data makes it difficult to balance model quality with efficient processing. Furthermore, interoperability across various platforms and software remains a significant challenge, limiting data transfer and collaboration.

Accordingly, the present invention aims to provide a large-scale data lightweight device and method that can quickly extract and utilize necessary data from a large amount of data.

The technology underlying this application is disclosed in Korean Patent Publication No. 10-1687497.

The present invention is intended to solve the problems of the prior art as described above, and the purpose of the present invention is to provide a device and method for reducing the size of large-capacity data and improving transmission efficiency through data compression, deduplication, and instancing techniques for large-capacity data that is large in size and takes up a lot of storage space and network bandwidth.

However, the technical tasks to be achieved by the embodiments of the present invention are not limited to the technical tasks described above, and other technical tasks may exist.

As a technical means for achieving the above-mentioned technical task, a large-capacity data lightweighting device according to one embodiment of the present invention may include a data collection unit that collects large-capacity data from a plurality of servers, a data preprocessing unit that preprocesses the collected large-capacity data, and a data conversion unit that lightweights the preprocessed large-capacity data based on an artificial intelligence model.

Additionally, the data collection unit can analyze the attribute information and quality of large amounts of data, classify them into categories, and store them in a database.

Additionally, the data preprocessing unit can perform preprocessing corresponding to the data format of the large amount of data including raster data and vector data.

Additionally, the data preprocessing unit can perform at least one of coordinate system transformation, resolution adjustment, interpolation, or noise removal on the raster data.

Additionally, the data preprocessing unit can perform at least one of preprocessing among duplication removal between vertices, error correction, mesh simplification, and attribute information filtering on vector data.

Additionally, the data preprocessing unit can input preprocessed raster data into a pre-built general model to derive basic data.

Additionally, the data conversion unit can convert the basic data into vector format data by applying an interpolation algorithm.

Additionally, the data conversion unit can apply the LoD technique to vector format data to distinguish it into multiple resolutions.

Additionally, the data conversion unit can classify preprocessed vector data into multiple levels of complexity by applying the LoD technique.

Additionally, the data transformation unit can apply a spatial partitioning technique or an instancing technique to preprocessed large-capacity data and store it in the database.

Meanwhile, a method for reducing large-capacity data according to one embodiment of the present invention may include a step of collecting large-capacity data from multiple servers, a step of preprocessing the collected large-capacity data, and a step of reducing the preprocessed large-capacity data based on an artificial intelligence model.

The above-described problem-solving methods are merely exemplary and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, additional embodiments may be included in the drawings and detailed description of the invention.

According to the aforementioned means of solving the problem of this invention, by lightening and optimizing large-capacity 3D data, efficiency and performance are maximized at all stages of storage, transmission, visualization, search, and management, thereby providing practical effects such as cost reduction, performance improvement, and data stability guarantee, and there is an effect of improving usability in various application fields (GIS, BIM, VR/AR, game engines, etc.).

However, the effects that can be obtained from this center are not limited to the effects described above, and other effects may exist.

Figure 1 is a drawing showing a schematic configuration of a large-capacity data lightweighting system according to one embodiment of the present invention.
Figure 2 is a schematic block diagram of a large-capacity data lightweight device according to one embodiment of the present invention.
FIG. 3 is a diagram illustrating an example of clustering large amounts of data into vertices by applying the KD Tree technique according to one embodiment of the present invention.
FIG. 4 is a drawing exemplarily showing the application of a spatial division technique to large-capacity data according to one embodiment of the present invention.
FIG. 5 is a drawing showing a Triangle Index configuration using the LoD technique according to one embodiment of the present invention.
FIG. 6 is a diagram illustrating an example of reducing large-capacity data based on a block reference according to one embodiment of the present invention.
Fig. 7 is a drawing exemplarily showing how to derive coordinates and height for vector format data according to one embodiment of the present invention.
Figure 8 is a flowchart of an operation for a large-capacity data lightweighting method according to one embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail to facilitate easy implementation by those skilled in the art. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity, and similar reference numerals have been used throughout the specification to indicate similar elements.

Throughout this specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" or "indirectly connected" with another element in between.

Throughout this specification, when it is said that a member is located “on,” “above,” “upper,” “lower,” “lower” or “lower” another member, this includes not only cases where the member is in contact with the other member, but also cases where another member exists between the two members.

Throughout this specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

The present invention relates to a device and method for reducing the weight of large-capacity data.

Figure 1 is a drawing showing a schematic configuration of a large-capacity data lightweight system (10) according to one embodiment of the present invention.

Referring to FIG. 1, a large-capacity data lightweight system (10) may include a large-capacity data lightweight device, a server (40), a user terminal (30), and a network.

In Fig. 1, one large-capacity data lightweight device is illustrated as being connected to one server (40) or user terminal (30), but it is not limited thereto and may be connected to multiple servers (40) that provide large-capacity data, and may include multiple user terminals (30) that access lightweight large-capacity data.

The user terminal (30) includes all personal computers, laptops, etc., and may include all types of wired/wireless communication devices such as smartphones, smart pads, tablet PCs, PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), and Wibro (Wireless Broadband Internet) terminals.

A large-capacity data lightweight device, a server (40) and a user terminal (30) can communicate with each other through a network (20).

A network (20) refers to a connection structure that enables information exchange between each node, such as terminals and servers (40), and examples of such a network (20) include, but are not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a 5G network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a wifi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, etc.

Figure 2 is a schematic block diagram of a large-capacity data lightweight device according to one embodiment of the present invention.

Referring to FIG. 2, a large-capacity data lightweight device may include a data collection unit (110) that collects large-capacity data from a plurality of servers (40), a data preprocessing unit (120) that preprocesses the collected large-capacity data, and a data conversion unit (130) that lightweights the preprocessed large-capacity data based on an artificial intelligence model.

According to one embodiment of the present invention, the data collection unit (110) can collect large amounts of data from multiple servers (40).

Here, large-capacity data may refer to BIM (Building Information Modeling) data including 3D modeling information of a building, and information on building elements, structures, facilities, and materials; CityGML data including building, traffic, terrain, and land use information in a standard format expressing 3D geographic information of a city; DEM (Digital Elevation Model) data which is elevation information data in raster format expressing the elevation value of the terrain; but is not limited thereto; and may include various large-capacity data such as LiDAR (Light Detection and Ranging) data collected through laser scanning, and satellite image data including multispectral and high-resolution images captured by satellites such as Landsat, Sentinel, and MODIS.

Specifically, the data collection unit (110) includes a data collection module capable of collecting data through a network connection with a plurality of servers (40), and the data collection module supports various protocols such as HTTP, FTP, MQTT, and WebSocket, so that it can collect large amounts of data in various formats such as REST API, real-time streaming data, and file transfer.

More specifically, the data collection unit (110) can collect 3D information model (BIM, Building Information Modeling) data including CityGML, a digital elevation model (DEM), a digital topographic map, an orthophoto, and interior and exterior shape information and attribute information, which are open standard data model formats, by communicating with multiple servers (40) through a network (20).

According to one embodiment of the present invention, the data collection unit (110) can analyze attribute information and quality of large-capacity data, classify categories, and store them in a database.

Specifically, the data collection unit (110) extracts attribute information (e.g., data format, coordinate system, resolution, data size, etc.) of large-volume data, analyzes metadata to identify the characteristics of the data, and uses a quality evaluation module to evaluate the accuracy, consistency, and omission of large-volume data, and can separately label data containing noise, data with errors, and data of low quality.

In addition, the data collection unit (110) can store large amounts of data in a database by classifying them into categories such as data format (BIM, CityGML, DEM, etc.), purpose (terrain analysis, building modeling, traffic simulation, etc.), LoD level, resolution, and spatial scope.

For example, for BIM data, you can extract attribute information in IFC format, for CityGML data, geometry information and LoD level, and for DEM data, resolution and elevation range, and create a spatial index based on a specific city (CityGML), a specific building project (BIM), or a specific region's topography (DEM) data.

According to one embodiment of the present invention, the database may include a source data management unit for storing and managing large amounts of data, a pipeline setting unit for converting, deleting, moving, or merging large amounts of data, and a lightweight data storage unit for storing preprocessed and lightweight large amounts of data by classifying them according to categories.

Specifically, the original data management unit can store original data of large-scale data such as CityGML, BIM, and DEM, and manage them to enable efficient searches based on data attribute information (LoD level, resolution, etc.) according to user requests.

Additionally, the pipeline setup section can perform data conversion to convert the format of original data, data deletion, data movement to change the storage location of data, and data fusion to create a single integrated data set by fusing multiple original data (e.g., combining CityGML data and DEM data).

Additionally, the pipeline setting unit can monitor pipeline setting changes and analyze the number of setting changes in conjunction with the user interface.

Specifically, the pipeline setting unit can manage pipeline setting changes by visually displaying the change history of pipeline settings, the status of each task (e.g., completed, failed, in progress), etc. through the screen of the user interface, and calculating the number of pipeline setting changes and providing the result to the user.

In addition, the pipeline setting unit analyzes the frequency and pattern of setting changes based on the number of pipeline setting changes, and if the number of setting changes is less than a preset threshold, the pipeline setting change pattern can be predicted based on a pipeline setting model built based on the setting change pattern, thereby automatically adjusting the pipeline.

For example, if the user is working on a project to build a city model using CityGML files and DEM data, the pipeline setting section can automatically predict pipeline settings and configure pipelines based on the pipeline setting model by repeatedly performing tasks such as data conversion, coordinate system conversion, and attribute filtering when the user uploads new large-volume data.

In addition, the lightweight data storage unit can store and categorize lightweight data by format (BIM, CityGML, DEM, etc.), purpose (terrain analysis, building modeling, traffic simulation, etc.), LoD level, resolution, spatial scope, etc., and optimize search and query performance through indexing.

According to one embodiment of the present invention, the data preprocessing unit (120) can preprocess collected large volumes of data.

Specifically, the data preprocessing unit (120) can convert large amounts of data into an optimized format and improve quality to facilitate subsequent processing and analysis.

According to one embodiment of the present invention, the data preprocessing unit (120) can perform preprocessing corresponding to the data format of large-capacity data including raster data and vector data.

Here, raster data represents pixel-based data, and is data that stores geographical or spatial information in the form of an image. DEM (Digital Elevation Model), orthophotos, satellite images, aerial photographs, land cover maps, etc. are representative raster data, and are saved in formats such as GeoTIFF, PNG, JPEG, and NetCDF. Each pixel has a value associated with spatial coordinates, and the higher the resolution, the larger the data size, and high-resolution data takes up a very large capacity. Vector data is data that expresses spatial information through geometric elements such as points, lines, and polygons. It is mainly used in BIM data, CityGML data, and GIS data, and defines various objects such as road networks, building boundaries, and terrain boundaries, and is saved in formats such as GeoJSON, Shapefile (.shp), GML, KML, and DXF.

Specifically, the data preprocessing unit (120) can classify the collected large volume of data into either raster data or vector data, and perform corresponding preprocessing according to the classification result.

According to one embodiment of the present invention, the data preprocessing unit (120) can perform at least one preprocessing among coordinate system transformation, resolution adjustment, interpolation, or noise removal on raster data.

Specifically, the data preprocessing unit (120) may perform at least one of preprocessing among coordinate system conversion to provide consistent geographic reference and prevent confusion in subsequent analysis processes by converting raster data using various coordinate systems into a standard coordinate system (e.g., EPSG:4326, etc.); resolution adjustment to adjust the size and resolution of raster data to suit the analysis purpose by changing the resolution of raster data; interpolation to fill in missing values or missing data; and noise removal to remove noise included due to sensor errors or data collection environment problems. However, it should be understood that the preprocessing techniques are not limited to the above-described preprocessing techniques and that raster data may be preprocessed using various preprocessing techniques.

According to one embodiment of the present invention, the data preprocessing unit (120) can perform at least one preprocessing among duplication removal between vertices, error correction, mesh simplification, and attribute information filtering on vector data.

Specifically, the data preprocessing unit (120) may perform at least one of preprocessing among: deduplication between vertices that constitute vector data to reduce data complexity and optimize capacity (for example, when there are duplicate vertices at the same location in polygon boundaries or linear data (road networks), merge them into a single vertex to make the data lightweight); error correction for unclosed polygons, intersecting line segments, duplicated objects, etc.; mesh simplification to simplify the mesh structure by reducing the number of triangles or vertices; and attribute information filtering to remove attributes unnecessary for analysis among various attribute information (for example, name, height, material, date, etc.) included in vector data. However, it should be understood that the present invention is not limited to the above-described preprocessing techniques, and that vector data may be preprocessed using various preprocessing techniques.

FIG. 3 is a diagram illustrating an example of corrective clustering of large-scale data by applying the KD Tree technique according to one embodiment of the present invention.

The KD Tree technique is a spatial partitioning technique designed to efficiently search, segment, and cluster multidimensional data. Referring to Fig. 3, the data preprocessing unit (120) partitions the space of large-scale data into vertical or horizontal lines, with dense areas being more finely divided and low-density areas being more broadly divided, based on a predetermined density threshold. Data for each segmented area can then be displayed to remove duplicate points.

According to one embodiment of the present invention, the data preprocessing unit (120) can input preprocessed raster data into a pre-built general model to derive basic data.

Specifically, the data preprocessing unit (120) can input preprocessed raster data into a pre-built general model to derive basic data required for analysis.

For example, the data preprocessing unit (120) can input DEM data into a general model to generate basic data such as slope, elevation value, and terrain outline, and in the case of satellite images, can extract land cover classification (forest, city, farmland, etc.) and object detection (building, road, etc.) data. The general model is built through pre-learning based on various raster data, and automatically performs analysis according to the characteristics of the input data, and the generated basic data can be converted into a standard format so that it can be used for subsequent processing and analysis.

In addition, a general model according to one embodiment of the present invention may be, for example, a model based on multiple machine learning neural networks. The object recognition model may be designed to simulate the structure of the human brain on a computer and may include multiple network nodes that simulate neurons of a human neural network and have weights. The multiple network nodes may each form a connection relationship to simulate the synaptic activity of neurons that exchange signals through synapses. In addition, the general model may include, for example, a neural network model or a deep learning model developed from a neural network model. In a deep learning model, multiple network nodes may be located at different depths (or layers) and exchange data according to connection relationships. Examples of artificial intelligence models may include, but are not limited to, a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), and a BRDNN (Bidirectional Recurrent Deep Neural Network).

According to one embodiment of the present invention, the data conversion unit (130) can reduce the weight of large-capacity data preprocessed based on an artificial intelligence model.

Specifically, the data conversion unit (130) can analyze large amounts of preprocessed data through an artificial intelligence model and perform a lightweight task corresponding to the format of the data.

For example, the data conversion unit (130) can reduce unnecessary pixel information by applying a compression algorithm in the case of raster data, and reduce the complexity of vector data by removing redundancy between vertices, simplifying meshes, and filtering attribute information.

According to one embodiment of the present invention, the data conversion unit (130) can apply an interpolation algorithm to the basic data to convert it into vector format data.

Specifically, the data conversion unit (130) can apply an interpolation algorithm to the basic data to predict continuous shapes and surfaces and convert them into vector format data.

Interpolation algorithms include, but are not limited to, linear interpolation, which linearly connects adjacent pixel values to create contour lines or converts continuous boundaries into vector format; spline interpolation, which is a curve-based interpolation technique that creates smoother terrain surfaces and thereby generates precise vector data; and kriging interpolation, which is an advanced interpolation technique that considers spatial correlation to predict elevation values or environmental data and generate geometry in continuous vector format.

The data conversion unit (130) can convert basic data (raster data) into vector format data (points, lines, polygons, etc.) based on the interpolation result obtained by applying an interpolation algorithm to the basic data. In addition, the data conversion unit (130) can store the generated vector format data in a lightweight data storage unit by classifying the categories in a standard format (GeoJSON, Shapefile, GML, etc.).

According to one embodiment of the present invention, the data conversion unit (130) can apply the LoD technique to vector format data to distinguish it into multiple resolutions.

LoD (Level of Detail) is a technology used in 3D modeling, GIS (Geographic Information Systems), and computer graphics, and is a technique that provides data of various resolutions by controlling the complexity and level of detail of an object. LoD is mainly divided into stages from LoD0 to LoD4. LoD0 is the simplest form, which only includes the location or basic boundary of an object (for example, in the case of a building, only the geographical location and simple outline are displayed); LoD1 is a simple 3D outline model that simply expresses only the shape and height of a building; LoD2 is a detailed exterior model including the roof shape and main structure of a building, which is used in urban simulation or terrain analysis and represents the rough exterior of a building; LoD3 is a high-resolution model that includes the exterior details of a building (such as windows, doors, and balconies); and LoD4 is the most detailed model that includes the interior structure of a building and expresses elements such as rooms, stairs, and interior decorations.

Specifically, the data conversion unit (130) applies the LoD technique to vector format data to classify data into a detailed level of resolution of one of LoD0 to LoD4 resolutions, and the generated vector format data for each resolution can be stored in a lightweight data storage unit in a standard format such as GeoJSON, Shapefile, or CityGML.

According to one embodiment of the present invention, the data conversion unit (130) can classify preprocessed vector data into multiple levels of complexity by applying the LoD technique.

Specifically, the data conversion unit (130) can apply the LoD technique to preprocessed vector data, i.e., vector data including preprocessed BIM data, CityGML data, GIS data, etc., to classify them into various levels of complexity according to the complexity.

In summary, the data conversion unit (130) can reduce the weight of data by adjusting the level of detail of terrain and boundaries by applying the LoD technique based on resolution to vector format data, which is preprocessed raster data, and can also reduce the weight by applying the LoD technique based on structural complexity to preprocessed vector data, such as BIM data, CityGML data, and GIS data, to structurally detail buildings and objects.

For example, the data conversion unit (130) can apply the LoD technique to preprocessed raster data based on DEM data to classify it by resolution, and in LoD0, only the main terrain outline is expressed, and in LoD2, detailed contour lines reflecting even small elevation changes are generated, thereby increasing the accuracy of terrain analysis. On the other hand, vector data such as BIM or CityGML can be classified by the LoD technique according to complexity, with LoD0 being a simple 2D outline, LoD3 including detailed structures such as windows, doors, and balconies, and LoD4 expressing even the internal structure, so that they can be used for VR/AR simulation or precise architectural design analysis, and can be stored in a lightweight data storage unit.

According to one embodiment of the present invention, the data conversion unit (130) can apply a spatial division technique or an instancing technique to preprocessed large-capacity data and store it in a database.

In relation to this, FIG. 4 is a drawing exemplarily showing the application of a spatial division technique to large-capacity data according to one embodiment of the present invention.

Referring to FIG. 4, the data conversion unit (130) can hierarchically divide preprocessed large-capacity data by applying the Octree partitioning technique, which is one of the spatial partitioning techniques, in order to efficiently manage preprocessed large-capacity data. Here, the Octree partitioning technique is a recursive structure that repeatedly divides a 3D space into 8 sub-nodes, and can divide it as finely as necessary based on the data density and location within the space. In addition, when handling large-scale 3D models or terrain data, memory usage can be optimized by dividing it into smaller units in areas with high data density and larger units in areas with low data density. The Octree partitioning technique provides an index to enable fast location-based searches of data objects within the space, thereby enabling efficient retrieval of data within a specific range or location.

In addition, the data conversion unit (130) can dynamically adjust the resolution or complexity level of objects in large-capacity data by applying the octree division technique and the LoD technique.

In this regard, FIG. 5 is a drawing showing a Triangle Index configuration by applying the LoD technique according to one embodiment of the present invention.

The LoD technique defines resolution or complexity based on a logarithmic scale, and is a technique that divides space more finely when the resolution or complexity is high (the LoD value is high), and divides space into large units when the precision is low (the LoD value is low). Referring to FIG. 5, the data conversion unit (130) applies the octree division technique and the LoD technique to dynamically adjust the LoD level by generating a detailed triangle index in an area where higher precision is required according to the camera's viewpoint or user's request, and generating a simplified index in an area where low precision is allowed, thereby generating a fast TriangleIndex in real-time 3D rendering or visualization.

In addition, the data conversion unit (130) stores data of a hierarchical structure generated by applying Octree division and LoD technique to preprocessed large-capacity data in a database (lightweight data storage unit), and each node can be defined to have a resolution or complexity level based on the LoD value.

In addition, the data conversion unit (130) can avoid duplicate storage and reduce memory usage by using the instancing technique when the same object appears repeatedly in multiple locations during the process of applying the space division technique.

Instancing is a method of efficiently managing objects by creating only references instead of copies of the object when the same object appears repeatedly, in order to optimize memory and performance.

In this regard, FIG. 6 is a drawing exemplarily showing a method of reducing large-capacity data based on a Block Reference according to one embodiment of the present invention.

Block Reference is a method of efficiently managing repetitive data objects by referencing the same data block without duplicating it. Referring to FIG. 6, the data conversion unit (130) can reduce duplicate data by using a reference structure and hierarchical division to efficiently manage large amounts of data.

For example, in urban modeling, the data conversion unit (130) can minimize data duplication by managing the same type of tree or furniture object as an instance when the same type of tree or furniture object is repeated in multiple locations. Data stored based on the LoD technique can be dynamically loaded, so that when a user requests data for a specific location or range, the data can be loaded in accordance with the required resolution or complexity, thereby reducing memory usage and maximizing rendering performance by selectively loading only the necessary portion without loading unnecessary high-precision, large-capacity data.

Fig. 7 is a drawing exemplarily showing how to derive coordinates and height for vector format data according to one embodiment of the present invention.

Referring to FIG. 7, the data conversion unit (130) can perform conversion and optimization work for real-time analysis and visualization based on preprocessed large-capacity data.

Specifically, the data conversion unit (130) can analyze preprocessed large-capacity data and dynamically apply LoD techniques according to the camera viewpoint or user request. More specifically, the data conversion unit (130) can determine an appropriate resolution or complexity level according to the user's viewpoint and, as needed, convert from low resolution or low complexity (simplified data in a wide range) to high resolution or high complexity (detailed data). For example, in city modeling, the data conversion unit (130) can load only simple outlines when the camera is far away, and load detailed structure and texture data when it approaches closer.

In addition, the data conversion unit (130) can apply an interpolation algorithm to calculate color and height values at arbitrary coordinates of large-capacity data. The data conversion unit (130) selects adjacent points (4 to 5) around specific coordinates based on the interpolation algorithm, and applies an interpolation algorithm such as linear interpolation, bilinear interpolation, or spline interpolation based on the selected points to derive and store color (RGB value) and height (Z value), and can reduce memory usage and maximize rendering performance through dynamic loading that selectively loads only data corresponding to a user's request.

In summary, the data collection unit (110) of the present invention collects and classifies data from a plurality of servers (40) (e.g., BIM files, CityGML data, DEM data), analyzes the format, properties, and quality of large-scale data during the collection process, classifies and stores data according to categories based on the analysis results, and identifies and excludes damaged or incomplete data based on the quality analysis results and stores them in a database. The data preprocessing unit (120) converts the collected large-scale data into a consistent standard format, performs preprocessing such as coordinate system conversion, resolution adjustment, and noise removal on raster data (DEM, orthophoto, satellite image), and performs preprocessing such as vertex duplication removal, error correction, and mesh simplification on vector data (BIM, CityGML). The KD Tree technique is used to spatially divide and cluster large-scale data to support fast searches, and can be effectively utilized in GIS analysis and large-scale 3D modeling. In addition, the data conversion unit (130) can optimize and reduce the weight of preprocessed large-capacity data by applying a spatial partitioning technique, a LoD (Level of Detail) technique, a Block Reference technique, an instancing technique, etc. to the preprocessed data. As a spatial partitioning technique, an Octree technique can be applied to hierarchically partition large-capacity data, and to partition more finely in high-density areas and more coarsely in low-density areas, thereby reducing memory usage. In addition, the data conversion unit (130) can improve performance in real-time visualization by applying the LoD technique to dynamically adjust the resolution or complexity of data according to the camera viewpoint, thereby displaying close objects in high precision and distant objects in a simple form, and by managing the same objects that appear repeatedly through the Block Reference technique and the instancing technique as a reference structure, thereby minimizing memory usage.

In addition, the large-capacity data lightweighting device can provide efficient search and accessibility to users by categorizing large-capacity data by format, attribute, and purpose in the database after data preprocessing and data conversion.

According to one embodiment of the present invention, a data compression unit (not shown) can transmit lightweight large-capacity data by applying either compression or streaming.

Specifically, the data compression unit can transmit lightweight, large-capacity data by applying any one of the following techniques: Geometry Compression, Attribute Data Compression, and Large-scale Data Streaming.

More specifically, the data compression unit can transmit large amounts of data by selecting one of the following: geometry compression, which compresses mesh data of large amounts of data to optimize information such as vertices, faces, and textures; attribute data compression, which compresses attribute information (color, texture, height, etc.); and large-scale data streaming, which partially streams only necessary data from large amounts of data to reduce memory usage and optimize network bandwidth.

According to one embodiment of the present invention, the optimization unit can perform data lightweighting by selecting an optimal technique among techniques performed during data preprocessing or data conversion for large amounts of data based on a pre-built optimization model.

Specifically, the optimization model is built by learning data lightweight processing information (lightening rate, lightening speed, etc. according to preprocessing and transformation techniques), and can determine the workload to perform data lightweighting by selecting one of multiple data preprocessing and data transformation techniques for the collected large amount of data.

Below, we will briefly review the operating flow of the present invention based on the detailed description above.

Figure 8 is a flowchart of an operation for a large-capacity data lightweighting method according to one embodiment of the present invention.

The large-capacity data lightweighting method illustrated in Fig. 8 can be performed using the large-capacity data lightweighting device described above. Therefore, even if omitted below, the description of the large-capacity data lightweighting device can be equally applied to the description of the large-capacity data lightweighting method.

In step S11, the data collection unit (110) can collect large amounts of data from multiple servers (40).

Additionally, in step S11, the data collection unit (110) can analyze attribute information and quality of large-capacity data, classify the categories, and store them in a database.

Next, in step S12, the data preprocessing unit (120) can preprocess the collected large volume of data.

Additionally, in step S12, the data preprocessing unit (120) can perform preprocessing corresponding to the data format of the large-capacity data including raster data and vector data.

Additionally, in step S12, the data preprocessing unit (120) can perform at least one of coordinate system transformation, resolution adjustment, interpolation, or noise removal preprocessing on the raster data.

Additionally, in step S12, the data preprocessing unit (120) can perform at least one of preprocessing among duplication removal between vertices, error correction, mesh simplification, and attribute information filtering on vector data.

Additionally, in step S12, the data preprocessing unit (120) can input preprocessed raster data into a pre-built general model to derive basic data.

Next, in step S13, the data conversion unit (130) can reduce the weight of large-capacity data preprocessed based on the artificial intelligence model.

Additionally, in step S13, the data conversion unit (130) can apply an interpolation algorithm to the basic data to convert it into vector format data.

Additionally, in step S13, the data conversion unit (130) can apply the LoD technique to vector format data to distinguish it into multiple resolutions.

Additionally, in step S13, the data conversion unit (130) can apply the LoD technique to the preprocessed vector data to classify it into multiple complexities.

Additionally, in step S13, the data conversion unit (130) can apply a spatial division technique or an instancing technique to the preprocessed large-capacity data and store it in the database.

In the above description, steps S11 to S13 may be further divided into additional steps or combined into fewer steps, depending on the implementation example of the present invention. Furthermore, some steps may be omitted as needed, and the order of the steps may be changed.

A method for reducing the weight of large-capacity data according to one embodiment of the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the medium may be those specifically designed and configured for the present invention or may be those known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Additionally, the aforementioned method for reducing the weight of large-capacity data can also be implemented in the form of a computer program or application executed by a computer and stored in a recording medium.

The above description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

10: Large-capacity data lightweight system
20: Network
30: User terminal
40: Server
110: Data Collection Department
120: Data Preprocessing Unit
130: Data conversion unit

Claims (12)

In a device that reduces large amounts of data,
A data collection unit that collects large amounts of data from multiple servers;
A data preprocessing unit that preprocesses the large volume of data collected above; and
A data conversion unit that reduces the weight of the preprocessed large-capacity data based on an artificial intelligence model;
Including,
The above data preprocessing unit,
Preprocessing corresponding to the data format of the above large data including raster data and vector data is performed,
The above data preprocessing unit,
For the above raster data, a compression algorithm is applied to reduce unnecessary pixel information, and for the above vector data, at least one of redundancy removal between vertices, error correction, mesh simplification, and attribute information filtering is performed for preprocessing,
The above data conversion unit,
The preprocessed large-capacity data is made lighter by applying at least one of the octree technique, LoD technique, and Block Reference technique.
A large-capacity data lightweight device. In the first paragraph,
The above data collection unit,
A large-capacity data lightweight device that analyzes the attribute information and quality of the above large-capacity data, classifies the categories, and stores them in a database. delete In the first paragraph,
The above data preprocessing unit,
A large-capacity data lightweighting device that performs at least one of coordinate system transformation, resolution adjustment, interpolation, or noise removal preprocessing on the above raster data. delete In paragraph 4,
The above data preprocessing unit,
A large-capacity data lightweight device that inputs the above preprocessed raster data into a pre-built general model to derive basic data. In paragraph 6,
The above data conversion unit,
A large-capacity data lightweight device that converts the above basic data into vector format data by applying an interpolation algorithm. In paragraph 7,
The above data conversion unit,
A large-capacity data lightweight device that applies the LoD technique to the above vector format data to divide it into multiple resolutions. In the first paragraph,
The above data conversion unit,
A large-capacity data lightweighting device that applies the LoD technique to the above preprocessed vector data to classify it into multiple levels of complexity. In the second paragraph,
The above data conversion unit,
A large-capacity data lightweight device that applies a spatial partitioning technique or an instancing technique to the above-mentioned preprocessed large-capacity data and stores it in the database. In a method for reducing large-capacity data by a large-capacity data reduction device,
A step of collecting large amounts of data from multiple servers;
A step of preprocessing the collected large volume of data; and
A step of reducing the weight of the preprocessed large-capacity data based on an artificial intelligence model;
Including,
The above preprocessing step is,
Preprocessing corresponding to the data format of the above large data including raster data and vector data is performed,
The above preprocessing step is,
For the above raster data, a compression algorithm is applied to reduce unnecessary pixel information, and for the above vector data, at least one of redundancy removal between vertices, error correction, mesh simplification, and attribute information filtering is performed for preprocessing,
The above lightweighting step is:
The preprocessed large-capacity data is made lighter by applying at least one of the octree technique, LoD technique, and Block Reference technique.
How to make large amounts of data lighter. A computer-readable recording medium having recorded thereon a program for executing the method of Article 11 on a computer.