CN120812053A - Brain inspired data service system and method thereof - Google Patents

Abstract

The present invention relates to a brain-inspired data service system and method, which is characterized by constructing a layered architecture: comprising a hybrid deployed data node cluster, specifically composed of core nodes, acquisition nodes and gateway nodes, which process core business data, real-time edge data and cross-domain connections respectively; the small-world connection layer realizes efficient communication in the same domain through high-cluster local connections, and breaks down data silos by using dynamic long-range cross-domain connections; the intelligent control layer integrates the DMN control center, a collaborative status monitoring engine, a predictive analysis module and a reinforcement learning-driven resource scheduler. The system initializes the topology structure through the small-world network manager, and dynamically inserts long-range connections when the frequency of cross-domain node communication exceeds a threshold, ensuring that the number of cross-domain path hops is stable and not higher than a preset value. The DMN control center implements dual-mode switching based on the system load threshold: data pre-cleaning, knowledge graph construction, predictive pre-fetching and resource pre-allocation are performed during the idle period; and routing is optimized in conjunction with the graph traversal algorithm during the task period.

Description

Brain inspired data service system and method thereof

Technical Field

The application relates to the technical field of distributed computing and artificial intelligence, in particular to a brain inspired data service system and a brain inspired data service method.

Background

With the rapid development of big data, cloud computing, internet of things and artificial intelligence technology, modern data service systems face dual challenges of architecture complexity and service intelligence. In a large enterprise or organization environment, data is often stored in a decentralized manner in multiple independent department systems, creating a severe data islanding phenomenon. The traditional centralized architecture is simple to manage, but as the data scale expands, the central node is very easy to become a performance bottleneck and a single point of failure source, and is difficult to deal with the cross-department data collaborative analysis requirement. The existing distributed system relieves the storage pressure through node expansion, but has obvious defects in the aspects of data discovery efficiency, service path optimization, resource dynamic scheduling and the like.

Currently, a mainstream data service system generally adopts a passive response mode, and corresponding operation is only executed when a user request is received, so that active integration and preprocessing capability of data resources are lacked. The mechanism causes three outstanding problems that firstly, cross-domain data access needs to be subjected to multi-layer transfer, communication delay is obviously increased, secondly, system resource allocation lacks predictability and cannot be dynamically adjusted according to service fluctuation, and thirdly, a large amount of computing resources are in an idle state in off-peak time, so that serious resource waste is caused. More importantly, the existing system lacks autonomous learning and self-adaption capability similar to a biological neural network, and is difficult to establish an intelligent data association network and a predictive service mechanism.

Disclosure of Invention

The application aims to provide a brain inspired data service system and a brain inspired data service method, which have the advantages of reducing cross-domain data access delay, realizing dynamic scheduling of resources, actively integrating preprocessing data resources and improving system response efficiency and resource utilization rate.

In order to solve the technical problems, the invention solves the technical problems by the following technical scheme that the data service system inspired by brain comprises:

The data node cluster is composed of a plurality of distributed data nodes and is used for storing, processing and analyzing data or providing data service, wherein the data nodes are at least one of servers, containers, micro-services, logic units or AIAgent, and comprise core nodes deployed in the cloud server cluster, edge nodes deployed in edge computing equipment and gateway nodes deployed in a containerized platform, and correspond to service domain core data processing, edge data acquisition and cross-domain data connection requirements respectively;

The small world connection layer is used for constructing connection topology among data nodes by adopting a small world network model and comprises a small world network manager, local high-clustering connection and long-range cross-domain connection, wherein the local high-clustering connection realizes dense interconnection of the data nodes in the same service domain, and the long-range cross-domain connection dynamically maintains a cross-domain shortcut through the small world network manager;

The intelligent control layer comprises a DMN control center and subordinate state monitoring engine, a prediction analysis module and a resource scheduler, which are used as a system control center to simulate a brain default mode network function;

the infrastructure layer is a hybrid architecture deployed on the cloud server cluster, the containerized platform, the edge computing equipment and the physical server;

The DMN control center automatically triggers mode switching through a threshold comparison algorithm based on real-time data of the state monitoring engine and pushes hot zone weights to a small-world network manager.

By adopting the technical scheme, the resource island and efficiency bottleneck problems of the traditional system are solved by constructing the data node cluster comprising the cloud core, the edge nodes and the containerized gateway and introducing the brain inspired small-world connection layer and the DMN intelligent control layer, the technical effects are that the accurate adaptation of resources according to scenes such as core processing, edge acquisition and cross-domain connection is realized, the data transmission efficiency is obviously improved by utilizing the topological characteristics of the small-world, namely the local efficient aggregation and the global short path accessibility, and the self-adaption and the cross-domain service capability of the system are enhanced by the global perception and the intelligent regulation such as hot zone weight push of the DMN control center, so that the foundation is laid for constructing the efficient, intelligent and robust data service system.

The invention is further arranged that the small world network manager of the small world connection layer realizes:

initializing network topology based on WattsStrogatz model variant algorithm, introducing long-range cross-domain connection through random reconnection edges;

And analyzing the node communication log according to a preset period, and dynamically inserting long-range connection through the service grid when the communication frequency among the cross-domain nodes exceeds a threshold value, so as to reduce the hop count of the communication path.

By adopting the technical proposal, the construction and dynamic adjustment mechanism of the small world network manager is specifically limited, the long-range connection is dynamically inserted according to the communication frequency threshold based on the initialization of the Watts-Strogatz variety, the technical effect is that the network is ensured to have high clustering initially, the topology structure can be intelligently optimized according to the actual cross-domain communication requirement, the direct 'shortcut' is dynamically established without sense through the service grid technology, so that the number of the original lengthy communication path hops is effectively reduced, the delay of cross-service domain data access is remarkably reduced, the response capability of the system to a dynamic data access mode is improved, and the problem of low cross-domain efficiency caused by data island is solved.

The invention is further arranged that the DMN control center of the intelligent control layer works cooperatively through the following modules:

the state monitoring engine is used for collecting CPU load, network throughput and container resource utilization index of the data node in real time and generating a system thermodynamic diagram;

The prediction analysis module is configured with a prediction engine, predicts a real-time data acquisition peak value of an edge node by utilizing an LSTM model, and analyzes a cross-domain query mode by utilizing a transducer model;

And the resource scheduler dynamically adjusts resource quota of the core node and the edge node based on the reinforcement learning algorithm, and when the data acquisition amount of the edge node is suddenly increased, automatically migrates the computing resource which is dynamically expanded according to 120% of the current load of the edge node to the edge node based on the CPU core number/memory quota of the containerized platform.

By adopting the technical scheme, the cooperative working modes of the intelligent control layer internal module, the state monitoring engine, the prediction analysis module and the resource scheduler and the specific functions thereof are described in detail, including the steps of collecting multidimensional indexes, predicting by utilizing LSTM/Transformer and dynamically scheduling resources based on reinforcement learning, the technical effect is that the thermodynamic diagram of the system state including the refinement, real-time monitoring and visualization of CPU, network and container resources is realized, and an advanced prediction model such as LSTM is utilized to capture short-term edge peaks, a transducer analyzes a long-term cross-domain mode, the change of demand is observed in advance, the intelligent and elastic migration and the capacity expansion as required of resources between a core and edge nodes are realized through a resource scheduler driven by reinforcement learning, the sudden load is effectively treated, and the utilization rate of the resources and the overall stability of the system are remarkably improved.

The invention is further configured such that the DMN control module supports dynamic switching between idle mode and task mode:

When entering an idle mode, triggering the following operations when the system load, the CPU utilization and the network I/O continuous preset time are lower than a threshold value:

Invoking a prediction engine to analyze historical data access logs and node communication frequency and generating a future data demand prediction result;

executing data cleaning, de-duplication and index optimization background tasks;

pushing the predicted hot zone information to a small world connection layer for pre-adjusting the high clustering connection weight of the local router;

when entering a task mode, a route optimization module is started, and an optimal task execution path among data service nodes is calculated through a graph traversal algorithm by combining the real-time load state of a cross-domain scheduler in a small world connection layer.

By adopting the technical scheme, the method has the technical effects that the DMN control center is limited to trigger the switching trigger conditions and respective core operations in a task mode when the system load, the CPU utilization rate and the network I/O continuous preset time are lower than the threshold value in the idle mode, the prediction engine is called in the idle mode to analyze history logs to generate future demand predictions, execute background tasks such as data cleaning index optimization and the like, push hot zone information to a connecting layer to be used for pre-adjusting local router weights, and the route optimization module is started in the task mode to calculate the optimal path through a graph traversal algorithm in combination with the real-time load.

The invention is further arranged such that the DMN control hub triggers the following background tasks in idle mode:

Performing data cleaning, de-duplication and normalization;

performing index and metadata optimization;

Integrating cross-domain data association relations based on the knowledge graph to form a high-level knowledge representation to support complex query;

performing predictive data prefetching, and caching high-frequency access data to the hot zone nodes;

And the DMN control center prefetches data which is likely to be accessed in the future to the adjacent data nodes according to the output result of the prediction analysis module in an idle mode, and pre-distributes computing resources.

By adopting the technical scheme, the specific background tasks executed by the DMN in the idle mode are further refined, including data cleaning, weighing and standardization processing, index and metadata optimization, forming a high-level knowledge representation based on a knowledge graph integration cross-domain data association relationship so as to support complex query, performing predictive data prefetching, caching high-frequency access data to a hot area node, and prefetching data which is likely to be accessed in the future to a neighboring node and pre-distributing computing resources according to output of the predictive analysis module.

The invention is further arranged that the prediction engine adopts a dual-model fusion algorithm:

Short-term prediction, when the short-term prediction is smaller than a preset time, capturing a burst access rule based on time sequence data by adopting an LSTM model;

and when the long-term prediction is longer than the preset time, adopting a transducer model, and excavating the clustering data association requirement by combining the topological relation of the small-world connection layer.

By adopting the technical scheme, the prediction engine is determined to capture burst access rules based on time sequence data by adopting a double-model fusion algorithm, namely an LSTM model when short-term prediction is smaller than preset time, and the clustering data association requirement is mined by adopting a Transformer model in combination with the topological relation of a small world connection layer when long-term prediction is larger than preset time.

The invention is further arranged that the small world connection layer comprises a local router and a cross-domain scheduler, and is used for respectively processing the same clustering and cross-domain requests;

the local router processes local data requests of data service nodes in the same cluster based on the high cluster connection relation;

The cross-domain scheduler forwards the data request of the cross-cluster through the long-range connection, when the topology update is triggered, the long-range connection topology of the cross-domain scheduler is preferentially adjusted, and the stable and no higher than the preset hop count of the cross-domain request path is ensured;

wherein the topology update is triggered by the following events:

The DMN control module outputs data access hot zone change under a prediction mode;

the continuous failure times of the long-range connection of the cross-domain scheduler are larger than or equal to the preset times;

the high clustering connection load rate of the local router is continuously larger than a preset proportion within a preset time.

By adopting the technical scheme, the internal component local router of the small world connection layer is defined to process the same clustering request based on high clustering connection, the cross-domain scheduler is used for preferentially adjusting the cross-domain scheduler topology through long-range connection to process the cross-clustering request and an optimization mechanism thereof, so that the cross-domain request path hop count is ensured to be not higher than a preset hop count, specific events triggering topology updating, such as hot zone change of DMN prediction output, the continuous failure times of long-range connection of the cross-domain scheduler are more than preset times, and the high clustering connection load rate of the local router is continuously over-limited.

A brain inspired data service system method comprising the steps of:

The hierarchical initialization step is that container data nodes are deployed in an infrastructure layer, an initial topology is generated through a small-world network manager of a small-world connection layer, and a state monitoring engine of an intelligent control layer starts full-link monitoring;

When the external access layer receives a cross-department query request, the DMN control center of the intelligent control layer calls a pre-calculation result of the prediction analysis module, and the real-time equipment data is synchronously provided by the edge node through long-range cross-domain connection of the small world connection layer to directly sell financial gateway nodes;

And the layering self-adaptive optimization step, namely triggering the small world connection layer to adjust the long-range connection distribution according to the state monitoring data by the intelligent control layer, and guiding the infrastructure layer to reallocate the container resources.

By adopting the technical scheme, the claim describes a layering method applying the system, which comprises three core steps of deploying containerized nodes in a layering initialization step, generating initial topology starting monitoring, calling a pre-calculation result by an intelligent control layer by a cross-layer cooperative processing step when an access layer receives a cross-department request, directly connecting a target node through a long distance, such as selling financial gateway nodes, integrating edge real-time data, triggering a connection layer according to the monitoring data by the intelligent control layer in a layering self-adaptive optimization step, adjusting topology, such as long-distance connection distribution, and guiding an infrastructure layer to redistribute container resources.

The hierarchical self-adaptive optimization method is further characterized by comprising a small world connection layer, an intelligent control layer, an infrastructure layer, a resource scheduler, a storage platform and a storage platform, wherein when the continuous preset number of days of communication between a production data node and a research and development data node exceeds a system average value, the small world connection layer is used for newly increasing long-range connection between the production data node and the research and development data node through a small world network manager and shortening the hop count of a communication path;

By adopting the technical scheme, an operation example such as a small world connection layer in a hierarchical self-adaptive optimization step is embodied, when communication between production and research nodes continuously exceeds a system average value by a preset number of days, the number of hops is greatly shortened through a manager newly increasing to reach a long-range connection, such as reducing from 5 hops to 1 hop, an intelligent control layer prediction analysis module pre-distributes a preset ratio such as 50% of database connection resources for a month financial settlement task in advance based on historical data, and an infrastructure layer resource scheduler automatically starts a new container example to perform load balancing when detecting that the CPU utilization of an edge node exceeds the preset percentage.

The intelligent control layer further comprises a learning and adapting engine for analyzing historical access modes and updating an access frequency model, evaluating task execution efficiency and optimizing a resource scheduling strategy, adjusting LSTM/transducer model parameters according to real-time feedback, and synchronously updating a knowledge base to reflect the latest data association relation.

The learning and adaptation engine updates the access mode model according to the task execution log, and optimizes the long-range connection distribution of the small world connection layer;

When the knowledge association relation in the knowledge base changes, index reconstruction and metadata updating of the data nodes are triggered, and a pre-calculation query strategy is synchronously adjusted.

By adopting the technical scheme, the function of a learning and adapting engine in the intelligent control layer is emphasized for analyzing a historical access mode and updating a frequency model, evaluating task execution efficiency and optimizing a resource scheduling strategy, adjusting LSTM or a transducer model parameter according to real-time feedback, synchronously updating a knowledge base to reflect the latest data association relation, and correlating the influence of the learning and adapting engine on a connection layer and a data node, namely, the learning engine optimizes the small world long-range connection distribution according to the task log updating access mode model, and when the knowledge base association relation changes, the learning and adapting engine triggers index reconstruction and metadata updating of the data node and synchronously adjusts a pre-calculated query strategy.

The invention has obvious technical effects due to the adoption of the technical scheme.

Drawings

FIG. 1 is a diagram of the overall architecture of the system;

FIG. 2 is a small world network connection topology;

FIG. 3 is a flowchart of DMN internal operation.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples.

Embodiment one:

In the prior art, with the wide application of big data and cloud computing technology, an enterprise data service system faces the problems of difficult integration of cross-department data, low resource utilization rate, response delay and the like, a traditional architecture relies on a centralized node to process requests, so that a core node is overloaded, edge data cannot be processed in time, the existing system cannot effectively utilize resources to preprocess in an idle period, cross-domain inquiry needs to skip for a plurality of times, communication delay is increased, for example, production data and financial data of a certain manufacturing enterprise belong to different server clusters, and a manual coordination data interface is needed when a comprehensive report is generated, so that response time exceeds a service tolerance threshold.

The inventor observes that the biological nervous system has the characteristics of self-adaptive connection and idle period self-optimization, proposes a working mechanism for simulating a brain default mode network, firstly analyzes the space-time distribution characteristics of cross-domain data access, discovers the coexistence of local high-frequency communication and cross-domain low-frequency access, secondly researches the high clustering and short-path characteristics of the small-world network, tries to apply the high clustering and short-path characteristics to the data node connection topology, further designs a hierarchical control architecture, performs predictive resource pre-allocation in the idle period, reduces the response delay when the task is triggered, and realizes the direct route of the cross-domain request by forming a dynamic connection network by the edge node and the core node.

The application provides a brain inspired data service system, which comprises a data node cluster, a small world connection layer, an intelligent control layer, an external access layer and an infrastructure layer, wherein the data node cluster is composed of a server, a container, a micro service and the like which are distributed, the core node, the edge node and the gateway node are covered, the small world connection layer adopts a topological structure combining high clustering and long-range connection, the intelligent control layer comprises a state monitoring, predictive analysis and resource scheduling module, the external access layer provides a standardized interface, and the infrastructure layer adopts a hybrid architecture for deployment.

The data node cluster refers to distributed computing units divided according to service requirements, rapid deployment can be realized by adopting a containerization technology, the data node cluster is used for processing the data service requirements of different service domains, the small world connection layer refers to a connection topology construction module based on a complex network theory, the communication efficiency in the service domains is maintained through local dense connection, the cross-domain direct connection is realized through random long-range connection, the intelligent control layer refers to a central control system simulating a brain default mode, the system optimization is driven through real-time monitoring and predictive analysis, the infrastructure layer refers to the physical and virtual resource combination of the operation of a support system, for example, core nodes are deployed on a cloud server, and edge nodes are deployed on industrial gateway equipment.

When the system operates, the mixed architecture of the infrastructure layer provides elastic resource support, when the external access layer receives a user request, the small world connection layer selects local routing or cross-domain scheduling according to the request type, the intelligent control layer continuously monitors the node state, data preprocessing and resource pre-allocation are performed in an idle period, for example, when frequent interaction between a production node and a quality node is detected, a long-range connection channel is automatically established, when a month-end financial settlement task is triggered, the pre-allocated database connection resource can be immediately put into use, and resource disputes are avoided.

The scheme realizes decoupling of control and execution through a layered architecture, avoids a bottleneck of a single control node, reduces the number of cross-domain hops while maintaining local efficiency by a small-world connection topology, reduces redundant links compared with a traditional full-connection mode, and converts traditional post-hoc response into pre-hoc preparation by a predictive resource scheduling mechanism of an intelligent control layer, thereby effectively utilizing idle resources of a system.

The application realizes the path optimization of cross-domain data service, shortens the average response time of cross-department query to the service acceptable range, improves the processing efficiency of burst tasks by a dynamic resource pre-allocation mechanism, improves the utilization rate of edge node resources, automatically executes the data optimization task in an idle period, avoids idle waste of computing resources and forms continuous self-optimized service capability.

The small world network manager of the small world connection layer initializes the network topology based on WattsStrogatz model variant algorithm, introduces long-range cross-domain connection through random reconnection edges, analyzes node communication logs according to a preset period, dynamically inserts the long-range connection through a service grid when the communication frequency among the cross-domain nodes exceeds a threshold value, and is used for reducing the hop count of a communication path.

The WattsStrogatz model variant algorithm refers to a topology generation algorithm based on small world network theory improvement, specifically, a random reconnection edge mechanism can be adopted to introduce long-range connection in an initial high clustering network, local connection density and global communication efficiency are balanced by adjusting reconnection probability parameters, random reconnection edge refers to probability replacement of part of connection in the initial ring topology, for example, local connection is replaced by cross-domain connection with preset probability, so that a network structure with high clustering and low path length is formed, service grid dynamic insertion long-range connection refers to real-time adjustment of communication links among nodes through a service grid technology, for example, when the cross-domain communication frequency is detected to exceed a set threshold value, a direct connection channel is automatically established among corresponding nodes.

In the network initialization stage, an improved WattsStrogatz algorithm is adopted to construct a basic topology, a random reconnection mechanism is adopted to introduce cross-domain connection while the local high clustering characteristic is reserved, communication logs of cross-domain nodes are periodically analyzed, for example, interaction frequencies among the nodes are counted in units of hours or days, when the communication frequency of a specific node pair exceeds a preset threshold, a dynamic link configuration function of a service grid is triggered, at the moment, a system automatically establishes long-range connection among corresponding nodes and bypasses an original intermediate node to form a direct path, so that the number of the cross-domain communication hops is reduced from multiple hops in the original topology to single hops.

The traditional system generally adopts a fixed topology or a simple distributed architecture, the cross-domain communication path depends on a statically configured intermediate node and cannot be dynamically optimized according to actual communication requirements, and the scheme can automatically identify a high-frequency cross-domain interaction scene and optimize a connection topology through a periodic communication analysis and service grid dynamic adjustment mechanism, so that the problems of high cross-domain communication delay and path redundancy in the traditional architecture are effectively solved.

According to the application, the cross-domain communication path can be automatically optimized according to the actual service demand, the data transmission delay in the high-frequency cross-domain interaction scene is obviously reduced, the intermediate forwarding link is reduced by dynamically inserting the long-range connection, the waste of bandwidth resources caused by path redundancy is avoided, and the response efficiency of the cross-domain data service is improved.

The DMN control center of the intelligent control layer cooperatively works through the following modules that a state monitoring engine collects CPU load, network throughput and container resource utilization index of a data node in real time and generates a system thermodynamic diagram, a prediction analysis module is configured with a prediction engine, an LSTM model is utilized to predict real-time data collection peak values of an edge node, a trans-former model is utilized to analyze a cross-domain query mode, a resource scheduler dynamically adjusts resource quota of a core node and the edge node based on a reinforcement learning algorithm, and when the data collection amount of the edge node is suddenly increased, the resource scheduler automatically migrates calculation resources which are dynamically expanded according to 120% of the current load of the edge node to the edge node based on the CPU core number or memory quota of a containerized platform.

The state monitoring engine is a module for continuously collecting system operation indexes, and can be realized by adopting a Prometheus monitoring framework and combining with a custom index collector, resource consumption data of each node is periodically acquired through an exposed API (application program interface), the LSTM model in the prediction analysis module is a long-period memory neural network, and can be realized by adopting a TensorFlow framework to construct a three-layer network structure for processing periodic fluctuation characteristics in time sequence data, the transducer model is a deep learning framework based on a self-attention mechanism, and can be realized by adopting a pre-training model in a HuggingFace library for fine tuning, and is used for capturing an association mode in cross-domain query, the reinforcement learning algorithm in a resource scheduler is a dynamic decision method based on environmental feedback, and can be realized by adopting a Q-learning algorithm and combining with an API interface of a container arrangement platform, and resource allocation strategies are optimized through a reward function.

The state monitoring engine periodically collects operation indexes of all nodes and generates a visual thermodynamic diagram, the color depth of the thermodynamic diagram reflects the resource consumption intensity of different areas, the prediction analysis module calls an LSTM model to predict data collection peaks within five minutes in the future according to the historical load curve of edge nodes in the thermodynamic diagram, meanwhile, semantic association in a cross-department query request is analyzed through a transducer model, when burst flow occurs to the edge nodes, the resource scheduler calculates an optimal capacity expansion scheme according to a reinforcement learning strategy, for example, the CPU core number of a certain edge node is increased from 4 cores to 5 cores in a containerization platform, the memory quota is increased from 8GB to 9.6GB, and the capacity expansion range is accurately matched with 120% of the current load.

The traditional system generally adopts a static threshold alarm mechanism when the resources are scheduled, a sudden flow scene cannot be dealt with, the scheme can prejudge the change trend of the resource demand in advance by fusing a time sequence prediction and semantic analysis dual-mode framework, and can realize accurate elastic expansion and contraction operation by combining a reinforcement learning algorithm, so that the problem of response delay caused by manual intervention is avoided.

The DMN control module supports dynamic switching between an idle mode and a task mode, when the system load, the CPU utilization rate and the network I/O continuous preset time are lower than a threshold value, the DMN control module triggers the following operations of calling a prediction engine to analyze historical data access logs and node communication frequency to generate a future data demand prediction result, executing data cleaning, de-duplication and index optimization background tasks, pushing prediction hot zone information to a small world connection layer for pre-adjusting high clustering connection weights of local routers, and when the DMN control module enters the task mode, starting a route optimization module, combining the real-time load state of a cross-domain scheduler in the small world connection layer, and calculating an optimal task execution path among data service nodes through a graph traversal algorithm.

The idle mode refers to a low-power-consumption running state triggered automatically when the utilization rate of system resources is lower than a preset threshold value, and the idle mode can be realized by adopting a load monitoring algorithm based on a sliding window and used for executing background optimization tasks in off-peak time, the task mode refers to a high-performance running state activated when the system responds to real-time requests and is realized by a dynamic path planning algorithm, for example, an improved version based on Dijkstra algorithm calculates an optimal task path, the prediction engine refers to a data access prediction module based on a machine learning model and can be realized by adopting an LSTM neural network and a Transformer model fusion architecture and used for identifying periodic access rules and cross-domain association requirements, the data prefetching refers to a caching mechanism for loading high-frequency access data to a target node in advance according to a prediction result and is realized by adopting a distributed cache management protocol, for example, an LRU cache replacement strategy and a prefetching queue are used for cooperation.

When the system detects that the CPU utilization rate and the network flow index are continuously lower than a set threshold value for reaching a preset duration, the system is automatically switched to an idle mode, at the moment, a prediction engine performs time sequence analysis on a historical access log, a data object set which is likely to be accessed by high frequency in the future is identified, a data cleaning task marks and merges redundant data in a storage node, an index optimization module reconstructs a B+ tree index structure to improve query efficiency, prediction hot zone information is pushed to a local router through a message queue, the connection weight distribution strategy is adjusted in advance, when an external request arrives, the system is immediately switched to a task mode, a route optimization module traverses a topological graph by adopting a breadth-first search algorithm according to the load state of a current cross-domain scheduler, and a task execution path with the minimum hop number and balanced load is screened.

The traditional system only maintains a basic running state in an idle period, idle resources are not effectively utilized to execute data optimization tasks, resource waste and response delay are caused, a mode switching mechanism in the prior art is dependent on a manual configuration strategy, the running state cannot be dynamically adjusted according to a real-time load, and the scheme actively completes data preprocessing and resource pre-allocation in the idle period of the system through automatic threshold monitoring and prediction driving background task execution, so that service response time in a task mode is remarkably reduced.

The application realizes the dual promotion of the utilization rate of system resources and the response efficiency, the index optimization executed in the idle mode improves the cross-domain query speed, the data prefetching mechanism reduces the acquisition delay of high-frequency access data, the dynamic path planning in the task mode effectively avoids network congestion, ensures the stable and controllable end-to-end processing time of cross-department data requests, the system can autonomously switch the operation mode according to the real-time load state, and reduces the energy consumption and the idle computing resources while guaranteeing the service quality.

The method comprises the steps of triggering the following background tasks in an idle mode, performing data cleaning, de-duplication and standardization processing, performing index and metadata optimization, integrating cross-domain data association relations based on a knowledge graph to form a high-level knowledge representation to support complex inquiry, performing predictive data prefetching, caching high-frequency access data to a hot zone node, prefetching data which is likely to be accessed in the future to a neighboring data node according to the output result of a predictive analysis module, and pre-distributing computing resources.

The data cleaning is to automatically identify and correct error or redundant information in data through preset rules, and can be realized by combining regular expression matching with a machine learning model, so that the data quality and consistency are improved, the index and metadata optimization is to reconstruct a database index structure, can be realized by a B+ tree balance algorithm or a column storage optimization technology, and is used for accelerating the query response speed, the knowledge graph integration cross-domain data association relationship is to construct an entity relationship network through a graph database, can be realized by Neo4j or JanusGraph, and is used for finding potential data association of a cross-service domain, the predictive data pre-fetching is to load data in advance based on an access mode prediction result, and can be realized by combining an LRU cache replacement algorithm with a time sequence prediction model, so that the subsequent query delay is reduced.

When the system enters an idle mode, a background task execution flow is automatically started, a data cleaning module traverses a storage node, an abnormal data format is identified through a rule engine and standardized, an index optimization engine analyzes a query log, an index structure of a high-frequency access field is reconstructed, a knowledge graph engine synchronously updates a cross-domain entity relation, a new semantic association path is established, a prediction analysis module generates a pre-fetching instruction in combination with a history access mode, a predicted data copy is distributed to a cache area of a target node, and computing resource pre-allocation among adjacent nodes is realized by dynamically adjusting resource quota through a container arrangement platform, so that the pre-fetching data has instant processing capability.

The traditional system only maintains a basic running state in an idle period, the computing resources are not effectively utilized for data optimization, the data preprocessing in the prior art needs manual triggering and lacks cross-domain association analysis, and a global optimization strategy cannot be formed.

The application realizes the full utilization of the computing resources in the idle period of the system, effectively improves the data query efficiency and accuracy, enhances the response capability of complex query by the continuous maintenance of the cross-domain data association relationship, obviously reduces the service delay in the high concurrency scene by the predictive data prefetching mechanism, ensures the availability of the data prefetching by the preassigned computing resource strategy, and forms a complete offline optimization and online service coordination mechanism.

The prediction engine adopts a dual-model fusion algorithm, short-term prediction adopts an LSTM model when the short-term prediction is smaller than preset time and captures burst access rules based on time sequence data, long-term prediction adopts a Transformer model when the short-term prediction is larger than preset time and combines the topological relation of a small world connection layer to mine cross-cluster data association requirements, wherein the LSTM model refers to a long-term memory neural network model, and particularly can be realized by adopting a cyclic neural network with a forgetting gate, an input gate and an output gate structure, and is used for capturing short-term dependency relations and burst access modes in time sequence data, the Transformer model refers to a deep learning model based on a self-attention mechanism, particularly can be realized by adopting an encoder structure consisting of a multi-head attention layer and a feedforward neural network layer, and is used for processing long-distance dependency relations and cross-cluster data association requirements, the dual-model fusion algorithm refers to a technical scheme for allocating prediction tasks of different time scales to different model processing, particularly can be realized by dividing prediction task types through preset time thresholds and establishing a model switching mechanism, and is used for considering short-term association rules.

When the prediction time range is below a preset time threshold, the system automatically calls an LSTM model to process time sequence data, the model processes time sequence characteristics layer by layer through a cyclic neural network structure, a memory unit is utilized to capture short-term fluctuation rules of indexes such as data acquisition quantity of a device sensor, query frequency of a user and the like, when the prediction time range exceeds the preset time threshold, the system is switched to a transducer model, topological connection relations across clustering nodes are analyzed through a self-attention mechanism, potential data association requirements among different service domains are identified, and prediction results of the two models are generated to be finally output through a weighted fusion module and used for guiding resource pre-allocation and data pre-fetching strategies.

The traditional prediction method generally adopts a single model to process prediction tasks of different time scales, so that short-term prediction ignores cross-domain association characteristics, and long-term prediction is difficult to capture sudden fluctuation.

According to the method, the optimal model can be automatically selected according to the time characteristics of the prediction task, the utilization rate of the edge node resources and the dispatching efficiency of the cross-domain data are improved, the short-term prediction module accurately captures the real-time data acquisition peak value, the resource deficiency caused by sudden load is avoided, the long-term prediction module analyzes and optimizes the pre-allocation of the cross-cluster resources through the topological relation, and the cross-domain query delay is reduced.

The small world connection layer comprises a local router and a cross-domain scheduler, wherein the local router and the cross-domain scheduler respectively process the same cluster and cross-domain requests, the local router processes local data requests of data service nodes in the same cluster based on a high cluster connection relation, the cross-domain scheduler forwards the cross-cluster data requests through long-range connection, when topology update is triggered, the long-range connection topology of the cross-domain scheduler is preferentially adjusted, the stable number of hops of a cross-domain request path is ensured to be not higher than a preset number of hops, the topology update is triggered by the following events that a DMN control module outputs data access hot area changes in a prediction mode, the continuous failure number of the long-range connection of the cross-domain scheduler is larger than or equal to the preset number, and the high cluster connection load rate of the local router is continuously larger than a preset proportion in the preset time.

The local router is a communication control unit deployed in a service domain, and can be specifically realized by adopting routing equipment based on an OSPF protocol, low-delay communication of nodes in the domain is realized by maintaining a high-clustering connection relation, the low-delay communication is used for reducing transmission delay of data interaction with the service domain, the cross-domain scheduler is a relay device for cross-service domain communication, can be specifically realized by adopting a service grid component expanded by a BGP protocol, shortens a cross-domain communication path by dynamically maintaining long-range connection and is used for solving the problem of low cross-domain access efficiency caused by data island, the topology update triggering event is a monitoring mechanism of network state change, and can be specifically used for counting connection failure times by adopting a sliding window algorithm, detecting load rate abnormality through time sequence analysis and realizing self-adaptive adjustment of a network structure so as to maintain service quality.

The local router forms an intra-domain communication backbone network by maintaining a high clustering connection relation, when a co-clustering request is received, the intra-domain communication backbone network is directly transmitted through a pre-established physical link, the cross-domain scheduler continuously monitors a long-range connection state, when the fact that the number of hops of the cross-domain communication path exceeds a threshold value is detected, topology adjustment operation of the cross-domain scheduler is preferentially triggered, for example, when a DMN control module predicts sales data hot zone migration, the cross-domain scheduler automatically establishes new long-range connection between a sales domain and a financial domain, meanwhile, the low-efficiency redundant connection is removed, the local router and the cross-domain scheduler form a layered control architecture, and the intra-domain communication efficiency is maintained, and meanwhile, the cross-domain path selection is optimized.

The traditional data system adopts a static routing strategy to cause the reduction of the cross-domain communication efficiency along with the fluctuation of the service, the scheme realizes the continuous optimization of the cross-domain path through a dynamic topology updating mechanism, the cross-domain connection adjustment in the prior art usually needs manual intervention, the scheme realizes the automatic network reconstruction based on the preset triggering condition, and the sudden cross-domain access requirement is effectively met.

The method and the system can automatically identify the high-load communication path and implement targeted optimization, obviously reduce communication delay in a cross-domain data access scene, ensure the same-domain communication efficiency through a layering processing mechanism, dynamically adjust the cross-domain connection topology, effectively avoid network congestion, improve the response stability of cross-department data service, timely eliminate the influence of single-point faults on the overall performance of the system based on an automatic updating mechanism of preset triggering conditions, and ensure continuous and efficient operation of the data service system in a complex service scene.

A data service system method inspired by brain comprises a layering initialization step, a cross-layer cooperative processing step and a layering self-adaptive optimization step, wherein the layering initialization step deploys containerized data nodes in an infrastructure layer, an initial topology is generated through a small-world network manager of a small-world connection layer, a state monitoring engine of an intelligent control layer starts full-link monitoring, when an external access layer receives a cross-department query request, a DMN control center of the intelligent control layer calls a pre-calculation result of a predictive analysis module, long-range cross-domain connection of the small-world connection layer is used for directly selling financial gateway nodes, real-time equipment data is synchronously provided by edge nodes, and the layering self-adaptive optimization step triggers the small-world connection layer to adjust long-range connection distribution according to the state monitoring data by the intelligent control layer and guides the infrastructure layer to conduct container resource reallocation.

The hierarchical initialization step refers to a process of establishing a system infrastructure, specifically, a container arrangement tool can be adopted to deploy data nodes, for example, node instances at different positions are managed through a Kubernetes cluster, the step lays a foundation for dynamic resource scheduling, the cross-layer cooperative processing step refers to a mechanism of multi-level component cooperative response complex requests, specifically, cross-domain connection routing can be realized through a service grid technology, the mechanism effectively solves the problem of data access delay of a traditional system in a cross-department manner, the hierarchical self-adaptive optimization step refers to a continuous self-adjusting process of the system, specifically, a reinforcement learning algorithm can be adopted to dynamically adjust resource quota, and the process realizes double improvement of resource utilization rate and service quality.

In the system starting stage, the containerization platform can be configured to deploy a core node at a cloud end, deploy a lightweight container instance at an edge device, when a cross-domain query is received, the prediction analysis module can predict a target node based on a historical access mode, for example, a financial data request is directly routed to a preset gateway container instance, in the running process, a state monitoring engine can periodically collect network delay data, when the fact that the number of communication hops between specific nodes exceeds the standard is detected, the transverse expansion operation of the container instance is triggered, and the hierarchical collaboration mechanism enables a data processing path to always maintain an optimal state and simultaneously avoids system oscillation caused by single-level adjustment.

The traditional centralized system needs to forward requests layer by layer when querying across departments, the method realizes the cross-domain direct route through a pre-calculation path, intermediate links are reduced, a static resource allocation mode in the prior art cannot cope with sudden loads, the method realizes the elastic expansion and contraction of calculation resources through hierarchical self-adaptive optimization, each level of the conventional distributed system operates independently, and the method realizes integral collaborative optimization through the cross-level transmission of state monitoring data.

The application realizes the end-to-end direct processing of the cross-domain data request, effectively reduces the inquiry response delay, ensures the service stability of the edge node under the sudden load by the dynamic resource redistribution mechanism, avoids the interruption of data processing caused by insufficient resources, and ensures that the system can automatically adapt to the change of service demands by multi-level collaborative optimization, thereby improving the processing efficiency of complex inquiry tasks.

The specific implementation mode of the layered self-adaptive optimization step comprises a collaborative optimization mechanism of a small world connection layer, an intelligent control layer and an infrastructure layer, when the continuous preset number of days of communication between a production data node and a research and development data node exceeds a system average value, a small world network manager is used for newly increasing long-range connection between the production data node and the research and development data node and shortening the number of hops of a communication path, a prediction analysis module pre-distributes database connection resources with preset proportion for a month financial settlement task in advance based on historical data, and a resource scheduler automatically starts a plurality of new examples in a containerization platform to carry out load balancing when detecting that the CPU utilization rate of an edge node exceeds the preset percentage.

The hierarchical self-adaptive optimization refers to that the system solves the problem of cross-domain communication and resource allocation through different levels of cooperative adjustment, and particularly can be realized by adopting a dynamic adjustment connection topology and resource allocation algorithm, a small-world connection layer optimizes a communication path through newly-added cross-domain long-range connection, for example, when the communication frequency between nodes exceeds a threshold value, topology update is triggered, an intelligent control layer responds to periodic task demands through predictive resource pre-allocation, for example, database connection resources are allocated in advance based on a historical access mode, and an infrastructure layer realizes load balancing through dynamic expansion and contraction of containerized examples, for example, when the resource utilization rate of edge nodes reaches a preset threshold value, a new example is started automatically.

When the communication frequency between the production and research data nodes is continuously higher than the system average value, the small world network manager automatically establishes cross-domain long-range connection, shortens the communication path which is originally needed to be forwarded through a plurality of intermediate nodes to be a direct link, the intelligent control layer pre-distributes database connection pool resources before starting a monthly financial settlement task by analyzing historical task execution logs, so that resources in a peak period of the task are prevented from being stricken, a resource scheduler of an infrastructure layer monitors the edge node load in real time, and when the CPU utilization rate is detected to exceed a preset threshold value, a new instance is automatically started in a containerized platform, and part of calculation tasks are migrated to the newly added instance for execution.

In some embodiments, the preset number of days may be, for example, 30 days, the system average may be dynamically calculated by a sliding window algorithm, the database connection resources of the preset proportion may be, for example, 40% of the total connection pool, the preset percentage may be, for example, 80% of the CPU utilization threshold, and the number of new instances may be dynamically calculated according to the current load and the resource margin.

The traditional system generally adopts a fixed routing strategy when the cross-domain communication is optimized, the topological structure cannot be dynamically adjusted according to an actual communication mode, the scheme realizes the self-adaptive optimization of the cross-domain connection through a layered cooperative mechanism, the traditional resource allocation method mostly adopts static quota or passive response capacity expansion, the scheme realizes active resource supply through predictive pre-allocation and dynamic instance creation, and the traditional load balancing technology relies on a fixed threshold value to trigger capacity expansion, and realizes flexible resource scheduling by combining a real-time monitoring and containerization technology.

The application effectively reduces the path hop count and transmission delay of cross-domain communication, improves the resource supply efficiency of periodic tasks, enhances the dynamic load processing capacity under the edge computing scene, reduces the cross-department data interaction time delay by optimizing the communication path, avoids the resource bottleneck when the tasks are executed by predictive resource pre-allocation, and ensures the stability of the system under the high-load scene by the dynamic expansion and contraction of containerized examples.

The intelligent control layer also comprises a learning and adapting engine which is used for analyzing historical access modes and updating an access frequency model, evaluating task execution efficiency and optimizing a resource scheduling strategy, adjusting LSTM or a transducer model parameter according to real-time feedback, synchronously updating a knowledge base to reflect the latest data association relation, updating the access mode model according to a task execution log by the learning and adapting engine, optimizing long-range connection distribution of the small world connection layer, triggering index reconstruction and metadata updating of data nodes when the knowledge association relation in the knowledge base changes, and synchronously adjusting a pre-calculation query strategy.

The learning and adapting engine is a computing module with autonomous learning and dynamic adjusting capability, and can be realized by combining a machine learning framework with a rule engine, realizing policy optimization by continuously analyzing system operation data, wherein the access frequency model is a probability distribution model for describing a data access rule, can be constructed by using a hidden Markov model or a time sequence analysis algorithm for predicting a future data access hot spot area, the resource scheduling policy is a decision rule for dynamically distributing computing resources, can be realized by combining a reinforcement learning algorithm with a queuing theory model, and can be realized by adjusting resource quota according to a real-time load state, the knowledge base is a structured storage system for storing data association relationship, and can be used for maintaining the cross-domain data association relationship by using a knowledge graph technology.

The learning and adapting engine continuously collects task execution logs and system operation indexes, an access mode model is updated in a mode of combining offline training and online reasoning, when a significant change of access frequency of a specific data node is detected, a long-range connection optimization suggestion is automatically generated and pushed to a small-world connection layer, when a knowledge base update event is triggered, the system automatically starts an index reconstruction flow, the association weight among the data nodes is recalculated, an optimized query execution plan is generated based on a new association relation, an online learning mechanism is adopted in a model parameter adjustment process, real-time feedback data is combined with historical training data, and weight parameters of a prediction model are iteratively updated through a gradient descent algorithm.

The traditional system lacks a continuous learning mechanism, the resource allocation strategy cannot be dynamically optimized according to the operation data, the scheme realizes automatic optimization of system parameters and real-time synchronization of a knowledge base by introducing a learning and adapting engine, the resource scheduling strategy in the prior art is mostly static configuration, and is difficult to adapt to dynamic load change, and the scheme realizes dynamic optimization of resource allocation through a reinforcement learning algorithm.

The method can effectively solve the problems of low resource utilization rate and poor cross-domain query efficiency of the traditional data service system, continuously optimize system configuration through an autonomous learning mechanism, reduce the manual maintenance cost, eliminate the data island phenomenon by a real-time synchronization mechanism of a knowledge base, improve the response speed of cross-domain data association query, enhance the adaptability of the system to a burst access mode by a dynamic model parameter adjustment function, and ensure that the prediction accuracy is continuously improved along with the time.

Claims (10)

1. A brain-inspired data service system, comprising:

The data node cluster is composed of a plurality of distributed data nodes and is used for storing, processing and analyzing data or providing data service, wherein the data nodes are at least one of servers, containers, micro-services, logic units or AIAgent, and comprise core nodes deployed in the cloud server cluster, edge nodes deployed in edge computing equipment and gateway nodes deployed in a containerized platform, and correspond to service domain core data processing, edge data acquisition and cross-domain data connection requirements respectively;

The small world connection layer is used for constructing connection topology among data nodes by adopting a small world network model and comprises a small world network manager, local high-clustering connection and long-range cross-domain connection, wherein the local high-clustering connection realizes dense interconnection of the data nodes in the same service domain, and the long-range cross-domain connection dynamically maintains a cross-domain shortcut through the small world network manager;

The intelligent control layer comprises a DMN control center and subordinate state monitoring engine, a prediction analysis module and a resource scheduler, which are used as a system control center to simulate a brain default mode network function;

the infrastructure layer is a hybrid architecture deployed on the cloud server cluster, the containerized platform, the edge computing equipment and the physical server;

The DMN control center automatically triggers mode switching through a threshold comparison algorithm based on real-time data of the state monitoring engine and pushes hot zone weights to a small-world network manager.

2. The brain-inspired data service system of claim 1, wherein the small world network manager of the small world connection layer implements:

initializing network topology based on WattsStrogatz model variant algorithm, introducing long-range cross-domain connection through random reconnection edges;

And analyzing the node communication log according to a preset period, and dynamically inserting long-range connection through the service grid when the communication frequency among the cross-domain nodes exceeds a threshold value, so as to reduce the hop count of the communication path.

3. The brain-inspired data service system of claim 2, wherein the DMN control hub of the intelligent control layer works cooperatively by:

the state monitoring engine is used for collecting CPU load, network throughput and container resource utilization index of the data node in real time and generating a system thermodynamic diagram;

The prediction analysis module is configured with a prediction engine, predicts a real-time data acquisition peak value of an edge node by utilizing an LSTM model, and analyzes a cross-domain query mode by utilizing a transducer model;

And the resource scheduler dynamically adjusts resource quota of the core node and the edge node based on the reinforcement learning algorithm, and when the data acquisition amount of the edge node is suddenly increased, automatically migrates the computing resource which is dynamically expanded according to 120% of the current load of the edge node to the edge node based on the CPU core number/memory quota of the containerized platform.

4. The brain-inspired data service system of claim 3, wherein the DMN control module supports dynamic switching of idle mode and task mode:

When entering an idle mode, triggering the following operations when the system load, the CPU utilization and the network I/O continuous preset time are lower than a threshold value:

Invoking a prediction engine to analyze historical data access logs and node communication frequency and generating a future data demand prediction result;

executing data cleaning, de-duplication and index optimization background tasks;

pushing the predicted hot zone information to a small world connection layer for pre-adjusting the high clustering connection weight of the local router;

when entering a task mode, a route optimization module is started, and an optimal task execution path among data service nodes is calculated through a graph traversal algorithm by combining the real-time load state of a cross-domain scheduler in a small world connection layer.

5. The brain-inspired data service system of claim 4, wherein the DMN control hub triggers the following background tasks in idle mode:

Performing data cleaning, de-duplication and normalization;

performing index and metadata optimization;

Integrating cross-domain data association relations based on the knowledge graph to form a high-level knowledge representation to support complex query;

performing predictive data prefetching, and caching high-frequency access data to the hot zone nodes;

And the DMN control center prefetches data which is likely to be accessed in the future to the adjacent data nodes according to the output result of the prediction analysis module in an idle mode, and pre-distributes computing resources.

6. The brain-inspired data service system of claim 5, wherein the prediction engine employs a dual-model fusion algorithm:

Short-term prediction, when the short-term prediction is smaller than a preset time, capturing a burst access rule based on time sequence data by adopting an LSTM model;

and when the long-term prediction is longer than the preset time, adopting a transducer model, and excavating the clustering data association requirement by combining the topological relation of the small-world connection layer.

7. A brain-inspired data service system according to claim 3, wherein the small world connection layer comprises a local router and a cross-domain scheduler, which process co-clustering and cross-domain requests respectively;

the local router processes local data requests of data service nodes in the same cluster based on the high cluster connection relation;

The cross-domain scheduler forwards the data request of the cross-cluster through the long-range connection, when the topology update is triggered, the long-range connection topology of the cross-domain scheduler is preferentially adjusted, and the stable and no higher than the preset hop count of the cross-domain request path is ensured;

wherein the topology update is triggered by the following events:

The DMN control module outputs data access hot zone change under a prediction mode;

the continuous failure times of the long-range connection of the cross-domain scheduler are larger than or equal to the preset times;

the high clustering connection load rate of the local router is continuously larger than a preset proportion within a preset time.

8. A brain-inspired data service system method, applied with a brain-inspired data service system according to any one of claims 1-9, comprising the steps of:

The hierarchical initialization step is that container data nodes are deployed in an infrastructure layer, an initial topology is generated through a small-world network manager of a small-world connection layer, and a state monitoring engine of an intelligent control layer starts full-link monitoring;

When the external access layer receives a cross-department query request, the DMN control center of the intelligent control layer calls a pre-calculation result of the prediction analysis module, and the real-time equipment data is synchronously provided by the edge node through long-range cross-domain connection of the small world connection layer to directly sell financial gateway nodes;

And the layering self-adaptive optimization step, namely triggering the small world connection layer to adjust the long-range connection distribution according to the state monitoring data by the intelligent control layer, and guiding the infrastructure layer to reallocate the container resources.

9. The method of claim 8, wherein the hierarchical adaptive optimization comprises:

the small world connection layer is used for newly increasing long-range connection between the production data node and the research data node through the small world network manager and shortening the hop count of a communication path when the continuous preset number of days of communication between the production data node and the research data node exceeds a system average value;

the intelligent control layer is used for pre-distributing database connection resources with preset proportion for the monthly financial settlement task in advance based on historical data by the prediction analysis module;

and the infrastructure layer is that when the resource scheduler detects that the CPU utilization rate of the edge node exceeds the preset percentage, a plurality of new examples are automatically started in the containerized platform to carry out load balancing.

10. The method of claim 9, wherein the intelligent control layer further comprises a learning and adaptation engine for analyzing historical access patterns and updating access frequency models, evaluating task execution efficiency and optimizing resource scheduling policies, adjusting LSTM/transducer model parameters based on real-time feedback, synchronizing update knowledge bases to reflect up-to-date data associations;

The learning and adaptation engine updates the access mode model according to the task execution log, and optimizes the long-range connection distribution of the small world connection layer;

When the knowledge association relation in the knowledge base changes, index reconstruction and metadata updating of the data nodes are triggered, and a pre-calculation query strategy is synchronously adjusted.