CN119299332B - A safety monitoring system and monitoring method based on machine vision - Google Patents

A safety monitoring system and monitoring method based on machine vision Download PDF

Info

Publication number
CN119299332B
CN119299332B CN202411404721.9A CN202411404721A CN119299332B CN 119299332 B CN119299332 B CN 119299332B CN 202411404721 A CN202411404721 A CN 202411404721A CN 119299332 B CN119299332 B CN 119299332B
Authority
CN
China
Prior art keywords
space
time
data
carrying
machine vision
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202411404721.9A
Other languages
Chinese (zh)
Other versions
CN119299332A (en
Inventor
金涛
庄会富
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kunming Institute of Botany of CAS
Original Assignee
Kunming Institute of Botany of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kunming Institute of Botany of CAS filed Critical Kunming Institute of Botany of CAS
Priority to CN202411404721.9A priority Critical patent/CN119299332B/en
Publication of CN119299332A publication Critical patent/CN119299332A/en
Application granted granted Critical
Publication of CN119299332B publication Critical patent/CN119299332B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/18—Status alarms
    • G08B21/182—Level alarms, e.g. alarms responsive to variables exceeding a threshold

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Business, Economics & Management (AREA)
  • Emergency Management (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to the field of network security monitoring and image processing, in particular to a security monitoring system and a security monitoring method based on machine vision. The method comprises the steps of collecting visual image data of server hardware in real time, carrying out multi-frequency decomposition on the visual image data, carrying out filtering treatment and recombination on obtained signals in different frequency domains to generate a denoised reconstructed image, obtaining global feature representation based on a depth network, constructing a space-time diagram and carrying out behavior analysis to obtain comprehensive space-time feature representation, carrying out fusion treatment on the comprehensive space-time feature representation and sensor data, and carrying out risk assessment to determine an early warning strategy. The problems that false alarm or missing alarm is generated due to the fact that noise interference cannot be effectively separated and restrained, complex abnormal behaviors or dangerous operations are difficult to effectively identify, risk assessment is not comprehensive due to the fact that multi-mode fusion is not carried out on the complex abnormal behaviors or dangerous operations and the data of the sensor cannot be carried out, and response speed is delayed or response measures are not accurate are solved.

Description

Safety monitoring system and monitoring method based on machine vision
Technical Field
The invention relates to the field of network security monitoring and image processing, in particular to a security monitoring system and a security monitoring method based on machine vision.
Background
With the rapid development of data centers and cloud computing technology, security monitoring of server hardware has become increasingly important. In a data center, monitoring of server hardware is an important link to ensure stable operation of the system. The monitoring of the prior server hardware mainly depends on manual inspection and basic hardware monitoring tools, and has the problems of untimely monitoring, high false alarm rate and the like. With the development of technology, some server hardware security monitoring systems based on machine vision are applied to actual scenes.
However, existing machine vision based server hardware security monitoring systems still have some drawbacks. Firstly, when the systems process images in a complex environment inside a server cabinet, it is often difficult to effectively separate and inhibit interference from various noise sources such as server equipment operation and light change inside the cabinet, so that a hardware state detection result is easily affected by noise, and false alarm or missing report is generated. Secondly, in terms of server hardware anomaly detection and maintenance operation analysis, the existing system is difficult to effectively identify a complex hardware failure mode or potential dangerous maintenance operation, and particularly in a multi-server and high-density deployment scene, the performance of the system often cannot meet the actual requirements. Moreover, many systems fail to fully utilize data from different sensors within the server for multimodal fusion, resulting in insufficiently comprehensive risk assessment of the running state of the server, possibly ignoring certain potential hardware failures or potential safety hazards. The early warning mechanism of the existing system often lacks flexibility, so that the response speed to the abnormal state of the server is lagged or the response measures are not accurate enough in practical application, and finally the safe operation effect of the data center is affected.
Disclosure of Invention
The invention provides a safety monitoring system and a monitoring method based on machine vision, which are used for solving the problems that the detection result is easily affected by noise and is false-positive or false-negative due to incapability of effectively separating and inhibiting interference from various noise sources such as equipment operation, environmental light change and the like, complex abnormal behaviors or dangerous operation are difficult to effectively identify, multi-mode fusion is not fully utilized for data from different sensors, risk assessment is not comprehensive and potential dangerous factors in the environment can be possibly ignored, and the early warning mechanism is lack of flexibility and can lead to lag in response speed or inaccurate response measures in practical application and finally influence safety guarantee.
The invention discloses a safety monitoring system and a monitoring method based on machine vision, which specifically comprise the following technical scheme:
A machine vision based security monitoring method comprising the steps of:
S1, acquiring visual image data of server hardware in real time, performing multi-frequency domain decomposition on the visual image data through generalized Fourier-Bessel transformation to obtain signals of different frequency domains, performing adaptive nonlinear filtering processing on the signals of the different frequency domains to obtain frequency domain signals after filtering processing;
s2, obtaining global feature representation through a depth network with nonlinear superposition of a high-order polynomial based on the denoised reconstructed image, constructing a space-time diagram based on the global feature representation, analyzing behaviors to obtain comprehensive space-time feature representation, carrying out fusion processing on the comprehensive space-time feature representation and sensor data to obtain fused fuzzy decision data, and carrying out risk assessment based on the fused fuzzy decision data to generate an early warning strategy.
Preferably, the S1 specifically includes:
And carrying out self-adaptive nonlinear filtering processing on signals in different frequency domains by introducing nonlinear transformation to obtain the frequency domain signals after the filtering processing.
Preferably, the S1 specifically includes:
and recombining the filtered frequency domain signals through weighted fusion to obtain a denoised reconstructed image.
Preferably, the S2 specifically includes:
Based on the denoised reconstructed image, a high-order polynomial nonlinear superimposed depth network is introduced, each layer of feature representation of the depth network is extracted, and the feature representations of all layers of the depth network are weighted and fused to obtain a global feature representation.
Preferably, the S2 specifically includes:
The method comprises the steps of constructing a space-time diagram based on global feature representation, introducing a high-order Bessel function and Laplacian operator joint transformation, analyzing the space-time diagram to obtain space-time feature representation, carrying out convolution operation based on the space-time feature representation, and aggregating space-time feature information at different moments to obtain comprehensive space-time feature representation.
Preferably, the S2 specifically includes:
And performing fusion processing on the comprehensive space-time characteristic representation and the sensor data through fuzzy logic and convolution integration to obtain fused fuzzy decision data.
Preferably, the S2 specifically includes:
And generating an early warning strategy through nonlinear mapping and a fuzzy reasoning model based on the risk assessment result.
A machine vision based security monitoring system comprising:
the system comprises an image acquisition module, a data preprocessing module, a target detection module, a behavior analysis module, an early warning response module and a database;
the image acquisition module captures visual image data of the server hardware in real time and transmits the acquired visual image data to the data preprocessing module;
the data preprocessing module carries out multi-frequency domain decomposition on the visual image data through generalized Fourier-Bezier transformation to obtain signals in different frequency domains; the method comprises the steps of carrying out self-adaptive nonlinear filtering processing on signals in different frequency domains to obtain frequency domain signals after filtering processing, carrying out weighted fusion on reconstructed images based on the frequency domain signals after filtering processing to generate denoised reconstructed images, and transmitting the denoised reconstructed images to a target detection module and a database;
the target detection module is used for extracting each layer of characteristic representation of the depth network from the denoised reconstructed image based on the depth network with nonlinear superposition of the high-order polynomial, and carrying out weighted fusion on the characteristic representations of all layers of the depth network to obtain a global characteristic representation;
the behavior analysis module is used for constructing a space-time diagram by utilizing a high-order Bezier transformation network based on the global feature representation, extracting the space-time feature representation, and aggregating the space-time features through convolution operation to form a comprehensive space-time feature representation;
The early warning response module is used for carrying out fusion processing on the comprehensive space-time characteristic representation and the sensor data to obtain fused fuzzy decision data, carrying out risk assessment based on the fused fuzzy decision data to obtain a risk assessment result, and generating an early warning strategy through nonlinear mapping and a fuzzy reasoning model based on the risk assessment result;
The database is used for storing the data transmitted by the data preprocessing module, the target detection module and the behavior analysis module.
The technical scheme of the invention has the beneficial effects that:
1. Environmental noise is effectively separated and suppressed through combination of generalized Fourier-Bessel transformation and adaptive nonlinear filtering, key visual image characteristics are reserved, the characteristic of local noise is considered in the adaptive nonlinear filtering process, and the contrast of visual image signals is enhanced through nonlinear transformation, so that the accuracy and the robustness of visual image processing are improved, and the reliability of subsequent target detection is ensured;
2. The method comprises the steps of carrying out multi-level feature extraction and fusion on a denoised reconstructed image by utilizing a depth network of a high-order polynomial nonlinear superposition, and effectively capturing complex features, particularly high-order features, in the denoised reconstructed image;
3. The machine vision system can accurately capture the behavior mode and the interrelation of the target object in the space-time dimension by constructing a space-time diagram and combining the high-order Bessel function and the Laplace operator joint transformation, and the aggregation of the space-time characteristics further enhances the capability of the machine vision system in the aspects of analyzing the target behavior and identifying abnormal operation, so that the potential safety risk can be effectively detected;
4. The invention not only relies on the data of the machine vision system, but also combines the data from various sensors (such as temperature, pressure, vibration and the like), and the multi-mode data are fused in a fuzzy logic and convolution integral mode to form unified risk assessment data;
5. Based on the fused fuzzy decision data, the machine vision system realizes dynamic assessment of risks through second derivative analysis, and triggers early warning response mechanisms of different levels according to risk assessment results. The machine vision system is provided with a plurality of early warning thresholds, so that different early warning levels can be flexibly controlled, and the machine vision system can respond timely and accurately under various risk conditions from low-level early warning (such as sound warning) to high-level early warning (such as automatic shutdown).
Drawings
FIG. 1 is a block diagram of a machine vision based security monitoring system according to the present invention;
FIG. 2 is a flow chart of a machine vision-based security monitoring method according to the present invention;
FIG. 3 is a topology diagram of a visual monitoring scene of a server according to the present invention;
fig. 4 is a design diagram of a safety monitoring system architecture based on machine vision according to the present invention.
Detailed Description
In order to further illustrate the technical means and effects adopted by the present invention to achieve the preset purpose, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The following specifically describes a specific scheme of a machine vision-based safety monitoring system and a monitoring method provided by the invention with reference to the accompanying drawings.
Referring to FIG. 1, a block diagram of a machine vision based security monitoring system according to one embodiment of the present invention is shown, the system comprising:
the system comprises an image acquisition module, a data preprocessing module, a target detection module, a behavior analysis module, an early warning response module and a database;
the image acquisition module captures visual image data of the server hardware in real time through the camera, and transmits the captured image data to the data preprocessing module;
The data preprocessing module carries out multi-frequency domain decomposition on the visual image data by adopting generalized Fourier-Bessel transformation to obtain signals of different frequency domains; noise suppression and characteristic enhancement are carried out on signals in different frequency domains through a self-adaptive nonlinear filter, and finally, reconstructed image data are fused through weighting to obtain a denoised reconstructed image;
The target detection module is used for extracting multi-layer features from the denoised reconstructed image based on a depth network of the nonlinear superposition of the high-order polynomials, generating a global feature representation for identifying potential dangerous targets or abnormal equipment states;
The behavior analysis module is used for constructing a space-time diagram by utilizing a high-order Bezier transformation network based on the global feature representation, extracting the space-time feature representation, aggregating the space-time features through convolution operation to form a comprehensive space-time feature representation, and analyzing a behavior mode of a target to identify dangerous behaviors or abnormal operation;
the early warning response module is used for carrying out fusion processing on the comprehensive space-time characteristic representation and other sensor data to obtain fused fuzzy decision data, carrying out risk assessment based on the fused fuzzy decision data to obtain a risk assessment result, generating an early warning signal based on the risk assessment result through nonlinear mapping and a fuzzy reasoning model and used for triggering corresponding safety response measures, and the machine vision system is provided with a plurality of early warning thresholds, wherein each early warning threshold corresponds to early warning or response mechanisms of different levels.
The database is used for storing the data transmitted by the data preprocessing module, the target detection module and the behavior analysis module.
Referring to fig. 2, a flow chart of a machine vision based security monitoring method according to an embodiment of the present invention is shown, the method includes the following steps:
S1, acquiring visual image data of server hardware in real time, performing multi-frequency domain decomposition on the visual image data through generalized Fourier-Bessel transformation to obtain signals of different frequency domains, performing adaptive nonlinear filtering processing on the signals of the different frequency domains to obtain frequency domain signals after filtering processing;
Referring to fig. 3 and 4, the client captures visual image data of the server hardware through the image acquisition module and transmits the visual image data to the server, the server comprises a data preprocessing module, a target detection module, a behavior analysis module and an early warning response module, the captured visual image data is stored in a standard format, and the visual image data is transmitted to the data preprocessing module in real time through a high-speed data transmission interface (such as GigE or USB 3.0) and is stored and managed in a server cabinet, so that the integrity and the ready availability of the data are ensured.
The collected visual image data contains abundant environmental information, but also contains a large amount of complex noise, and the noise may come from equipment operation, environmental light change and other uncertain factors, so preprocessing is needed to improve the accuracy of subsequent analysis, and a data preprocessing module in the service-side elastic computing equipment adopts generalized Fourier-Bessel transformation to carry out multi-frequency-domain decomposition on the visual image data. The generalized fourier-bessel transform formula is:
Wherein F i (x, y) represents the value of the visual image signal in the ith frequency domain, and is taken as the frequency domain component of the input visual image data after being subjected to the generalized Fourier-Bessel transformation, namely the signal in the ith frequency domain; Is a complex exponential term representing the contributions of different frequency components in the Fourier transform, ω m,n is the angular frequency, J is the ordinal unit, J v (ar) is a Bessel function representing the amplitude distribution in the frequency domain, v is the order of the Bessel function, α is a parameter regulating the frequency amplitude, r is the spatial distance
After the multi-frequency-domain decomposition of the visual image data is completed, the data preprocessing module also carries out self-adaptive nonlinear filtering processing on signals F i (x, y) in different frequency domains so as to inhibit noise and enhance key characteristics, and the self-adaptive nonlinear filter not only considers the characteristics of local noise, but also introduces nonlinear transformation to further optimize the processing effect. The specific formula of the adaptive nonlinear filtering is as follows:
Wherein, F' i (x, y) represents the visual image signal on the ith frequency domain after the adaptive nonlinear filtering process, namely the frequency domain signal after the filtering process; An estimated value representing noise of the visual image signal at the position (x, y) in the ith frequency domain is obtained by statistically analyzing gray scale variation of a local area of the visual image signal, lambda i is an adjustment parameter of an adaptive nonlinear filter for controlling the intensity of the filter to ensure that a useful signal is retained while suppressing noise, a nonlinear function tanh (gamma.f i (x, y)) further enhances the contrast of the visual image signal in the ith frequency domain, wherein gamma is a nonlinear coefficient for adjusting the strength of enhancement of the visual image signal in the ith frequency domain, and omega is a frequency component in the frequency domain for describing the distribution of the visual image signal in the frequency domain in the ith frequency domain.
And recombining the filtered frequency domain signals, and weighting and fusing the results of different frequency domains to form a denoised reconstructed image. The formula is as follows:
Wherein, I' (x, y) is the value of the denoised reconstructed image at the space coordinates (x, y), alpha i is the weighting coefficient of the ith frequency domain, which represents the importance of the ith frequency domain in the final image reconstruction; Cos (βω) is a cosine function used for frequency domain fusion, β is a parameter for adjusting the frequency of the cosine function, and ω is a frequency component in the frequency domain. The above-mentioned de-noised reconstructed image I' (x, y) has noise effectively suppressed while retaining important image features.
S2, obtaining global feature representation through a depth network with nonlinear superposition of a high-order polynomial based on the denoised reconstructed image, constructing a space-time diagram based on the global feature representation, analyzing behaviors to obtain comprehensive space-time feature representation, carrying out fusion processing on the comprehensive space-time feature representation and sensor data to obtain fused fuzzy decision data, and carrying out risk assessment based on the fused fuzzy decision data to generate an early warning strategy.
The target detection module takes the denoised reconstructed image as input, and uses a high-order polynomial nonlinear superimposed depth network to perform target detection, so that a potential dangerous target or abnormal equipment state is accurately identified from the denoised reconstructed image.
Specifically, the multi-layer characteristic representation is extracted through a depth network of nonlinear superposition of a high-order polynomial, and the formula is as follows:
Wherein G l is the characteristic representation of the first layer depth network, Y l is the input characteristic of the first layer depth network, Y 1 =I' (x, Y) is the input characteristic of the 1 st layer depth network, and the input denoising reconstructed image is subjected to multi-layer nonlinear transformation The generated characteristic representation is used as an input characteristic of the first layer depth network and the second layer depth network; Is the power of q of the input features of the depth of layer network, V l is the weight matrix of the depth of layer network, b l is the bias term of the depth of layer network, q represents the order of the polynomial by taking the derivative of q order for a particular dimension in the input features of the depth of layer network The method can further capture the higher-order features in the denoised reconstructed image, wherein z is a specific dimension (such as a space coordinate) in the input features of the first layer depth network, p is the polynomial order of the highest order, namely the depth of the extracted features, and the nonlinear expression capability of the depth network is determined; is the superposition coefficient, controlling the impact of the feature representation of the previous depth-of-layer network on the feature representation of the current depth-of-layer network. Each layer of feature representation G l of the depth network will be superimposed stepwise in multiple levels, forming a more expressive global feature representation.
The generation of the global feature representation is accomplished by the following formula:
Wherein G global is a global feature representation, delta l is a fusion weight of feature representations of each layer of depth network, and L is the total layer number of the depth network with nonlinear superposition of high-order polynomials. And obtaining a global feature representation integrating information of all layers of the depth network by carrying out weighted fusion on the feature representations of all layers of the depth network and integrating the feature representations in a time dimension r.
The behavior analysis module constructs a space-time diagram G based on the global feature representation G global, the purpose of which is to capture the interrelationship and behavior pattern of the target object in time and space through the structured representation of the diagram, identifying dangerous behavior or abnormal operation. The specific implementation steps are as follows:
The global feature representation G global is a feature representation of each detection target, namely each layer of the depth network, and is used as a node of a time-space diagram, the feature vector of the node is from a specific subset in the global feature representation and is used for representing the spatial position, speed, appearance features and the like of the detection targets, the edges between the nodes represent the space-time relationship between different detection targets, the features of the edges are calculated by the feature vectors of the two nodes, and common calculation methods comprise Euclidean distance, cosine similarity and the like. The weight of the edge depends on the physical distance and relative speed between the detection targets, namely, the natural logarithm is taken as a base, the index is the Euclidean distance between two nodes divided by the adjusting parameter used for controlling the attenuation speed of the edge weight, and the time space diagrams at different moments are connected through the time dimension to form a time sequence diagram. The nodes at each moment are connected with the nodes at the next moment through edges, so that the continuity in time and space is ensured.
After the time-space diagram G is built, the time-space diagram G is analyzed, and in order to better capture the time-space characteristics, a high-order Bessel function and Laplacian joint transformation is introduced, wherein the formula is as follows:
H0＝σ(W0·Gglobal+b0)
Wherein H t+1 is the spatiotemporal feature representation of the next time, the spatiotemporal feature representation H t;H0 which combines the second derivative feature of the spatiotemporal graph and the spatiotemporal feature representation of the current time is the initial spatiotemporal feature representation; Is Laplacian, and is used for extracting local second derivative characteristics of a space-time diagram and enhancing the representation of space-time dependency relationship, wherein J k is a kth order Bessel function; Representing the highest order of the Bessel function; for the weight corresponding to the k-th order Bessel function, the contribution of different order space-time characteristics in the space-time diagram is controlled The method comprises the steps of respectively adjusting coefficient and frequency parameters for adjusting the amplitude and frequency of the change of the space-time characteristic in the time dimension, cosh is a hyperbolic cosine function for further enhancing the expression capability of the space-time characteristic, W 0 is a weight matrix for carrying out linear transformation on the global characteristic representation, b 0 is a bias vector added to the result of the linear transformation for adjusting the central position of the space-time characteristic representation and avoiding overreliance on zero values.
The extracted space-time characteristic representations are aggregated by convolution operation to form a comprehensive space-time characteristic representation, and the formula is as follows:
wherein H final is a comprehensive spatiotemporal feature representation; The method is characterized in that the method is a weighting coefficient of a time dimension, W c is a convolution kernel, the time-space characteristic information at different moments is aggregated through convolution operation on the time-space characteristic representation, and finally a comprehensive time-space characteristic representation is formed, and T is the length of a time sequence, namely the total time step number.
The comprehensive space-time characteristic representation is used as a comprehensive video stream capable of reflecting the current risk state, the server side displays the processed video stream through the front end, and an operator can check real-time data, adjust safety monitoring system parameters and check a historical alarm record through a configuration page.
The early warning response module takes the comprehensive space-time characteristic representation generated by the behavior analysis module as input, and performs fusion processing on the comprehensive space-time characteristic representation and data acquired by other sensors (such as a temperature sensor, a pressure sensor, a vibration sensor, a humidity sensor and the like), wherein the other sensor data are used for supplementing image data of a machine vision system, and more dimension data are provided to help a safety monitoring system to evaluate the environment state and potential safety risks more accurately. The fusion process is realized by fuzzy logic and convolution integral, and the formula is as follows:
wherein D f is the fuzzy decision data after fusion as the basic data of risk assessment, u is the index of other sensor data, N is the total number of other sensor data; membership function of fuzzy set, expressed according to The degree of membership calculated is determined by the method,The characteristic value extracted from S u, S u is a u-th sensor data matrix, the u-th sensor data matrix is obtained by fusion of convolution operation and comprehensive space-time characteristics, dω is integral of frequency variation, the integral operation ensures that all frequency domain components in the sensor data are effectively processed, and the integral operation shows that the sensor data are comprehensively processed in the frequency domain.
And performing risk assessment on the fused fuzzy decision data, wherein the formula is as follows:
wherein R (t) is the risk level of the current moment t, and represents a risk value obtained by evaluation according to the input data of the current moment, namely a risk evaluation result; Representing the second derivative of the fused fuzzy decision data in the time dimension, and reflecting the dynamic change of the fused fuzzy decision data along with the time; and R (t-1) is the risk level at the previous moment.
According to the risk assessment result, an early warning strategy is determined through nonlinear mapping and a fuzzy reasoning model, and the formula is as follows:
wherein D output (t) is a decision output signal at the time t, θ is a decision weight representing the influence of a risk assessment result in early warning decision, τ is a normalization coefficient for adjusting the scale of the risk level R (t) and ensuring the smoothness of the decision output signal. The final output D output (t) is an early warning signal of the alarm system for triggering corresponding safety measures, such as alarms, notifications or automatic stops.
The alarm system sets a plurality of early warning thresholds, each early warning threshold corresponds to an early warning or response mechanism of different levels, and when D .utput (t) reaches or exceeds a certain early warning threshold, the alarm system triggers the corresponding early warning mechanism. And according to the triggered early warning mechanism, the warning system decides what early warning measures are started. For example, the early warning levels are divided into low, medium and high levels, respectively corresponding to different risk levels, the low level early warning may trigger an audible alarm or display a warning message on a monitor screen to prompt the attention of an operator, the medium level early warning may send a short message or email to inform security manager to further investigate the potential risk, and the high level early warning may immediately execute emergency measures such as automatic shutdown and security isolation to prevent serious accidents.
Therefore, flexible control of different early warning levels is realized, and the machine vision system can make timely and accurate response under various risk conditions, so that the safety of a network environment is ensured.
In summary, the safety monitoring system and the monitoring method based on machine vision are completed.
The sequence of the embodiments of the invention is merely for description and does not represent the advantages or disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
In this specification, each embodiment is described in a progressive manner, and the same or similar parts of each embodiment are referred to each other, and each embodiment mainly describes differences from other embodiments.
The foregoing embodiments are merely for illustrating the technical solution of the present invention, but not for limiting the same, and although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solution described in the foregoing embodiments may be modified or substituted for some of the technical features thereof, and that such modifications or substitutions do not depart from the spirit and scope of the technical solution of the embodiments of the present invention and are intended to be included in the scope of the present invention.

Claims (7)

1. A machine vision based security monitoring method, comprising the steps of:
S1, acquiring visual image data of server hardware in real time, performing multi-frequency domain decomposition on the visual image data through generalized Fourier-Bessel transformation to obtain signals of different frequency domains, performing adaptive nonlinear filtering processing on the signals of the different frequency domains to obtain frequency domain signals after filtering processing;
S2, extracting feature representations of each layer of the depth network through a depth network with nonlinear superposition of high-order polynomials based on the denoised reconstructed image, weighting and fusing the feature representations of all layers of the depth network to obtain global feature representations, constructing a space-time diagram based on the global feature representations, analyzing the space-time diagram to obtain space-time feature representations, carrying out convolution operation based on the space-time feature representations, aggregating space-time feature information at different moments to obtain comprehensive space-time feature representations, fusing the comprehensive space-time feature representations with sensor data to obtain fused fuzzy decision data, and carrying out risk assessment based on the fused fuzzy decision data to generate an early warning strategy.
2. The machine vision-based safety monitoring method according to claim 1, wherein in S1, adaptive nonlinear filtering processing is performed on signals in different frequency domains to obtain filtered frequency domain signals, and the method specifically comprises:
And carrying out self-adaptive nonlinear filtering processing on signals in different frequency domains by introducing nonlinear transformation to obtain the frequency domain signals after the filtering processing.
3. The machine vision based security monitoring method according to claim 2, wherein in S1, the generating a denoised reconstructed image based on the filtered frequency domain signal specifically comprises:
and recombining the filtered frequency domain signals through weighted fusion to obtain a denoised reconstructed image.
4. The machine vision-based safety monitoring method according to claim 1, wherein in S2, the space-time diagram is analyzed to obtain a space-time feature representation, and the method specifically comprises:
And introducing a high-order Bessel function and Laplacian operator joint transformation, and analyzing the time-space diagram to obtain a time-space characteristic representation.
5. The machine vision based safety monitoring method according to claim 4, wherein in S2, the integrated space-time feature representation and the sensor data are fused to obtain the fused fuzzy decision data, and the method specifically comprises:
And performing fusion processing on the comprehensive space-time characteristic representation and the sensor data through fuzzy logic and convolution integration to obtain fused fuzzy decision data.
6. The machine vision-based safety monitoring method according to claim 5, wherein in S2, risk assessment is performed based on the fused fuzzy decision data to generate an early warning strategy, and the method specifically comprises:
And generating an early warning strategy through nonlinear mapping and a fuzzy reasoning model based on the risk assessment result.
7. A machine vision based security monitoring system for use in a machine vision based security monitoring method as defined in claim 1, comprising the steps of:
The system comprises an image acquisition module, a data preprocessing module, a target detection module, a behavior analysis module, an early warning response module and a database;
the image acquisition module captures visual image data of the server hardware in real time and transmits the acquired visual image data to the data preprocessing module;
The data preprocessing module carries out multi-frequency domain decomposition on the visual image data through generalized Fourier-Bezier transformation to obtain signals in different frequency domains; the method comprises the steps of carrying out self-adaptive nonlinear filtering processing on signals in different frequency domains to obtain frequency domain signals after filtering processing, carrying out weighted fusion on reconstructed images based on the frequency domain signals after filtering processing to generate denoised reconstructed images, and transmitting the denoised reconstructed images to a target detection module and a database;
The target detection module is used for extracting each layer of characteristic representation of the depth network from the denoised reconstructed image based on the depth network with nonlinear superposition of the high-order polynomial, and carrying out weighted fusion on the characteristic representations of all layers of the depth network to obtain a global characteristic representation;
The behavior analysis module is used for constructing a space-time diagram by utilizing a high-order Bezier transformation network based on the global feature representation, extracting the space-time feature representation, and aggregating the space-time features through convolution operation to form a comprehensive space-time feature representation;
the database is used for storing the data transmitted by the data preprocessing module, the target detection module and the behavior analysis module;
The early warning response module is used for carrying out fusion processing on the comprehensive space-time characteristic representation and the sensor data to obtain fused fuzzy decision data, carrying out risk assessment based on the fused fuzzy decision data to obtain a risk assessment result, and generating an early warning strategy through nonlinear mapping and a fuzzy reasoning model based on the risk assessment result.
CN202411404721.9A 2024-10-09 2024-10-09 A safety monitoring system and monitoring method based on machine vision Active CN119299332B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202411404721.9A CN119299332B (en) 2024-10-09 2024-10-09 A safety monitoring system and monitoring method based on machine vision

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202411404721.9A CN119299332B (en) 2024-10-09 2024-10-09 A safety monitoring system and monitoring method based on machine vision

Publications (2)

Publication Number Publication Date
CN119299332A CN119299332A (en) 2025-01-10
CN119299332B true CN119299332B (en) 2025-06-17

Family

ID=94148131

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202411404721.9A Active CN119299332B (en) 2024-10-09 2024-10-09 A safety monitoring system and monitoring method based on machine vision

Country Status (1)

Country Link
CN (1) CN119299332B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120031388B (en) * 2025-04-02 2025-08-05 广东省建设工程质量安全检测总站有限公司 Method, device and equipment for evaluating the operational risk factor of static load single pile test platform

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109522793A (en) * 2018-10-10 2019-03-26 华南理工大学 More people's unusual checkings and recognition methods based on machine vision
CN110044405A (en) * 2019-05-16 2019-07-23 吉林大学 A kind of automobile instrument automatic detection device and method based on machine vision

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115018073A (en) * 2022-08-09 2022-09-06 之江实验室 Time-space perception information prediction method and system based on graph neural network
WO2024119360A1 (en) * 2022-12-06 2024-06-13 山东科技大学 Elevator safety fault diagnosis method based on multi-source information fusion

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109522793A (en) * 2018-10-10 2019-03-26 华南理工大学 More people's unusual checkings and recognition methods based on machine vision
CN110044405A (en) * 2019-05-16 2019-07-23 吉林大学 A kind of automobile instrument automatic detection device and method based on machine vision

Also Published As

Publication number Publication date
CN119299332A (en) 2025-01-10

Similar Documents

Publication Publication Date Title
Franklin et al. Anomaly detection in videos for video surveillance applications using neural networks
CN120088957A (en) Camera linkage alarm method and system for intelligent environment monitoring
KR20230103890A (en) Video surveillance system based on multi-modal video captioning and method of the same
CN118952253B (en) Intelligent inspection robot system based on Internet of things
CN116309590B (en) Visual computing method, system, electronic equipment and medium based on artificial intelligence
CN120471293A (en) Civil aviation airport security method and system based on multi-source perception and dynamic risk assessment
CN119299332B (en) A safety monitoring system and monitoring method based on machine vision
CN119046846A (en) Deep learning-based ATM (automatic teller machine) smashing recognition method and related equipment
CN113989930A (en) Human Abnormal Behavior Detection System Based on Small Target and Action Recognition
CN116129275A (en) Mountain fire smoke identification method based on interference suppression threshold voting strategy
CN121167487A (en) Intelligent metering laboratory environment monitoring and evaluating method and system
CN120954100A (en) A method and device for human motion recognition
CN120561674A (en) Converter station fault identification method, device, computer equipment and readable storage medium
CN121094526A (en) Multi-mode data fusion-based power cross operation dynamic risk prediction and early warning method, system, equipment and medium
Vununu et al. Sound based machine fault diagnosis system using pattern recognition techniques
CN118175065A (en) A device port security alarm system
CN119625801A (en) Real-time face-changing detection method, system, device and medium based on deep learning
CN117636601A (en) An intelligent early warning method based on scene deformation information
CN116740627A (en) Violation alarm method based on operational knowledge
CN116822929A (en) Alarm method, alarm device, electronic equipment and storage medium
CN118736683B (en) Method and system for monitoring operator rule violation based on distributed control ball
CN120976837B (en) A method, electronic equipment, and storage medium for identifying dangerous postures of personnel in boom trucks based on multi-source sensor fusion.
CN121119039B (en) Automated detection device for the risk of AI model training data contamination
CN119992450B (en) Electronic evidence identification method and system
CN120469381B (en) Industrial safety monitoring system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant