CN120234324B - A laboratory data collection management method, system, equipment and medium - Google Patents
A laboratory data collection management method, system, equipment and mediumInfo
- Publication number
- CN120234324B CN120234324B CN202510716638.3A CN202510716638A CN120234324B CN 120234324 B CN120234324 B CN 120234324B CN 202510716638 A CN202510716638 A CN 202510716638A CN 120234324 B CN120234324 B CN 120234324B
- Authority
- CN
- China
- Prior art keywords
- data
- delay
- database
- sensor
- laboratory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Security & Cryptography (AREA)
- Databases & Information Systems (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Automatic Analysis And Handling Materials Therefor (AREA)
Abstract
The invention discloses a laboratory data acquisition management method, system, equipment and medium, which relate to the technical field of data acquisition management and comprise the steps of associating data with a unique sample identifier to generate a standardized data packet, attaching a decision variable, optimizing balance delay and retransmission rate by multiple targets, dynamically adjusting batch writing quantity and a database and table dividing strategy, marking abnormal data, classifying the abnormal data by adopting a clustering algorithm, and adaptively adjusting a detection threshold value. The method of the invention realizes throughput self-adaptive adjustment to keep data integrity through message queue dynamic partitioning, keeps high concurrency scene writing delay stable through dynamic scaling of batch writing quantity, shortens inquiry response time through SSD/HDD (solid state disk) database partitioning strategy and time partition joint index, improves abnormal classification accuracy through a clustering algorithm, reduces false alarm rate, ensures data rollback accuracy when a database fails through transaction retry back-off, avoids dirty data generation, and optimizes a network in real time through network state dynamic calculation.
Description
Technical Field
The invention relates to the technical field of data acquisition management, in particular to a laboratory data acquisition management method, system, equipment and medium.
Background
The traditional laboratory sample management method has the problems of more manual operation, inflexible scheduling, low sample storage and taking efficiency and the like. These problems are particularly prominent in the fields of high-throughput experiments, drug development, clinical detection, etc., and as the number of samples increases, the management difficulty also increases significantly. In the prior art, although some sample tracking systems based on bar codes and RFID technology exist, many sample tracking systems lack intelligent scheduling functions, and optimal scheduling cannot be effectively performed according to factors such as laboratory requirements, sample storage conditions, environmental conditions and the like, so that waste of sample storage space and taking efficiency is caused. Particularly, when a laboratory needs to schedule a large number of samples, how to efficiently and rapidly schedule, manage and track the samples on the premise of ensuring the safety of the samples is still a technical problem to be solved.
Disclosure of Invention
The present invention has been made in view of the above-described problems.
Therefore, the invention solves the technical problems of how to solve various problems in laboratory data acquisition and management and improve the acquisition efficiency, processing capacity, storage safety, real-time performance and traceability of experimental data.
The technical scheme includes that the laboratory data acquisition management method comprises the steps of acquiring experimental data, associating the data with a sample unique identifier to generate a standardized data packet with decision variables, establishing a transmission delay model and a retransmission probability model, dynamically adjusting batch writing quantity and a database and table dividing strategy when the data are written into a database through multi-objective optimization balance delay and retransmission probability, marking abnormal data based on a preset threshold value and a change rate rule, classifying the abnormal data by adopting an improved fuzzy C-means clustering algorithm, and adaptively adjusting a detection threshold value by a random gradient descent method.
The laboratory data acquisition management method comprises the steps of automatically acquiring experimental data by using an RFID scanner, a sensor and an IoT device, wherein the RFID scanner is installed at a key node of a sample entering and exiting laboratory, a scanning range covers a whole sample moving process, the sensor comprises a temperature sensor, a pressure sensor, a PH sensor, a flowmeter and a spectrum analyzer, when the sensor is installed, calibration is carried out according to the layout of an experimental area, the temperature sensor is calibrated at three points at a standard temperature of 25 ℃ in a constant temperature box, a calibration coefficient is generated and stored in a LIMS, the PH sensor is calibrated by using a standard buffer solution, if the deviation between three continuous measured values and the standard value exceeds +/-0.2, automatic alarm is triggered and data acquisition is suspended, a loT gateway is deployed at the central position of the experimental area, the sensor data is acquired through the Modbus protocol period, when the network delay exceeds 200ms, the data change amount is more than or equal to 5%, and the data is uploaded.
The method for managing laboratory data acquisition is characterized in that the identification of the data comprises the steps that when a data packet is generated, a system distributes a globally unique identifier for each sample and binds the globally unique identifier with an RFID tag, data packet metadata comprises experiment types, operator IDs and equipment states, the decision variables are added to the data packet header, and the decision variables comprise ACK waiting time thresholdsMaximum number of retransmissionsNumber of message queue partitionsACK latency thresholdComprises dynamic calculation according to network state, maximum retransmission timesIncludes setting initial value and upper limit value, if two continuous transmissions failChanging maximum retransmission times upper limit, message queue partition numberIncluding dynamic adjustment based on data inflow rate and processing rate.
The method for managing the laboratory data acquisition is used for establishing a transmission delay model and a retransmission probability model, wherein the method comprises the steps of defining the transmission delay model, designing a transmission delay objective function, establishing the retransmission probability model, designing the retransmission probability objective function, establishing a multi-objective optimization problem based on data integrity verification and system load limitation, and obtaining an optimal solution of delay and retransmission rate through a non-dominant ranking genetic algorithm.
As a preferable scheme of the laboratory data acquisition management method, the method comprises the steps of dynamically adjusting the batch writing quantity and the database and table dividing strategy, wherein the method comprises the steps of setting the batch writing quantityInitial value, if write delayGreater than a high threshold of delayAccording toGradually reducing if write delayLess than the low delay thresholdAccording toGradually increasing, dividing a database strategy according to experimental types, storing high frequency data into an SSD (solid state disk) database, storing low frequency data into an HDD (hard disk drive) database, dividing a table dividing strategy according to time ranges, generating a new table each month, establishing a joint index for a sample ID (identity) and a time stamp, setting transaction commit timeout, and storing the data in the SSD databaseIf the transaction fails, according toRetrying, if retrying for 3 times, rolling back the transaction and recording the log.
The method for managing laboratory data acquisition is characterized in that the marked abnormal data comprises the steps of detecting the acquired data by LIMS (local information modeling system) in real time, judging temperature abnormality of temperature sensor data, setting a normal interval for a pH (potential hydrogen) sensor, calculating the change rate of any sensor data in unit time, defining preliminary abnormality judgment of data in unit time, and if any data triggers abnormality, marking data points as abnormality.
The method for acquiring and managing the laboratory data is characterized in that the abnormal classification comprises classifying the abnormal data according to the characteristics of the abnormal data and constructing a characteristic vector of each abnormal data point through an improved fuzzy C-means clustering algorithm, the improved fuzzy C-means clustering algorithm comprises introducing a time continuity punishment item based on a standard fuzzy C-means clustering algorithm to construct an improved objective function, the self-adaptive adjustment of a detection threshold comprises updating a threshold parameter through a random gradient descent method, and if loss is continuous and not reduced, an expert review flow is triggered.
The laboratory data acquisition management system comprises a data acquisition module, a data transmission module and a data abnormality detection module, wherein the data acquisition module is used for acquiring experimental data, associating the data with a sample unique identifier to generate a standardized data packet with decision variables, the data transmission module is used for establishing a transmission delay model and a retransmission probability model, dynamically adjusting batch writing quantity and a database and table dividing strategy when data are written into a database through multi-objective optimization balance delay and retransmission rate, and the data abnormality detection module is used for marking abnormal data based on a preset threshold value and a change rate rule, classifying the abnormal data by adopting an improved fuzzy C mean value clustering algorithm and adaptively adjusting a detection threshold value through a random gradient descent method.
A computer device comprising a memory storing a computer program and a processor executing the computer program is a step of implementing a laboratory data acquisition management method.
A computer readable storage medium having stored thereon a computer program which when executed by a processor implements the steps of a laboratory data acquisition management method.
The laboratory data acquisition management method has the advantages that the laboratory data acquisition management method is linked with multiple types of sensors through RFID full-node coverage, automatic acquisition of full-life cycle data from sample feeding to experiment is achieved, manual input errors are eliminated, data acquisition accuracy is guaranteed through sensor calibration, data transmission delay in actual measurement in a laboratory is reduced through double-objective optimization of a non-dominant ordering genetic algorithm, retransmission probability is controlled, throughput self-adaptive adjustment is achieved through message queue dynamic partitioning, data integrity can be kept under a sudden flow scene, writing delay of a high concurrency scene is kept stable through dynamic scaling of batch writing quantity, time partitioning joint indexes are used for shortening inquiry response time through SSD/HDD (solid state disk)/HDD (hard disk) database partitioning strategy, abnormal classification accuracy is improved through improving fuzzy C mean value clustering algorithm, false report rate is reduced, data rollback accuracy is guaranteed when a database fails is guaranteed through transaction retry back, dirty data is avoided, and network is dynamically calculated through network state real-time optimization.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a general flow chart of a method for laboratory sample management according to one embodiment of the present invention.
Fig. 2 is a block diagram of a system scheme of a laboratory sample management system according to an embodiment of the present invention.
Detailed Description
So that the manner in which the above recited objects, features and advantages of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to the embodiments, some of which are illustrated in the appended drawings. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
Embodiment 1, referring to fig. 1, for an embodiment of the present invention, there is provided a laboratory data acquisition management method, including:
s1, collecting experimental data, associating the data with a unique sample identifier to generate a standardized data packet, and attaching a decision variable.
Still further, collecting experimental data includes automatically collecting experimental data using an RFID scanner, sensor, ioT device.
The RFID scanner is arranged at a key node of a sample entering and exiting laboratory, and the scanning range covers the whole sample moving process.
The sensors include temperature sensors, pressure sensors, PH sensors, flow meters and spectrum analyzers.
When the sensor is installed, the sensor is calibrated according to the layout of an experimental area, the temperature sensor is calibrated at three points at 25 ℃ standard temperature in the incubator, calibration coefficients are generated and stored in a LIMS laboratory information management system, and the calibration coefficients are stored in the incubatorExpressed as:
Wherein, the The slope is indicated as such,The intercept is indicated as the intercept and,And the PH sensor is calibrated by using a standard buffer, and if the deviation between the measured value and the standard value is more than +/-0.2 in three continuous times, the PH sensor triggers an automatic alarm and pauses data acquisition.
The loT gateway is deployed at the center of the experimental area, sensor data is collected through Modbus protocol period, if the network delay exceeds 200ms, the network is switched to an event triggering mode, and the data is uploaded when the data variation is more than or equal to 5%.
It should be noted that the identification of the data includes, when the data packet is generated, the system assigns a globally unique identifier to each sample and binds with the RFID tag.
The packet metadata includes the type of experiment, operator ID, and equipment status.
The decision variable is appended to the packet header.
Decision variables include ACK latency thresholdMaximum number of retransmissionsNumber of message queue partitions。
ACK latency thresholdIncluding dynamic calculations based on network state, expressed as:
Wherein, the For an average delay of the past 10 minutes,Is the maximum delay.
Maximum number of retransmissionsIncludes setting initial value and upper limit value, if two continuous transmissions failThe upper limit of the maximum retransmission number is changed.
Message queue partition countIncluding dynamic adjustment according to data inflow rate and processing rate, expressed as:
Wherein, the Indicating the data inflow rate of the data,Representing the data processing rate.
And S2, establishing a transmission delay model and a retransmission probability model, balancing delay and retransmission rate through multi-objective optimization, and dynamically adjusting batch writing quantity and a database and table dividing strategy when data are written into a database.
Further, establishing the transmission delay model and the retransmission probability model includes defining the transmission delay model and designing a transmission delay objective function.
A transmission delay objective function expressed as:
Wherein, the The objective function value representing the transmission delay,A delay estimation function is represented and is used,Representing the weight coefficient.
And constructing a retransmission probability model and designing a retransmission probability objective function.
Let retransmission probability beThe retransmission probability objective function is expressed as:
Wherein, the The objective function value representing the retransmission probability,Representing the corresponding weights.
The multi-objective optimization comprises the steps of establishing a multi-objective optimization problem based on data integrity verification and system load limitation, and obtaining an optimal solution of delay and retransmission rate through a non-dominant ordering genetic algorithm.
Data integrity check, check passing rate is satisfied, expressed as:
the system load limit is not exceeded, expressed as:
Parameter ranges, expressed as:
,
comprehensively considering, a multi-objective optimization problem is established, expressed as:
Wherein, the 。
The non-dominant ranking genetic algorithm comprises setting parameters as population size, cross rate and variability rate, and selecting the solution with highest comprehensive score based on iterative output optimal solution set, namelyThe minimum is taken as the final parameter.
It should be noted that, when the data is verified and the pH value is extremely abnormal (for example, exceeds a standard deviation of ±2 set), calibration is automatically performed. The data with abnormality is isolated and marked as 'need to be checked manually'. If multiple devices or sensors feed back similar anomalies, the system automatically recommends suspending the experiment and performing device inspection.
It should also be noted that the dynamic adjustment of the batch write quantity and the split-bank split-table strategy includes setting the batch write quantityInitial value, if write delayGreater than a high threshold of delayAccording toGradually reducing;
If write delay Less than the low delay thresholdAccording toGradually increasing.
The database dividing strategy divides according to experimental types, stores high-frequency data into an SSD database, and stores low-frequency data into an HDD database.
The sub-table strategy generates a new table monthly according to the time range division, and establishes a joint index for the sample ID and the time stamp.
Setting transaction commit time-outsIf the transaction fails, according toAnd (5) retrying.
If the retry is 3 times, the transaction rolls back and logs.
It should also be noted that the data managed by the synchronization optimization write database is transmitted to the data server of the LIMS through a wireless or wired network and stored in the database. The data storage is in a standardized data format. And carrying out data storage and database management modeling to realize high efficiency and data consistency of database writing, reduce writing delay and ensure transaction integrity.
The data is transmitted to the data server of the LIMS via a wireless or wired network and stored in a database. The data store employs standardized data formats for subsequent query, analysis, and report generation. The module can ensure the safety, the integrity and the reliability of data.
In addition to a single data threshold, the rate of change of data may also be an important indicator. Especially in dynamic data collected by flow meters, thermometers, etc., abrupt data change rates may be indicative of equipment failure or environmental anomalies. Flow meter data assuming a normal rate of change of the flow meter in the range of 1-5L/s, if the rate of change of the data exceeds this range (e.g., fluctuates or suddenly changes), the system processes by the following rules:
and starting a preset rule, and checking whether the equipment has mechanical failure or transmission failure. The sampling frequency is automatically adjusted, so that smoother data is ensured to be acquired. And triggering the data rollback, and triggering the manual review and the equipment inspection if the data fluctuation is continuous and is not recovered.
Designing an objective function, namely, minimum write delay objective and establishing a write delay function:
Wherein, the Representing the value of the write delay function,Indicating the average write delay of the write signal,Representing the weights.
Data consistency target, using data consistency check indexTo express, the goal is:
Wherein, the Representing the value of the data consistency objective function,The rate of pass of the consistency is indicated,Is the weight.
Constraint that batch write size B does not exceed system throughput,The transaction commit time t_trans must be less than the system response requirement-the cache flush interval p_cache matches the data update frequency.
The built multi-objective problem is:
Wherein y= 。
It should also be noted that based on real-time data feedback and historical data trends, the LIMS system may implement dynamic adjustment of the threshold. For example, when the system detects an increase in the volatility of certain sensor data, the threshold of the sensor may be automatically adjusted to avoid excessive alarms.
And S3, marking abnormal data based on a preset threshold value and a change rate rule, classifying the abnormal data by adopting an improved fuzzy C-means clustering algorithm, and self-adaptively adjusting a detection threshold value by a random gradient descent method.
Further, marking the abnormal data includes detecting the abnormal data in real time by the LIMS and judging the abnormal temperature of the temperature sensor data.
Setting the normal interval asTemperature anomaly determination, expressed as:
Wherein, the A lower temperature threshold value is indicated,Represents an upper temperature threshold value of the temperature,Indicating that the temperature sensor is at a point in timeIs a measurement of (2);
The pH sensor was set to a normal range, which is expressed as:
Wherein, the Indicating that the pH sensor is at a time pointIs a measurement of (2);
for any sensor data, the rate of change per unit time is calculated as:
Wherein, the Representing the current point in timeIs used for measuring the sensor measurement value of the (a),Representing the previous point in timeIs a measurement of (2);
if any data triggers an anomaly, the marker data point is anomalous.
Setting the normal rate range as(E.g., [1,5L/s ] for a flow meter), the anomaly detection function is:
defining preliminary abnormality determination of unit time data, expressed as:
。
It should be noted that the anomaly classification includes classifying the anomaly data by a modified fuzzy C-means clustering algorithm according to the characteristics of the anomaly data, and constructing a feature vector for each anomaly data point.
The eigenvector for each outlier data point is represented as:
where μ is the historical mean of the corresponding data, |Δr (t) | is the absolute value of the rate of change of the data, and d (t) represents the duration of the anomaly (e.g., the number of consecutive anomaly data points).
The improved fuzzy C-means clustering algorithm comprises the steps of introducing a time continuity penalty term based on a standard fuzzy C-means clustering algorithm, and constructing an improved objective function.
The modified objective function is expressed as:
Wherein: A data point representing a time t belongs to the membership degree of the abnormal class k; represents the center of category k, m represents the ambiguity index, typically m >1, λ represents the weight of the time penalty term; Representing a temporal continuity penalty function, designed to:
The method is used for ensuring that the attribution relation of the data at adjacent moments does not have severe fluctuation.
The self-adaptive adjustment of the detection threshold value comprises updating a threshold value parameter by adopting a random gradient descent method;
if the loss is not continuously reduced, triggering an expert review process.
It should also be noted that the anomaly data is categorized into three categories by the modified fuzzy C-means clustering algorithm:
Mutation abnormalities, which fluctuate widely in a short time, are usually accompanied by a randomly extremely high |Δr (t) |.
Progressive shift-data continued to deviate from normal values, manifested as long-term cumulative deviations.
Single point anomalies, isolated errors, are usually instantaneous deviations from normal before and after.
It should also be noted that, during actual operation, the system will count the situations of false alarm (normal data is marked as abnormal by mistake) and missing alarm (true abnormal is not detected), and define a loss function to measure the influence of the errors, wherein the loss function is based on the number of false alarms and the number of missing alarms, and balances the influence of the false alarms and the missing alarms by setting weights;
And (3) gradient descent updating parameters, wherein according to the gradient of the loss function, the system automatically adjusts preset threshold parameters such as temperature, pH, speed and the like. That is, if the system finds that the current parameter setting results in excessive false positives or false negatives, the thresholds are "turned down" or "turned up" appropriately by calculating the effect of each parameter on the loss function until the detection performance of the system reaches an optimal state, and the process and results of each adjustment of the parameters are recorded to form a "regular feedback loop".
Through the data analysis and processing, when the system monitors data abnormality, an alarm system can be used for triggering alarm information to laboratory staff. And (3) automatically adjusting the system, namely automatically adjusting experimental environment parameters or equipment states, such as temperature and pressure, if conditions allow. And (3) manual intervention, namely when the system judges that the abnormal situation is complex, notifying the manual intervention to operate.
It should be noted that the system is provided with user management and authority control functions, so that the access, editing and deleting authorities of laboratory staff to experimental data are strictly managed. Only authorized personnel can perform specific operations, and the safety and compliance of the data are ensured. The system can be integrated with the automation equipment and the environment monitoring system of a laboratory, and can monitor the running state of the equipment, experimental environment parameters (such as temperature, humidity, air pressure and the like) and report abnormal conditions in time. The module helps laboratory manager to discover equipment failure or environment abnormality in time, and reduces experimental error.
Embodiment 2 provides a laboratory data acquisition management method for one embodiment of the invention, and in order to verify the beneficial effects of the invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.
Firstly, a complete automatic data acquisition system is adopted in a laboratory to acquire, identify, transmit, store, analyze and detect abnormality in real time each item of key data in the experimental process, and trace and feed back adjustment of the abnormality data are realized. At the same time, a temperature sensor, a pressure sensor, a pH sensor, a flowmeter and a spectrum analyzer are respectively arranged in the experimental area, and all the sensors are strictly calibrated to ensure that the output data of the sensors are within a standard range. The IoT gateway is deployed in the environment monitoring system for centrally transmitting each sensor data to the data acquisition terminal. The software system of the acquisition module adopts an embedded control program, acquires real-time data through a fixed period and an event triggering mode, and matches RFID scanning data with sensor data to form a standardized data packet with a time stamp, an equipment ID, a sample ID and other metadata. Meanwhile, in each packet, decision variable parameters such as an ACK latency threshold (t_ack), a maximum number of retransmissions (r_max), the number of partitions in the message queue (n_p), a data inflow rate (λ), and a message processing rate (μ) are attached. The data is then transferred in real time to the data server of the LIMS system via standardized APIs and middleware (e.g., kafka or rabitmq), and optimized using a transfer delay model and a retransmission probability model. In the transmission process, the system utilizes preset weights to minimize the delay of data from the acquisition point to the target server, reduces retransmission probability through multi-target optimization, and ensures that the data verification passing rate meets a set threshold. The uploaded data enters a database management module through a wireless or wired network, and the system stores the data by adopting strategies such as batch writing, transaction management, cache refreshing and the like, so that low writing delay is ensured, and the data consistency meets the preset requirement. In the writing process, the database management part dynamically adjusts parameters such as batch writing quantity, transaction submission overtime, cache refreshing interval, database/table dividing quantity and the like so as to realize the balance of low delay and high consistency. And the LIMS system immediately analyzes and processes the data in real time after the data storage is completed. The system firstly carries out preliminary abnormality detection on various data such as temperature, pH, flow and the like according to preset normal ranges, for example, the temperature is set to be 20-30 ℃, the pH is set to be 6-8, the flow change rate is set to be 1-5L/s, and any data exceeding the ranges is automatically marked as abnormal. Then, for the data marked as abnormal, the system constructs a feature vector, comprehensively considers the deviation of the data from the historical mean value, the change rate in unit time and the duration of the abnormality, classifies the abnormal data by adopting an improved fuzzy C-means clustering algorithm (introducing a time continuity punishment item on the basis of traditional clustering), classifies the abnormality into three categories of mutation abnormality, progressive deviation and single-point abnormality, and ranks the severity of each abnormality in a grading manner. Finally, the system carries out self-adaptive adjustment on the threshold value parameters through a gradient descent method according to the false alarm and false alarm conditions to form a rule feedback loop, so that the whole data acquisition and anomaly detection flow is continuously optimized.
Table 1 test data recording table
It can be seen from table 1 that under different experimental objects, each decision variable parameter and database writing parameter are finely regulated and controlled, so as to verify the remarkable advantages of the scheme in terms of data transmission and writing optimization. Firstly, from the data of T_ack (ACK waiting time threshold), the setting of each experimental object is between 180 milliseconds and 210 milliseconds, which is obviously reduced compared with the common 250 milliseconds of the traditional system, so that the invention realizes the optimization on the data confirmation response speed and can recognize the transmission abnormality more quickly.
In the aspect of R_max (maximum retransmission times), the method is generally set to 2 to 4 times, and compared with the conventional system, the method has the advantages that the retransmission rate is reduced, and meanwhile, the stability and the instantaneity of data transmission are effectively ensured. Looking again at the number of partitions n_p and the data inflow rate λ in the message queue, the data of the experimental object is distributed between 4 to 5 partitions and 50 to 60 data/second, which indicates that the system can reasonably split and process the data, and ensures that delay rise caused by overload of a single processing node is not caused under the condition of high data flow.
Example 3, which is a third example of the present invention, is different from the first two examples in that:
the functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. The storage medium includes a U disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RAM, randomAccessMemory), a magnetic disk, an optical disk, or other various media capable of storing program codes.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium include an electrical connection (an electronic device) having one or more wires, a portable computer diskette (a magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented with any one or combination of discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like, as is known in the art.
Embodiment 4, referring to fig. 2, is a fourth embodiment of the present invention, and the embodiment provides a laboratory sample management system, which includes a data acquisition module, a data transmission module, and a data anomaly detection module.
The data transmission module is used for establishing a transmission delay model and a retransmission probability model, optimizing balance delay and retransmission rate through multiple targets, dynamically adjusting batch writing quantity and a database dividing and table dividing strategy when the data is written into a database, and marking abnormal data based on a preset threshold value and a change rate rule, classifying the abnormal data by adopting an improved fuzzy C-means clustering algorithm and adaptively adjusting a detection threshold value by a random gradient descent method.
It should be noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the technical solution of the present invention, which is intended to be covered by the scope of the claims of the present invention.
Claims (8)
1. A laboratory data acquisition management method, comprising:
Collecting experimental data, associating the data with a unique sample identifier to generate a standardized data packet, and attaching a decision variable;
establishing a transmission delay model and a retransmission probability model, balancing delay and retransmission rate through multi-objective optimization, and dynamically adjusting batch writing quantity and a database and table dividing strategy when data are written into a database;
Marking abnormal data based on a preset threshold value and a change rate rule, classifying the abnormal data by adopting an improved fuzzy C-means clustering algorithm, and adaptively adjusting a detection threshold value by a random gradient descent method;
The step of associating the data with the unique identifiers of the samples comprises the step that when the data packet is generated, a system distributes a global unique identifier for each sample and binds the global unique identifier with the RFID tag;
The data packet metadata includes experiment type, operator ID, and equipment status;
The decision variable is attached to the data packet header;
Decision variables include ACK latency threshold Maximum number of retransmissionsNumber of message queue partitions;
ACK latency thresholdIncluding dynamic calculations based on network state, expressed as:
, wherein, For an average delay of the past 10 minutes,Is the maximum delay;
maximum number of retransmissions Includes setting initial value and upper limit value, if two continuous transmissions failChanging the maximum retransmission frequency upper limit;
Message queue partition count Including dynamic adjustment according to data inflow rate and processing rate, expressed as:
, wherein, Indicating the data inflow rate of the data,Representing a data processing rate;
the establishing of the transmission delay model and the retransmission probability model comprises the steps of defining a transmission delay model and designing a transmission delay objective function;
the transmission delay objective function is expressed as:
, wherein, A delay estimation function is represented and is used,Representing the weight coefficient;
constructing a retransmission probability model and designing a retransmission probability objective function;
Let retransmission probability be The retransmission probability objective function is expressed as:
, wherein, Representing the corresponding weights;
the multi-objective optimization comprises the steps of establishing a multi-objective optimization problem based on data integrity verification and system load limitation, and obtaining an optimal solution of delay and retransmission rate through a non-dominant ordering genetic algorithm.
2. The laboratory data acquisition management method of claim 1, wherein the acquiring experimental data comprises automatically acquiring experimental data using an RFID scanner, a sensor, an IoT device;
the RFID scanner is arranged at a key node of a sample entering and exiting laboratory, and the scanning range covers the whole process of sample movement;
the sensor comprises a temperature sensor, a pressure sensor, a PH sensor, a flowmeter and a spectrum analyzer;
When the sensor is installed, the calibration is carried out according to the layout of an experimental area, the temperature sensor is calibrated at three points in an incubator at a standard temperature of 25 ℃, a calibration coefficient is generated, and the calibration coefficient is stored in the LIMS;
the PH sensor is calibrated by using a standard buffer, and if the deviation between the measured value and the standard value of three continuous times exceeds +/-0.2, an automatic alarm is triggered and data acquisition is suspended;
The loT gateway is deployed at the center of the experimental area, sensor data is collected through Modbus protocol period, if the network delay exceeds 200ms, the network is switched to an event triggering mode, and the data is uploaded when the data variation is more than or equal to 5%.
3. The method for managing laboratory data collection according to claim 2, wherein the dynamically adjusting the batch write amount and the split-library split-table strategy comprises setting the batch write amountInitial value, if write delayGreater than a high threshold of delayAccording toGradually reducing;
If write delay Less than the low delay thresholdAccording toGradually increasing;
the database dividing strategy divides according to experimental types, stores high-frequency data into an SSD database, and stores low-frequency data into an HDD database;
The sub-table strategy generates a new table every month according to the time range division, and establishes a joint index for the sample ID and the time stamp;
setting transaction commit time-outs If the transaction fails, according toRetry;
if the retry is 3 times, the transaction rolls back and logs.
4. The laboratory data acquisition and management method as set forth in claim 3, wherein the labeling of the anomaly data comprises LIMS performing real-time anomaly data detection on the acquired data and performing temperature anomaly determination on the temperature sensor data;
Setting a normal interval for the pH sensor;
calculating the change rate of any sensor data in unit time, and defining preliminary abnormality judgment of the data at unit time;
if any data triggers an anomaly, the marker data point is anomalous.
5. The method for managing laboratory data collection according to claim 4, wherein the anomaly classification comprises classifying the anomaly data by an improved fuzzy C-means clustering algorithm based on characteristics of the anomaly data and constructing a feature vector for each anomaly data point;
the improved fuzzy C-means clustering algorithm comprises the steps of introducing a time continuity penalty term based on a standard fuzzy C-means clustering algorithm to construct an improved objective function;
the self-adaptive adjustment of the detection threshold value comprises updating a threshold value parameter by adopting a random gradient descent method;
if the loss is not continuously reduced, triggering an expert review process.
6. A system adopting the laboratory data acquisition management method as claimed in any one of claims 1 to 5, which is characterized by comprising a data acquisition module, a data transmission module and a data abnormality detection module;
The data acquisition module is used for acquiring experimental data, associating the data with a unique sample identifier to generate a standardized data packet, and attaching a decision variable;
The data transmission module is used for establishing a transmission delay model and a retransmission probability model, balancing delay and retransmission rate through multi-objective optimization, and dynamically adjusting batch writing quantity and a database and table dividing strategy when data are written into a database;
the data anomaly detection module is used for marking anomaly data based on a preset threshold value and a change rate rule, classifying the anomalies by adopting an improved fuzzy C-means clustering algorithm, and self-adaptively adjusting the detection threshold value by a random gradient descent method.
7. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the laboratory data acquisition management method of any one of claims 1 to 5.
8. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the laboratory data acquisition management method of any one of claims 1to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510716638.3A CN120234324B (en) | 2025-05-30 | 2025-05-30 | A laboratory data collection management method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510716638.3A CN120234324B (en) | 2025-05-30 | 2025-05-30 | A laboratory data collection management method, system, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN120234324A CN120234324A (en) | 2025-07-01 |
| CN120234324B true CN120234324B (en) | 2025-09-16 |
Family
ID=96162062
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202510716638.3A Active CN120234324B (en) | 2025-05-30 | 2025-05-30 | A laboratory data collection management method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN120234324B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120873057B (en) * | 2025-07-04 | 2026-02-24 | 华电邹县发电有限公司 | Real-time data acquisition buffering throughput management method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118739184A (en) * | 2024-07-31 | 2024-10-01 | 广东威顺电力工程有限公司 | A method for automatic fault detection and repair of self-healing intelligent power lines |
| CN118884129A (en) * | 2024-08-16 | 2024-11-01 | 广东电网有限责任公司 | Distribution network fault location method and storage medium based on artificial intelligence |
| CN119829626A (en) * | 2024-12-20 | 2025-04-15 | 北京拓扑岭科技有限公司 | LSM engine data cache optimization method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10311442B1 (en) * | 2007-01-22 | 2019-06-04 | Hydrojoule, LLC | Business methods and systems for offering and obtaining research services |
| CN104769582B (en) * | 2012-11-02 | 2018-11-02 | 埃森哲环球服务有限公司 | Real-time data management for the grid |
| CN119322941B (en) * | 2024-12-18 | 2025-05-02 | 国网江西省电力有限公司南昌供电分公司 | Small unmanned aerial vehicle inspection data management method |
| CN119690974B (en) * | 2025-02-21 | 2025-04-18 | 广州市卓航信息科技有限公司 | An artificial intelligence data aggregation method based on big data |
-
2025
- 2025-05-30 CN CN202510716638.3A patent/CN120234324B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118739184A (en) * | 2024-07-31 | 2024-10-01 | 广东威顺电力工程有限公司 | A method for automatic fault detection and repair of self-healing intelligent power lines |
| CN118884129A (en) * | 2024-08-16 | 2024-11-01 | 广东电网有限责任公司 | Distribution network fault location method and storage medium based on artificial intelligence |
| CN119829626A (en) * | 2024-12-20 | 2025-04-15 | 北京拓扑岭科技有限公司 | LSM engine data cache optimization method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN120234324A (en) | 2025-07-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN118094449B (en) | Intelligent production monitoring method, system and device based on industrial Internet | |
| CN119250802B (en) | A method and system for managing oil drilling equipment based on big data analysis | |
| CN120234324B (en) | A laboratory data collection management method, system, equipment and medium | |
| CN118691191B (en) | Digital three-dimensional storage intelligent control method and system | |
| CN118036985A (en) | Agricultural product big data analysis system | |
| CN119941084A (en) | A multi-dimensional logistics information supervision and early warning system and method based on artificial intelligence | |
| CN116907772A (en) | Self-diagnosis and fault source identification method and system of bridge structure monitoring sensor | |
| CN120220318A (en) | A multi-sensor collaborative monitoring method for IoT data early warning | |
| CN118503794A (en) | Transformer substation equipment abnormality detection system and method based on multi-mode data | |
| CN119273364A (en) | A threshold management method and system based on financial risk | |
| CN120896835A (en) | Intelligent Monitoring and Fault Early Warning System for Hospital IoT Devices Based on Edge Computing | |
| CN121032287B (en) | AI-powered intelligent decision-making system for smart factories driven by digital twins | |
| CN120872665A (en) | Computer equipment fault detection system and method based on artificial intelligence | |
| CN120472640A (en) | A monitoring and early warning system and method based on smart door lock | |
| CN120469846A (en) | Hard disk failure detection method and electronic device | |
| CN118133189B (en) | Bridge structure health state real-time monitoring method and system | |
| CN119851973A (en) | Virus propagation prediction method based on neural network | |
| CN119377898A (en) | An accurate modeling method for photovoltaic module faults based on fusion algorithm | |
| CN118230979A (en) | Influenza prediction and monitoring early warning method | |
| CN117633678A (en) | A meter reading problem and cause analysis system and its analysis method | |
| CN120992972B (en) | Medical science measurement node self-learning dynamic calibration system | |
| CN116011305A (en) | Water system fault location method, device and water system fault location system | |
| CN118332356B (en) | Intelligent supervision system and method for improving supervision management efficiency | |
| CN121303385B (en) | Automatic artificial intelligent model system based on self-learning | |
| CN120408505B (en) | Big Data Intelligent Analysis and Processing System Based on Information Fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |