CN118802897B

CN118802897B - A data transmission method, apparatus, electronic device, and storage medium

Info

Publication number: CN118802897B
Application number: CN202410715434.3A
Authority: CN
Inventors: 薛景; 宗伟; 李雨珂; 国黎; 袁湘湘; 冯卫
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Guizhou Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Guizhou Co Ltd
Priority date: 2024-06-04
Filing date: 2024-06-04
Publication date: 2025-11-18
Anticipated expiration: 2044-06-04
Also published as: CN118802897A

Abstract

This disclosure discloses a data transmission method, apparatus, electronic device, and storage medium. Based on message queue configuration parameters, it obtains a first name and a first file record count of the target transmission file, and writes the first name as the key and the first file record count as the value into a first record file. It then obtains a second name and a second file record count of the target transmission file from the target partition and writes them into a second record file. The second record file is read, and the second name as the key and the second file record count as the value are stored in a hash function. Finally, the first record file is read, and the values of the key-value pairs in the first record file are compared with the values of the key-value pairs in the hash function. Based on the comparison result, it determines whether there is an anomaly in the data transmission. This disclosure has simple logic, is easy to implement, has low dependence on networks and terminals, high detection efficiency, can improve the accuracy of data transmission, and solves the problem of data loss during data transmission.

Description

Data transmission method and device, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of data transmission, and in particular relates to a method and a device for auditing data transmission, electronic equipment and a storage medium.

Background

With the development of computer technology, more and more enterprises use Message Queue (MQ) as a core mode of internal data transmission of an IT system of the enterprise. It has a series of functions of low coupling, reliable delivery, broadcasting, flow control, final consistency, etc. Currently more used messages are queued with RabbitMQ, rocketMQ, activeMQ, kafka, zeroMQ, metaMQ, etc.

In the related art data transmission technology, in the process of data transmission, the accuracy of data transmission and whether the data transmission is successful are generally determined through a message fed back by a receiver. Taking Kafka data transfer as an example, when Kafka transfers data, the production end (producer) receives a successful response from the Kafka server (server) when the data is transferred to the master (leader) node and synchronized to all copies. When the production end sends data to the server end, if temporary errors occur, the data cannot be sent, and the production end can resend the data, so that the integrity of the data is guaranteed.

Although the related message queuing techniques provide a reliable guarantee in terms of data loss, in some special scenarios, there may be situations where data is lost. For example, taking data transmission by Kafka as an example, data loss is easily caused when the following occurs:

1. component flow problems Kafka data loss occurs in the upstream or downstream segments.

2. The network problem is that the Kafka rate exceeds the network bandwidth, and once a transmission failure occurs, no retry mechanism is started, data is lost and cannot be retrieved.

3. The data size is problematic, and a single batch of data size is too long, which can cause data loss. The data size produced by the producer is larger than the maximum message size of the pull configured by the consumer, and the pull capability of the consumer is far exceeded, so that the large data can be consumed and fail, and the Kafka can be seriously blocked, so that the data is lost.

4. Hard disk problem the kafka message queue is stored in the form of log (log) file, and when the host disk fails and the data cannot be recovered, the data may be lost.

Under the scene, the data is easy to lose, and the lost data can not be found out in time and the lost data can be timely repaired, so that the problem that the lost data can not be retrieved is caused.

Disclosure of Invention

The disclosure provides an auditing method, device, electronic equipment and storage medium for data transmission. The method and the device mainly aim to solve the problems that data loss easily occurs in the data transmission process, and lost data cannot be found out in time and the lost data cannot be recovered due to timely retransmission.

According to a first aspect of the present disclosure, there is provided a data transmission method, including:

Based on configuration parameters of a message queue, acquiring a first name and a first file record number of a target transmission file, taking the first name as a key, taking the first file record number as a value, writing the first file record number into a first record file, wherein the configuration parameters also comprise a target partition of the target transmission file, the message queue is used for storing the target transmission file into the target partition, and the first file record number is used for recording the sending number of the target transmission file;

acquiring a second name and a second file record number of the target transmission file from the target partition, and writing the second name and the second file record number into a second record file, wherein the second file record number is used for recording the receiving number of the target transmission file;

reading the second record file, taking the second name as a key, taking the second file record number as a value, and storing the second record file in a hash function, wherein the hash function is stored in a key value mode;

and reading the first record file, comparing the value of the key value pair in the first record file with the value of the key value pair in the hash function, and determining whether the transmission of the data is abnormal or not according to the comparison result.

Optionally, before acquiring the first name and the first file record number of the target transmission file based on the configuration parameters of the message queue, the method further includes:

Transferring the file to be transmitted from the input directory to the temporary directory by adopting a contention mode, and adding the file to be transmitted into a message queue;

Reading the file name and the file content of the file to be transmitted from a message queue, and determining the size, the path, the temporary directory and the backup directory of the file to be transmitted as file information;

And splicing file contents corresponding to the file to be transmitted and file information serving as values by taking the file name corresponding to the file to be transmitted as a key, and packaging the file to be transmitted to obtain the target transmission file.

Optionally, after the obtaining the target transmission file, the method further includes:

and transmitting the target transmission files to the target partition based on the target partition in the configuration parameters, and determining the target number of the target transmission files transmitted to the target partition.

Optionally, the obtaining the first name and the first file record number of the target transmission file based on the input end configuration parameter of the message queue, taking the first name as a key, taking the first file record number as a value, and writing the first file record includes:

determining the target number as a first file record number of the target transmission file;

Writing the first name as a key, the first file record number as a value into the first record file;

After receiving a recording signal of the first recording file, determining that the transmission of the target transmission file is completed, wherein the recording signal is used for determining that the writing of the first name and the first file record number is completed.

Optionally, the reading the first record file, comparing the value of the key value pair in the first record file with the value of the key value pair in the hash function, and determining whether the transmission of the data is abnormal according to the comparison result includes:

traversing the first record file, searching the same key as the first record file in the hash function, and determining the same key as a target key in the hash function;

After the target key is determined, judging whether the value of the target key is consistent with the value of the value in the first record file or not;

And determining whether the data transmission is abnormal or not according to the comparison result of the value of the target key and the value of the value in the first record file.

Optionally, the determining whether the data transmission is abnormal according to the comparison result includes:

If the key in the first record file is determined to be not in the hash function, determining the key in the first record file as an abnormal key, and determining the second file record number corresponding to the abnormal key as 0;

If the value of the target key is consistent with the value of the value in the first record file, determining that the data transmission is normal;

If the value of the target key is inconsistent with the value of the value in the first record file, determining that the data transmission is abnormal;

Determining a target transmission file corresponding to the abnormal key and the target key with abnormal data transmission as an abnormal file;

And outputting a first name, a first file record number and a second file record number corresponding to the abnormal file.

Optionally, the method further comprises:

And after the abnormal file is determined, acquiring a backup file corresponding to the abnormal file, and retransmitting the backup file.

According to a second aspect of the present disclosure, there is provided a data transmission apparatus including:

The first writing unit is used for acquiring a first name and a first file record number of a target transmission file based on configuration parameters of a message queue, taking the first name as a key, taking the first file record number as a value, writing the first record file, wherein the configuration parameters also comprise a target partition of the target transmission file, the message queue is used for storing the target transmission file into the target partition, and the first file record number is used for recording the sending number of the target transmission file;

The second writing unit is used for acquiring a second name and a second file record number of the target transmission file from the target partition, writing the second name and the second file record number into a second record file, and recording the receiving number of the target transmission file by the second file record number;

the storage unit is used for reading the second record file, taking the second name as a key, taking the second file record number as a value, and storing the second record file in a hash function, wherein the hash function is stored in a key value opposite mode;

And the comparison unit is used for reading the first record file, comparing the value of the key value pair in the first record file with the value of the key value pair in the hash function, and determining whether the transmission of the data is abnormal or not according to the comparison result.

Optionally, the apparatus further comprises a packaging unit, the packaging unit comprising:

the adding module is used for transferring the file to be transmitted from the input catalogue to the temporary catalogue in a contention mode before acquiring the first name and the first file record number of the target transmission file based on the configuration parameters of the message queue, and adding the file to be transmitted into the message queue;

The reading module is used for reading the file name and the file content of the file to be transmitted from the message queue and determining the size, the path, the temporary directory and the backup directory of the file to be transmitted as file information;

And the packaging module is used for packaging the file to be transmitted in a way of splicing the file content corresponding to the file to be transmitted and the file information serving as values by taking the file name corresponding to the file to be transmitted as a key to obtain the target transmission file.

Optionally, the apparatus further comprises a sending unit, where the sending unit is configured to:

after the target transmission file is obtained, the target transmission file is sent to the target partition based on the target partition in the configuration parameters, and the target number of the target transmission files sent to the target partition is determined.

Optionally, the first writing unit includes:

the first determining module is used for determining the target quantity as a first file record number of the target transmission file;

The writing module is used for writing the first name as a key, the first file record number as a value and the first record file;

and the second determining module is used for determining that the transmission of the target transmission file is completed after receiving a recording signal of the first recording file, wherein the recording signal is used for determining that the writing of the first name and the first file record number is completed.

Optionally, the alignment unit includes:

The searching module is used for traversing the first record file, searching the same key as the first record file in the hash function, and determining the same key as a target key in the hash function;

the judging module is used for judging whether the value of the target key is consistent with the value of the value in the first record file after the target key is determined;

And the third determining module is used for determining whether the data transmission is abnormal or not according to the comparison result of the value of the target key and the value of the value in the first record file.

Optionally, the alignment unit further includes:

a fourth determining module, configured to determine, if it is determined that the key in the first record file does not exist in the hash function, the key in the first record file as an abnormal key, and determine a second file record number corresponding to the abnormal key as 0;

A fifth determining module, configured to determine that data transmission is normal if it is determined that the value of the target key is consistent with the value of the value in the first record file;

A sixth determining module, configured to determine that data transmission is abnormal if it is determined that the value of the target key is inconsistent with the value of the value in the first record file;

A seventh determining module, configured to determine, as an abnormal file, a target transmission file corresponding to the abnormal key and a target key with abnormal data transmission;

And the output module is used for outputting the first name, the first file record number and the second file record number corresponding to the abnormal file.

Optionally, the apparatus further comprises a retransmission unit for:

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

At least one processor, and

A memory communicatively coupled to the at least one processor, wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of the preceding first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of the first aspect described above.

The data transmission method, the data transmission device, the electronic equipment and the storage medium are based on configuration parameters of a message queue, the first name and the first file record number of a target transmission file are obtained, the first name is used as a key, the first file record number is used as a value, the first record file is written in, the configuration parameters further comprise a target partition of the target transmission file, the message queue is used for storing the target transmission file into the target partition, the first file record number is used for recording the sending number of the target transmission file, the second name and the second file record number of the target transmission file are obtained from the target partition, the second name and the second file record number are written in the second record file, the second file record number is used for recording the receiving number of the target transmission file, the second record file is read, the second name is used as a key, the second file record number is used as a value, the second file record number is stored in a hash function, the hash function is stored in the key value, the hash function is read, and whether the hash function has abnormal data compared with the hash function in the hash function according to the hash value. According to the data transmission method, the number of file records before and after data transmission is recorded through the first recorded file and the second recorded file, and the values of the key value pairs are compared, so that the logic is simple and easy to realize, the dependence on a network and a terminal is low, the detection efficiency is high, lost data in the data transmission process can be timely found, the data can be uploaded again, the accuracy of the data transmission is improved, and the problem that the data loss is easy to occur in the data transmission process is solved. It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the application or to delineate the scope of the application. Other features of the present application will become apparent from the description that follows.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a flow chart of a data transmission method according to an embodiment of the disclosure;

Fig. 2 is a schematic diagram illustrating detection of a data transmission method according to an embodiment of the disclosure;

fig. 3 is a flowchart illustrating another data transmission method according to an embodiment of the disclosure;

Fig. 4 is a flowchart of another data transmission method according to an embodiment of the disclosure;

fig. 5 is a schematic structural diagram of a data transmission device according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of another data transmission device according to an embodiment of the disclosure;

fig. 7 is a schematic block diagram of an example electronic device 400 provided by an embodiment of the disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Data transmission methods and apparatuses, electronic devices, and storage media according to embodiments of the present disclosure are described below with reference to the accompanying drawings.

Fig. 1 is a flow chart of a data transmission method according to an embodiment of the disclosure.

As shown in fig. 1, the method comprises the steps of:

Step 101, based on configuration parameters of a message queue, obtaining a first name and a first file record number of a target transmission file, using the first name as a key, using the first file record number as a value, writing the first file record, wherein the configuration parameters also comprise a target partition of the target transmission file, the message queue is used for storing the target transmission file into the target partition, and the first file record number is used for recording the sending number of the target transmission file.

In the embodiment of the application, the message queue is a middleware of a queue structure, and data can be directly returned after being put into the middleware, and the system is not required to process the data immediately, and a program can read the data and sequentially process the data. The message queue mainly solves the problems of application coupling, asynchronous processing, flow cutting and the like. Currently more used message queues have RabbitMQ, rocketMQ, activeMQ, kafka, zeroMQ, metaMq, etc., while partial databases such as Redis, mysql, and phxsql may also implement the message queue functions.

In order to detect whether the data transmission is normal, the first name is used as a key, the first file record number is used as a value, and the first record file is written in, so that the first file record number is used as a comparison basis for detection, and whether the data transmission is successful is determined. The configuration parameters of the message queue comprise a first name of a target transmission file, a first file record number and a target partition, wherein the first name and the first file record number are used for checking data transmission, and the first file record number comprises the number of analysis anomalies, the number of successful transmission, the number of failed transmission and the number of transmission to the target partition of the target transmission file. The first record file is a file with a suffix name of stat, and is used for recording the first name and the first file record number in a key value pair mode as a detection basis of data transmission, and is also used for recording other information of the target transmission file, such as the size, the storage path, the storage time and the like of the target transmission file.

In order to better understand the first record file, the embodiment of the present application provides an example of file content of the first record file, for example:

the first record file content with file name in_stat_20221020 is:

Fi leName:gprs_ol_30200_B_D08_452718909.txt;FileSize:2562235;Earlist:20221023;latest:20221023;ExtALines:0;TotalLines:1512;SendLines:1512;NonTimeo utExceptionLines:0;TimeoutExceptionLines:0;StartTime:20221023112804;EndTim e:20221023112805;Speed:2119;TopicList:{CloudXdrTopicA0.1:1512;};BillDateList:{20221023:1512;};TargetLines:0

From the first record file of the above example, information about the target transmission file may be obtained, for example Fi leName:gprs_ol_30200_b_d08_452718909.Txt is the first name of the target transmission file, fi leSize:2562235 is the size of the target transmission file, totalLines:1512 is the total number of the target transmission files, SENDLINES:1512 is the number of target partitions to be transmitted to the target transmission file, and the like.

Step 102, obtaining a second name and a second file record number of the target transmission file from the target partition, and writing the second name and the second file record number into a second record file, where the second file record number is used to record the receiving number of the target transmission file.

In order to check whether the condition of losing data occurs in the data transmission process, after the target transmission file is acquired, the second name and the second file record number of the target transmission file are written into a second record file, the second file record number is used for recording the receiving number of the target transmission file, and the second record file is used for recording the second name, the second file record number and related information of the received target transmission file so as to be convenient for comparison with the first record file and searching the target transmission file with the data loss.

And 103, reading the second record file, taking the second name as a key, taking the second file record number as a value, and storing the second record file in a hash function, wherein the hash function is stored in a key value opposite mode.

In order to improve the searching efficiency, the application adopts a hash comparison method in a Perl language, wherein the Perl language is a computer program language with rich functions, is operated on more than 100 computer platforms, has wide application, is originally developed for text processing, and is now used for various tasks including system management, web development, network programming, GUI development and the like. In the Perl programming language, a hash (hash) data structure may be used to store key-value pairs (keys-value pairs). The hash function is a mapping relation, and a function of the element storage position is calculated according to a key of data and a certain function relation. The hash is to temporarily store data in a memory in a key value pair mode, directly search a previously written hash key value pair (key value pair in a first record file in the application) in the hash through keys (keys can be numerical values or characters or similar indexes), search a previously written value and compare a current value through keys, and can efficiently realize the comparison of two groups of numerical values and improve the search efficiency.

And 104, reading the first record file, comparing the value of the key value pair in the first record file with the value of the key value pair in the hash function, and determining whether the transmission of the data is abnormal or not according to the comparison result.

In the embodiment of the application, after the second record file is stored in a hash function, the first record file is read, the first record file is traversed one by one, the same key as the first record file is searched in the hash function, if the key in the first record file is determined not to exist in the hash function, the key in the first record file is determined to be an abnormal key, the second file record number corresponding to the abnormal key is determined to be 0, if the key in the first record file is determined to exist in the hash function, the same key in the hash function is determined to be a target key, after the target key is determined, whether the value of the target key is consistent with the value in the first record file or not is judged, if the value of the target key is determined to be consistent with the value in the first record file, the data transmission is determined to be normal, if the value of the target key is determined not to be consistent with the value in the first record file, the data transmission is determined to be abnormal, the key and the first record file corresponding to the abnormal file is determined to be the abnormal file, and the first record file corresponding to the abnormal file is output.

In order to better understand the first record file and the second record file, the embodiment of the application further provides a detection schematic diagram of a data transmission method, as shown in fig. 2, the first record file and the second record file are read respectively, then the same key in the two record files is searched, after the same key is determined, the values corresponding to the keys are compared, if the values of the values are consistent, data transmission is finished, if the values are inconsistent, related information of the abnormal files is output, the related information comprises a first name of the abnormal file, a first file record number and a second file record number, so that data lost in the transmission process of the abnormal files can be determined conveniently, and after the related information of the abnormal files is output, data transmission is finished.

The method has the advantages that the related information such as the first file record number of the target transmission file is recorded by the first record file before data transmission, the second transmission information of the target transmission file is recorded by the second record file after data transmission is generated, the data transmission is detected only by comparing the first record file with the second record file, the data transmission process is not detected, the logic is simple and easy to realize, the efficient detection is realized by Ha Xibi pairs, and the abnormal file in the data transmission is quickly searched.

The data transmission method includes the steps of obtaining a first name and a first file record number of a target transmission file based on configuration parameters of a message queue, taking the first name as a key, taking the first file record number as a value, writing the first record file, taking the second name as a key, taking the second file record number as a value, storing the target transmission file in a hash function, storing the first file record number in the target partition by the message queue, obtaining a second name and a second file record number of the target transmission file from the target partition, writing the second name and the second file record number into the second record file, taking the second name as a key, taking the second file record number as a value, storing the hash function in the hash function, reading the first record file, comparing the hash function with the hash function, and determining whether abnormal data exist in the hash function according to the comparison result of the hash value of the first record file. Compared with the method that whether the data transmission is successful or not is determined by the information fed back by the receiving party in the data transmission process of other related technologies, the data transmission method provided by the disclosure records the file record numbers before and after the data transmission by the first record file and the second record file, and compares the values of key value pairs, has simple logic and easy realization, lower dependence on a network and a terminal, high detection efficiency, can realize timely finding out lost data in the data transmission process, re-upload the lost data, improves the accuracy of the data transmission, and solves the problem that the data loss is easy to occur in the data transmission process.

In an embodiment of the present application, the data transmission method further includes a data transmission encapsulation process and a data transmission sending process. In order to better understand the packaging flow in the data transmission method, fig. 3 is a flow diagram of another data transmission method according to an embodiment of the disclosure.

As shown in fig. 3, the method further comprises the steps of:

And step 201, transferring the file to be transmitted from the input directory to the temporary directory in a contention mode, and adding the file to be transmitted to a message queue.

The said competing mode, also called Work mode, the producer in the message queue sends the message to the exchanger, the exchanger sends the message to the message queue according to the attribute of the message, multiple consumers bind and monitor a message queue at the same time, form the effect of competing the message.

Step 202, reading the file name and the file content of the file to be transmitted from the message queue, and determining the size, the path, the temporary directory and the backup directory of the file to be transmitted as file information.

Packaging files refers to combining related resources such as software, files or data together and packaging the same into a single file or archiving. This process consolidates multiple files or resources into a single entity to facilitate distribution, transmission, or deployment. In order to facilitate the transmission of the target transmission file and in order to obtain information about the writing of the target transmission file into the first record file, file information is obtained from the message queue.

And 203, splicing file contents corresponding to the file to be transmitted and file information serving as values by taking the file name corresponding to the file to be transmitted as a key, and packaging the file to be transmitted to obtain the target transmission file.

In order to acquire the target transmission file, a mode of splicing file contents and file information is used as a value, and a file name is used as a key to be packaged into a FileInfo-class file which is used as the target transmission file. The FileInfo class is a class located in the System. IO namespace that provides a range of properties and methods for manipulating files. Through FileInfo classes, we can easily obtain information about the file, such as file size, creation date, modification date, etc. In addition, fileInfo classes also provide some convenient methods for copying, moving, deleting, etc. files. For example, a Producer Record component (Producer Record) is used in the kafka cluster to encapsulate the file to be transferred into a target transfer file such as Producer Record (cloudxdrdtopica 0.1, filename, text@info).

By the packaging method, related information written in the first record file is obtained simultaneously in the process of packaging the file to be transmitted, and the file to be transmitted is packaged, so that the subsequent transmission of the target transmission file is facilitated.

In some embodiments, after the file to be transmitted is encapsulated and the target transmission file is obtained, the encapsulated target transmission file (Fi leInfo types of files in the above embodiments) is sent to different partitions of different topics of the message middleware (kafka cluster in the above embodiments) in a load balancing manner, and meanwhile, the target number of the target transmission files sent to the target partition is determined. The Load balancing (LB for short) is an essential key component of a high-concurrency and high-availability system, and aims to evenly distribute network traffic to a plurality of servers so as to improve the response speed and availability of the whole system. For example, if there are n partitions and n target transport files, each target transport file is sent to a different partition to achieve load balancing.

In some embodiments, after the target transmission file is sent to the target partition, waiting for a recording signal sent by the first recording file, determining that the sending of the target transmission file is completed after receiving the recording signal of the first recording file, otherwise, waiting until the recording signal is received, and continuing to send the next target transmission file, wherein the recording signal is used for determining that the successful writing of the first name and the first file record number is completed, so that omission is avoided.

In order to better understand the data transmission process, taking the kafka cluster as an example, the embodiment of the application further provides another flow chart of data transmission, as shown in fig. 4, the packaging and backup of the file to be transmitted are completed when the file to be transmitted is input, meanwhile, a first record file is generated, and when the target transmission file is output, a corresponding second transmission file is generated.

In some embodiments, after the abnormal file is determined by comparing the first record file and the second record file, a backup file corresponding to the abnormal file is obtained, and the backup file is retransmitted, so that accuracy of data transmission is improved, and occurrence of a data loss condition is avoided.

By packaging the file to be transmitted and transmitting the backup method of the abnormal file after determining the abnormal file, the accuracy of data transmission is improved, and the probability of data loss is reduced.

In summary, the embodiments of the present disclosure can achieve the following effects:

1. The method has the advantages that the related information such as the first file record number of the target transmission file is recorded by the first record file before data transmission, the second transmission information of the target transmission file is recorded by the second record file after data transmission is generated, the data transmission is detected only by comparing the first record file with the second record file, the data transmission process is not detected, the logic is simple and easy to realize, the efficient detection is realized by Ha Xibi pairs, and the abnormal file in the data transmission is quickly searched.

2. By packaging the file to be transmitted and transmitting the backup method of the abnormal file after determining the abnormal file, the accuracy of data transmission is improved, and the probability of data loss is reduced.

Corresponding to the data transmission method, the invention also provides a data transmission device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment may refer to the above-mentioned method embodiment, and details are not described in detail in the present invention.

Fig. 5 is a schematic structural diagram of a data transmission device according to an embodiment of the present disclosure, and as shown in fig. 5, the data transmission device includes a first writing unit 31, a second writing unit 32, a storage unit 33, and a comparison unit 34.

A first writing unit 31, configured to obtain a first name and a first file record number of a target transmission file based on a configuration parameter of a message queue, and write the first name as a key, the first file record number as a value into a first record file, where the configuration parameter further includes a target partition of the target transmission file, the message queue is configured to store the target transmission file into the target partition, and the first file record number is configured to record a transmission number of the target transmission file;

A second writing unit 32, configured to obtain a second name and a second file record number of the target transmission file from the target partition, and write the second name and the second file record number into a second record file, where the second file record number is used to record a received number of the target transmission file;

a storage unit 33, configured to read the second record file, store the second record file in a hash function with the second name as a key and the second record number as a value, where the hash function is stored in a key value mode;

And the comparison unit 34 is configured to read the first record file, compare the value of the key value pair in the first record file with the value of the key value pair in the hash function, and determine whether there is an abnormality in data transmission according to the comparison result.

The data transmission method and device provided by the disclosure are based on configuration parameters of a message queue, a first name and a first file record number of a target transmission file are obtained, the first name is used as a key, the first file record number is used as a value, the first record file is written in, the configuration parameters further comprise a target partition of the target transmission file, the message queue is used for storing the target transmission file into the target partition, the first file record number is used for recording the sending number of the target transmission file, a second name and a second file record number of the target transmission file are obtained from the target partition, the second name and the second file record number are written in a second record file, the second file record number is used for recording the receiving number of the target transmission file, the second record file is used as a key, the second file is stored in a hash function with the second file record number as a value, the hash function is stored in the hash function with the key value, whether the comparison result of the hash function exists is determined according to whether the comparison result of the hash function exists in the comparison value of the first record file. Compared with the method that whether the data transmission is successful or not is determined by the information fed back by the receiving party in the data transmission process of other related technologies, the data transmission method provided by the disclosure records the file record numbers before and after the data transmission by the first record file and the second record file, and compares the values of key value pairs, has simple logic and easy realization, lower dependence on a network and a terminal, high detection efficiency, can realize timely finding out lost data in the data transmission process, re-upload the lost data, improves the accuracy of the data transmission, and solves the problem that the data loss is easy to occur in the data transmission process. Further, in a possible implementation manner of this embodiment, as shown in fig. 6, the apparatus further includes an encapsulation unit 35, where the encapsulation unit includes:

The adding module 351 is configured to transfer the file to be transmitted from the input directory to the temporary directory in a contention mode before acquiring the first name and the first file record number of the target transmission file based on the configuration parameters of the message queue, and add the file to be transmitted to the message queue;

The reading module 352 is configured to read a file name and a file content of the file to be transmitted from the message queue, and determine a size, a path, a temporary directory and a backup directory of the file to be transmitted as file information;

And the packaging module 353 is configured to package the file to be transmitted by using the file name corresponding to the file to be transmitted as a key and splicing the file content corresponding to the file to be transmitted and the file information as values, so as to obtain the target transmission file.

Further, in a possible implementation manner of this embodiment, as shown in fig. 6, the apparatus further includes a sending unit, and the sending unit 36 is configured to:

Further, in one possible implementation manner of this embodiment, as shown in fig. 6, the first writing unit 31 includes:

a first determining module 311, configured to determine the target number as a first file record number of the target transmission file;

a writing module 312, configured to write the first record file with the first name as a key and the first file record number as a value;

the second determining module 313 is configured to determine that the transmission of the target transmission file is completed after receiving a recording signal of the first recording file, where the recording signal is used to determine that the writing of the first name and the first file record number is completed.

Further, in one possible implementation manner of this embodiment, as shown in fig. 6, the comparing unit 34 includes:

The searching module 341 is configured to traverse the first record file, search the hash function for the same key as the first record file, and determine the same key as the target key in the hash function;

a judging module 342, configured to determine whether the value of the target key is consistent with the value of the value in the first record file after determining the target key;

the third determining module 343 is configured to determine whether there is an abnormality in data transmission according to the comparison result of the value of the target key and the value in the first record file.

Further, in one possible implementation manner of this embodiment, as shown in fig. 5, the comparing unit 34 further includes:

a fourth determining module 344, configured to determine, if it is determined that the key in the first record file does not exist in the hash function, the key in the first record file as an abnormal key, and determine a second file record number corresponding to the abnormal key as 0;

A fifth determining module 345, configured to determine that the data transmission is normal if it is determined that the value of the target key is consistent with the value of the value in the first record file;

A sixth determining module 346, configured to determine that the data transmission is abnormal if it is determined that the value of the target key is inconsistent with the value of the value in the first record file;

a seventh determining module 347, configured to determine, as an abnormal file, a target transmission file corresponding to the abnormal key and a target key with abnormal data transmission;

The output module 348 is configured to output a first name, a first file record number, and a second file record number corresponding to the abnormal file.

Further, in a possible implementation manner of this embodiment, as shown in fig. 6, the apparatus further includes a retransmission unit 37, configured to:

The foregoing explanation of the method embodiment is also applicable to the apparatus of this embodiment, and the principle is the same, and this embodiment is not limited thereto.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 7 shows a schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the apparatus 400 includes a computing unit 401 that can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 402 or a computer program loaded from a storage unit 408 into a RAM (Random Access Memory ) 403. In RAM 403, various programs and data required for the operation of device 400 may also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other by a bus 404. An I/O (Input/Output) interface 405 is also connected to bus 404.

The various components in the device 400 are connected to an I/O interface 405, including an input unit 406, e.g., keyboard, mouse, etc., an output unit 407, e.g., various types of displays, speakers, etc., a storage unit 408, e.g., magnetic disk, optical disk, etc., and a communication unit 409, e.g., network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The computing unit 401 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a CPU (Central Processing Unit ), a GPU (Graphic Processing Units, graphics processing unit), various specialized AI (ARTIFICIAL INTELL IGENCE ) computing chips, various computing units running machine learning model algorithms, a DSP (DIGITAL SIGNAL Processor ), and any suitable Processor, controller, microcontroller, etc. The computing unit 401 performs the respective methods and processes described above, for example, a transmission method of data. For example, in some embodiments, the method of transmitting data may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 400 via the ROM 402 and/or the communication unit 409. When the computer program is loaded into RAM 403 and executed by computing unit 401, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned data transmission method in any other suitable way (e.g. by means of firmware).

Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated Circuit systems, FPGAs (Field Programmable GATE ARRAY, field programmable gate arrays), ASICs (Appl ication-SPECIFIC INTEGRATED circuits, application specific integrated circuits), ASSPs (Appl ication SPECIFIC STANDARD products ), SOCs (systems On Chip systems), CPLDs (Complex Programmable Logic Device, complex programmable logic devices), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be a special or general purpose programmable processor, operable to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, RAM, ROM, EPROM (ELECTRICALLY PROGRAMMABLE READ-Only-Memory, erasable programmable read-Only Memory) or flash Memory, an optical fiber, a CD-ROM (Compact Di sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid CRYSTAL DISPLAY) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include LAN (Local Area Network ), WAN (Wide Area Network, wide area network), the Internet, and blockchain networks.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual PRIVATE SERVER" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.

It should be noted that, artificial intelligence is a subject of studying a certain thought process and intelligent behavior (such as learning, reasoning, thinking, planning, etc.) of a computer to simulate a person, and has a technology at both hardware and software level. The artificial intelligence hardware technology generally comprises technologies such as a sensor, a special artificial intelligence chip, cloud computing, distributed storage, big data processing and the like, and the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, a machine learning/deep learning technology, a big data processing technology, a knowledge graph technology and the like.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A method of transmitting data, comprising:

2. The method of claim 1, wherein prior to obtaining the first name and the first number of file records of the target transmission file based on the configuration parameters of the message queue, the method further comprises:

3. The method of claim 2, wherein after the obtaining the target transmission file, the method further comprises:

4. The method of claim 3, wherein the obtaining the first name and the first record number of the target transmission file based on the input configuration parameter of the message queue, and writing the first record file with the first name as a key and the first record number as a value includes:

5. The method of claim 1, wherein the reading the first record file, comparing the value of the key pair in the first record file with the value of the key pair in the hash function, and determining whether there is an abnormality in the transmission of the data according to the comparison result, comprises:

6. The method of claim 5, wherein determining whether there is an anomaly in the data transmission based on the comparison result comprises:

7. The method of claim 6, wherein the method further comprises:

8. A method of transmitting data, comprising:

9. An electronic device, comprising:

At least one processor, and

A memory communicatively coupled to the at least one processor, wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.