Summary of the invention
The present invention provides a kind of bill processing method, device and computer readable storage medium, main purpose and is reality
Show reconciliation performance to stablize, as the data volume of transaction is increasing, reconciliation and export performance will not significantly decrease;
Scalability is strong.
To achieve the above object, the present invention also provides a kind of bill processing methods, which comprises
The transaction request for obtaining ordering system, is stored in order system for the relevant transaction data of the transaction request of ordering system
In the database of system;
The transaction data that channel side is read from the file server of channel side, is stored in order for the transaction data of channel side
On the network attached storage NAS disk of system;
The transaction data of ordering system and the transaction data of channel side are called, by the transaction data and channel side of ordering system
Transaction data be compared;
When the transaction data of the transaction data of ordering system and channel side is compared successfully, by the NAS of ordering system
File in disk uploads in the file server of channel side.
Optionally, the relevant transaction data of the transaction request by ordering system is stored in the database of ordering system
Include:
According to sliding window strategy, start timed task in each time window, by the ordering system in time window
Transaction data fragment batch write-in Hadoop distributed file system, according to order application odd numbers Hash codes hashcode into
Row fragment.
Optionally, the transaction data for calling ordering system and the transaction data of channel side include:
Ordering system calls the Driver of Spark cluster, starts Spark task schedule, each task task of Spark from
The transaction data of channel side is read in Spark node on NAS disk, wherein the transaction data of channel side includes channel source mark
Know;
And based on channel source identification, according in the transaction data of channel side order application odd numbers Hash codes
Hashcode carries out repartition processing, and Spark node reads in ordering system from Hadoop distributed file system after having handled
Transaction data, read the corresponding fragment of task task number ID of Spark respectively.
Optionally, described when the transaction data of the transaction data of ordering system and channel side is compared successfully, it will order
File in the NAS disk of single system uploads in the file server of channel side, comprising: when the transaction data and canal of ordering system
When the transaction data of road side is compared successfully, the Map object of ordering system is changed into String object and is written to HDFS by Spark
System;
After the completion of all fragment write-ins, the API of Hadoop distributed file system is called, the file of each distribution is closed
And and upload to the NAS disk of ordering system, the file under the NAS disk is uploaded to the sftp server of channel side by ordering system
On.
Optionally, when the transaction data of the transaction data of ordering system and channel side is compared unsuccessful, reception pair
The processing of problem data, and re-start reconciliation.
Optionally, the Hadoop distributed file system be configured with fault-tolerant architecture, wherein fault-tolerant architecture by metadata with
Timestamp composition, same order number, the big person of timestamp are valid data.
To achieve the above object, the present invention also provides a kind of bill processing unit, described device includes memory and processing
Device is stored with the bill processing routine that can be run on the processor on the memory, and the bill processing routine is by institute
It states when processor executes and realizes following steps:
The transaction request for obtaining ordering system, is stored in order system for the relevant transaction data of the transaction request of ordering system
In the database of system;
The transaction data that channel side is read from the file server of channel side, is stored in order for the transaction data of channel side
On the network attached storage NAS disk of system;
The transaction data of ordering system and the transaction data of channel side are called, by the transaction data and channel side of ordering system
Transaction data be compared;
When the transaction data of the transaction data of ordering system and channel side is compared successfully, by the NAS of ordering system
File in disk upload in the file server of channel side when the transaction data of ordering system and the transaction data of channel side into
When row is compared successfully, the file in the NAS disk of ordering system is uploaded in the file server of channel side.
Optionally, the bill processing routine can also be executed by the processor, also realization following steps:
According to sliding window strategy, start timed task in each time window, by the ordering system in time window
Transaction data fragment batch write-in Hadoop distributed file system (HDFS) distributed file system, according to the request slip of order
Number Hash codes hashcode carry out fragment.
Optionally, the bill processing routine can also be executed by the processor, realize following steps:
Ordering system calls the Driver of Spark cluster, starts Spark task schedule, each task task of Spark from
The transaction data of channel side is read in Spark node on NAS disk, wherein the transaction data of channel side includes channel source mark
Know;
And based on channel source identification, according in the transaction data of channel side order application odd numbers Hash codes
Hashcode carries out repartition processing, and Spark node reads in ordering system from Hadoop distributed file system after having handled
Transaction data, read the corresponding fragment of task task number ID of Spark respectively.
In addition, to achieve the above object, it is described computer-readable the present invention also provides a kind of computer readable storage medium
Bill processing routine is stored on storage medium, the bill processing routine can be executed by one or more processor, with reality
Now the step of bill processing method as described above.
The present invention obtains the transaction request of ordering system, and the relevant transaction data of the transaction request of ordering system is stored in
In the database of ordering system, the transaction data of channel side is read from the file server of channel side, by the number of deals of channel side
According on the NAS disk for being stored in ordering system, the transaction data of ordering system and the transaction data of channel side are called, by ordering system
Transaction data and the transaction data of channel side be compared, when ordering system transaction data and channel side transaction data into
When row is compared successfully, the file in the NAS disk of ordering system is uploaded in the file server of channel side.Reconciliation of the present invention
It can stablize, as the data volume of transaction is increasing, reconciliation and export performance will not significantly decrease;Scalability
It is strong: performance can be improved by way of increasing machine node;Reduce system pressure: thought handled using class sliding window formula,
Reduce reconciliation and the processing of the export pressure caused by database;Business is fault-tolerant: being directed to Hadoop distributed file system
Only newly-increased and additional feature designs fault-tolerant architecture, avoids data modification that big performance is brought to be lost.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
The present invention provides a kind of bill processing method.Shown in referring to Fig.1, handled for the bill that one embodiment of the invention provides
The flow diagram of method.This method can be executed by a device, which can be by software and or hardware realization.
In the present embodiment, bill processing method includes:
S10, the transaction request for obtaining ordering system, the relevant transaction data of the transaction request of ordering system is stored in and is ordered
In the database of single system.
In the present embodiment, channel side sends real-time deal request and is given to ordering system, and ordering system in real time asks transaction
Relevant transaction data is asked to be stored in database.
Preferably, the relevant transaction data of the transaction request by ordering system is stored in the database of ordering system
Include:
According to sliding window strategy, start timed task in each time window, by the ordering system in time window
Transaction data fragment batch write-in Hadoop distributed file system (HDFS), according to the Hash codes of the application odd numbers of order
Hashcode carries out fragment.Such as filename starts with 0, step-length 1 is incremented by.
S20, the transaction data that channel side is read from the file server of channel side, the transaction data of channel side is stored
In on the NAS disk of ordering system.
In the present embodiment, periodically (such as daily) number of deals for reading channel side from the file server of channel side
According to the transaction data of channel side is stored on the network attached storage NAS disk of ordering system.The temporal frequency periodically read can
With configured in advance.
Wherein, fragment is carried out according to the Hash codes hashcode of the application odd numbers of the order of channel side.Such as filename is with 0
Start, step-length 1, is incremented by.
The transaction data of S30, the transaction data for calling ordering system and channel side, by the transaction data and canal of ordering system
The transaction data of road side is compared.
Wherein with the transaction data of the architectural form management ordering system of cluster and the transaction data of channel side.Spark is answered
It, with the operation of independent process collection, is adjusted on cluster with SparkContext object in main program (referred to as driver)
Section.Particularly, in order to run on cluster, SparkContext can be with cluster manager dual system (Spark itself list of several types
Only cluster manager dual system or Mesos/YARN) it is connected, these cluster manager dual systems can distribute resource between application.Once even
Connect, Spark needs thread pool child node on cluster, that is, those execute calculate and storage using data work into
Journey.Then, it will send your application code (with JAR or the Python file defined and being transmitted to SparkContext)
To thread pool.Thread pool is allowed to run finally, SparkContext sends task.
Each application has the thread pool process of oneself, can keep in the operational process entirely applied and in multiple threads
Operation task.The advantage of doing so is that application is mutually isolated, both in scheduling aspect (task of each driving scheduling own)
Also in terms of execution (task of different application is run on different JVM).
For potential cluster manager dual system, Spark is unknowable.As long as it need thread pool process and they
Between communication, even when be run on the cluster manager dual system (for example, Mesos/YARN) for also supporting other application it is also opposite
Simply.
Because of driving scheduler task on cluster, it should run and be close to working node, in identical local area network more
It is good.
Preferably, the transaction data for calling ordering system and the transaction data of channel side include:
Ordering system calls the Driver of Spark cluster, starts Spark task schedule, each task task of Spark from
The transaction data of channel side is read in Spark node on NAS disk, wherein the transaction data of channel side includes channel source mark
Know;And based on channel source identification, according in the transaction data of channel side order application odd numbers Hash codes hashcode into
Row repartition step process, Spark node reads in the transaction data of ordering system from HDFS system after having handled, and reads respectively
To the corresponding fragment of task task number ID of Spark.
Preferably, after by processing step above, there is the transaction of ordering system in each fragment in Spark
The transaction data of data and channel side, each task task start to compare the transaction of the transaction data and channel side to ordering system
Data processing.
S40, when the transaction data of the transaction data of ordering system and channel side is compared successfully, by ordering system
File in NAS disk uploads in the file server of channel side.
Preferably, it when the transaction data of the transaction data of ordering system and channel side is compared successfully, that is, orders
When the transaction data of single system and the completely the same transaction data of channel side, Spark changes into the Map object of ordering system
String object is written to HDFS system;After the completion of all fragment write-ins, the API of HDFS system is called, by the text of each distribution
Part merges, and uploads to the NAS disk of ordering system, and ordering system takes the sftp that the file under the NAS disk uploads to channel side
It is engaged on device.
When the transaction data of the transaction data of ordering system and channel side is compared unsuccessful, that is to say, that reconciliation has
Problem then needs artificial treatment data, after the completion of data processing, repeats above step reconciliation again secondary.
It is therefore preferred that being connect when the transaction data of the transaction data of ordering system and channel side is compared unsuccessful
The processing to problem data is received, and re-starts reconciliation.
Preferably, the HDFS system configuration has fault-tolerant architecture, and wherein fault-tolerant architecture is made of metadata and timestamp.Phase
With order number, the big person of timestamp is valid data.Because may be modified for data on HDFS, while HDFS does not support to repair
The problem of changing is related to a fault-tolerant architecture, avoids importing database data again because of data modification.Fault-tolerant architecture is by metadata
It is formed with timestamp, same order number, the big person of timestamp is valid data.
The present invention obtains the transaction request of ordering system, and the relevant transaction data of the transaction request of ordering system is stored in
In the database of ordering system, the transaction data of channel side is read from the file server of channel side, by the transaction of channel side
Data are stored on the NAS disk of ordering system, call the transaction data of ordering system and the transaction data of channel side, by order system
The transaction data of system and the transaction data of channel side are compared, when the transaction data of ordering system and the transaction data of channel side
When being compared successfully, the file in the NAS disk of ordering system is uploaded in the file server of channel side.Reconciliation of the present invention
Performance is stablized, and as the data volume of transaction is increasing, reconciliation and export performance will not significantly decrease;Scalability
It is strong: performance can be improved by way of increasing machine node;Reduce system pressure: thought handled using class sliding window formula,
Reduce reconciliation and the processing of the export pressure caused by database;Business is fault-tolerant: only newly-increased and additional for HDFS
Feature designs fault-tolerant architecture, avoids data modification that big performance is brought to be lost.
The present invention also provides a kind of bill processing units.At the bill shown in Fig. 2, provided for one embodiment of the invention
Manage the schematic diagram of internal structure of device.
In the present embodiment, bill processing unit 1 can be PC (Personal Computer, PC), can also be with
It is the terminal devices such as smart phone, tablet computer, portable computer.The bill processing unit 1 includes at least memory 11, processing
Device 12, communication bus 13 and network interface 14.
Wherein, memory 11 include at least a type of readable storage medium storing program for executing, the readable storage medium storing program for executing include flash memory,
Hard disk, multimedia card, card-type memory (for example, SD or DX memory etc.), magnetic storage, disk, CD etc..Memory 11
It can be the internal storage unit of bill processing unit 1, such as the hard disk of the bill processing unit 1 in some embodiments.It deposits
Reservoir 11 is also possible in further embodiments on the External memory equipment of bill processing unit 1, such as bill processing unit 1
The plug-in type hard disk of outfit, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD)
Card, flash card (Flash Card) etc..Further, memory 11 can also both include the storage inside of bill processing unit 1
Unit also includes External memory equipment.Memory 11 can be not only used for the application software that storage is installed on bill processing unit 1
And Various types of data, such as the code of bill processing routine 01 etc., it can be also used for temporarily storing and exported or will be defeated
Data out.
Processor 12 can be in some embodiments a central processing unit (Central Processing Unit,
CPU), controller, microcontroller, microprocessor or other data processing chips, the program for being stored in run memory 11
Code or processing data, such as execute bill processing routine 01 etc..
Communication bus 13 is for realizing the connection communication between these components.
Network interface 14 optionally may include standard wireline interface and wireless interface (such as WI-FI interface), be commonly used in
Communication connection is established between the device 1 and other electronic equipments.
Optionally, which can also include user interface, and user interface may include display (Display), input
Unit such as keyboard (Keyboard), optional user interface can also include standard wireline interface and wireless interface.It is optional
Ground, in some embodiments, display can be light-emitting diode display, liquid crystal display, touch-control liquid crystal display and OLED
(Organic Light-Emitting Diode, Organic Light Emitting Diode) touches device etc..Wherein, display can also be appropriate
Referred to as display screen or display unit, for being shown in the information handled in bill processing unit 1 and for showing visually
User interface.
Fig. 2 illustrates only the bill processing unit 1 with component 11-14 and bill processing routine 01, art technology
Personnel may include than illustrating more it is understood that structure shown in fig. 1 does not constitute the restriction of reconciliation processing apparatus 1
Perhaps more component perhaps combines certain components or different component layouts less.
In 1 embodiment of device shown in Fig. 2, bill processing routine 01 is stored in memory 11;Processor 12 executes
Following steps are realized when the bill processing routine 01 stored in memory 11:
The transaction request for obtaining ordering system, is stored in order system for the relevant transaction data of the transaction request of ordering system
In the database of system.
In the present embodiment, channel side sends real-time deal request and is given to ordering system, and ordering system in real time asks transaction
Relevant transaction data is asked to be stored in database.
Further, in another embodiment of apparatus of the present invention, bill processing routine can also be called by processor, with reality
Existing following steps:
According to sliding window strategy, start timed task in each time window, by the ordering system in time window
Transaction data fragment batch write-in Hadoop distributed file system (HDFS), according to the Hash codes of the application odd numbers of order
Hashcode carries out fragment.Such as filename starts with 0, step-length 1 is incremented by.
The transaction data that channel side is read from the file server of channel side, the transaction data of channel side is stored in and is ordered
On the NAS disk of single system.
In the present embodiment, periodically (such as daily) number of deals for reading channel side from the file server of channel side
According to the transaction data of channel side is stored on the network attached storage NAS disk of ordering system.The temporal frequency periodically read can
With configured in advance.
Wherein, fragment is carried out according to the Hash codes hashcode of the application odd numbers of the order of channel side.Such as filename is with 0
Start, step-length 1, is incremented by.
The transaction data of ordering system and the transaction data of channel side are called, by the transaction data and channel side of ordering system
Transaction data be compared.
Wherein with the transaction data of the architectural form management ordering system of cluster and the transaction data of channel side.Spark is answered
It, with the operation of independent process collection, is adjusted on cluster with SparkContext object in main program (referred to as driver)
Section.Particularly, in order to run on cluster, SparkContext can be with cluster manager dual system (Spark itself list of several types
Only cluster manager dual system or Mesos/YARN) it is connected, these cluster manager dual systems can distribute resource between application.Once even
Connect, Spark needs thread pool child node on cluster, that is, those execute calculate and storage using data work into
Journey.Then, it will send your application code (with JAR or the Python file defined and being transmitted to SparkContext)
To thread pool.Thread pool is allowed to run finally, SparkContext sends task.
Each application has the thread pool process of oneself, can keep in the operational process entirely applied and in multiple threads
Operation task.The advantage of doing so is that application is mutually isolated, both in scheduling aspect (task of each driving scheduling own)
Also in terms of execution (task of different application is run on different JVM).
For potential cluster manager dual system, Spark is unknowable.As long as it need thread pool process and they
Between communication, even when be run on the cluster manager dual system (for example, Mesos/YARN) for also supporting other application it is also opposite
Simply.
Because of driving scheduler task on cluster, it should run and be close to working node, in identical local area network more
It is good.
Preferably, bill processing routine can also be called by processor, to realize that following steps include:
Ordering system calls the Driver of Spark cluster, starts Spark task schedule, each task task of Spark from
The transaction data of channel side is read in Spark node on NAS disk, wherein the transaction data of channel side includes channel source mark
Know;And based on channel source identification, according in the transaction data of channel side order application odd numbers Hash codes hashcode into
Row repartition step process, Spark node reads in the transaction data of ordering system from HDFS system after having handled, and reads respectively
To the corresponding fragment of task task number ID of Spark.
Preferably, after by processing step above, there is the transaction of ordering system in each fragment in Spark
The transaction data of data and channel side, each task task start to compare the transaction of the transaction data and channel side to ordering system
Data processing.
When the transaction data of the transaction data of ordering system and channel side is compared successfully, by the NAS of ordering system
File in disk uploads in the file server of channel side.
Preferably, it when the transaction data of the transaction data of ordering system and channel side is compared successfully, that is, orders
When the transaction data of single system and the completely the same transaction data of channel side, Spark changes into the Map object of ordering system
String object is written to HDFS system;After the completion of all fragment write-ins, the API of HDFS system is called, by the text of each distribution
Part merges, and uploads to the NAS disk of ordering system, and ordering system takes the sftp that the file under the NAS disk uploads to channel side
It is engaged on device.
When the transaction data of the transaction data of ordering system and channel side is compared unsuccessful, that is to say, that reconciliation has
Problem then needs artificial treatment data, after the completion of data processing, repeats above step reconciliation again secondary.
It is therefore preferred that being connect when the transaction data of the transaction data of ordering system and channel side is compared unsuccessful
The processing to problem data is received, and re-starts reconciliation.
Preferably, the HDFS system configuration has fault-tolerant architecture, and wherein fault-tolerant architecture is made of metadata and timestamp.Phase
With order number, the big person of timestamp is valid data.Because may be modified for data on HDFS, while HDFS does not support to repair
The problem of changing is related to a fault-tolerant architecture, avoids importing database data again because of data modification.Fault-tolerant architecture is by metadata
It is formed with timestamp, same order number, the big person of timestamp is valid data.
The present invention obtains the transaction request of ordering system, and the relevant transaction data of the transaction request of ordering system is stored in
In the database of ordering system, the transaction data of channel side is read from the file server of channel side, by the transaction of channel side
Data are stored on the NAS disk of ordering system, call the transaction data of ordering system and the transaction data of channel side, by order system
The transaction data of system and the transaction data of channel side are compared, when the transaction data of ordering system and the transaction data of channel side
When being compared successfully, the file in the NAS disk of ordering system is uploaded in the file server of channel side.Reconciliation of the present invention
Performance is stablized, and as the data volume of transaction is increasing, reconciliation and export performance will not significantly decrease;Scalability
It is strong: performance can be improved by way of increasing machine node;Reduce system pressure: thought handled using class sliding window formula,
Reduce reconciliation and the processing of the export pressure caused by database;Business is fault-tolerant: only newly-increased and additional for HDFS
Feature designs fault-tolerant architecture, avoids data modification that big performance is brought to be lost.
Optionally, in other embodiments, bill processing routine can also be divided into one or more module, and one
Or multiple modules are stored in memory 11, and performed by one or more processors (the present embodiment is processor 12)
To complete the present invention, the so-called module of the present invention is the series of computation machine program instruction section for referring to complete specific function, is used
In implementation procedure of the description bill processing routine in bill processing unit.
It is the program mould of the bill processing routine in one embodiment of bill processing unit of the present invention for example, referring to shown in Fig. 3
Block schematic diagram, in the embodiment, bill processing routine can be divided into plate and obtain module 10, memory module 20, compare mould
Block 30 and sending module 40, illustratively:
It obtains module 10 to be used for: the transaction request of ordering system is obtained, by the relevant transaction of the transaction request of ordering system
Data are stored in the database of ordering system;
Memory module 20 is used for: the transaction data of channel side is read from the file server of channel side, by channel side
Transaction data is stored on the network attached storage NAS disk of ordering system;
Comparison module 30 is used for: the transaction data of ordering system and the transaction data of channel side is called, by ordering system
Transaction data and the transaction data of channel side are compared;
Sending module 40 is used for: when the transaction data of the transaction data of ordering system and channel side is compared successfully,
File in the NAS disk of ordering system is uploaded in the file server of channel side.
The program modules such as above-mentioned acquisition module 10, memory module 20, comparison module 30 and sending module 40 are performed institute
Functions or operations step and above-described embodiment of realization are substantially the same, and details are not described herein.
In addition, the embodiment of the present invention also proposes a kind of computer readable storage medium, the computer readable storage medium
On be stored with bill processing routine, the bill processing routine can be executed by one or more processors, to realize following operation:
The transaction request for obtaining ordering system, is stored in order system for the relevant transaction data of the transaction request of ordering system
In the database of system;
The transaction data that channel side is read from the file server of channel side, the transaction data of channel side is stored in and is ordered
On the network attached storage NAS disk of single system;
The transaction data of ordering system and the transaction data of channel side are called, by the transaction data and channel side of ordering system
Transaction data be compared;
When the transaction data of the transaction data of ordering system and channel side is compared successfully, by the NAS of ordering system
File in disk uploads in the file server of channel side.
Computer readable storage medium specific embodiment of the present invention and above-mentioned bill processing unit and each embodiment of method
It is essentially identical, do not make tired state herein.
It should be noted that the serial number of the above embodiments of the invention is only for description, do not represent the advantages or disadvantages of the embodiments.And
The terms "include", "comprise" herein or any other variant thereof is intended to cover non-exclusive inclusion, so that packet
Process, device, article or the method for including a series of elements not only include those elements, but also including being not explicitly listed
Other element, or further include for this process, device, article or the intrinsic element of method.Do not limiting more
In the case where, the element that is limited by sentence "including a ...", it is not excluded that including process, device, the article of the element
Or there is also other identical elements in method.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in one as described above
In storage medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that terminal device (it can be mobile phone,
Computer, server or network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.