CN116610723B

CN116610723B - Method for acquiring big data of multiple household appliances and computing equipment

Info

Publication number: CN116610723B
Application number: CN202310382337.2A
Authority: CN
Inventors: 陶莎; 徐博; 靳佳为
Original assignee: Shenzhen Zhige Digital Technology Co ltd
Current assignee: Shenzhen Zhige Digital Technology Co ltd
Priority date: 2023-04-04
Filing date: 2023-04-04
Publication date: 2025-12-16
Anticipated expiration: 2043-04-04
Also published as: CN116610723A

Abstract

This application provides a method and computing device for acquiring big data from multiple e-commerce platforms. The method includes: simultaneously launching N first threads; using the multiple first threads to request first-type data from different online stores on the e-commerce platform; storing the first-type data in a distributed file system; and storing the first-type data in the file system in a distributed database system. The capability of the N first threads to request data from the e-commerce platform exceeds the e-commerce platform's request limit for first-type data, and the capability of the N-2 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for first-type data. The solution of this application maximizes the utilization of the interface capabilities provided by the e-commerce platform.

Description

Method for acquiring big data of multiple household appliances and computing equipment

Technical Field

The application relates to the technical field of machine learning and business big data, in particular to a method and computing equipment for acquiring multi-household appliance business big data.

Background

With the development of network computing technology, a large amount of business big data is generated in electronic commerce. For example, compared to the traditional industry, e-commerce can produce a vast amount of raw e-commerce data available on its ecological value chain. The acquisition, processing, or efficient use of such data may provide assistance to the business operations or support for business decisions.

Some e-commerce platforms have opened a data interface. How to efficiently obtain data from an e-commerce platform is a notable problem, particularly for data analysis service type systems that require the acquisition of numerous e-commerce online store data.

The above information disclosed in the background section is only for enhancement of understanding of the background of the application and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

Disclosure of Invention

The application aims to provide a method and computing equipment for acquiring big data of multiple home appliances, which can maximize and utilize interface capability provided by an e-commerce platform.

The user characteristics and advantages of the present application will become apparent from the detailed description set forth below, or may be learned in part by practice of the application.

According to one aspect of the application, a method for acquiring big data of multiple home appliances is provided, which comprises the steps of starting a plurality of first threads with the number of N at the same time, respectively requesting first types of data of different online stores from an e-commerce platform by utilizing the plurality of first threads, storing the first types of data in a distributed file system, and storing the first types of data in the file system in a distributed database system, wherein the capacity of the N first threads for requesting the data from the e-commerce platform is larger than the request limit of the electricity Shang Ping for the first types of data, and the capacity of the N-2 first threads for requesting the data from the e-commerce platform is smaller than the request limit of the electricity Shang Ping for the first types of data.

According to another aspect of the application there is provided a computing device comprising a processor, a memory having a computer program stored thereon, the foregoing method being implemented when the processor executes the computer program.

According to another aspect of the application, there is provided a computer readable medium having stored thereon a computer program which when executed by a processor implements the method as described above.

According to some embodiments, by controlling the number of threads, the interface capabilities provided by the e-commerce platform may be maximized.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.

Drawings

The above and other objects, features and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

Fig. 1 shows a schematic diagram of an application scenario of the technical solution of the present application.

Fig. 2 shows a flowchart of a method of acquiring multi-home vendor big data according to an exemplary embodiment of the present application.

FIG. 3 illustrates an operational flow diagram for obtaining online store IDs in a thread-safe manner and requesting first type data for different online stores from an e-commerce platform, respectively, using a plurality of first threads, according to an embodiment.

FIG. 4 illustrates an operational flow diagram for obtaining online store IDs in a thread-safe manner and requesting first type data for different online stores from an e-commerce platform, respectively, using a plurality of first threads in accordance with another embodiment.

Fig. 5 illustrates a flowchart of a method for acquiring multi-home merchant big data using a priority queue according to an example embodiment.

FIG. 6 illustrates a flowchart of a method for prioritizing completion of a request for corresponding e-commerce data to an e-commerce platform based on an online store ID in a non-execution priority queue in accordance with an example embodiment.

FIG. 7 illustrates a flowchart of a method for requesting corresponding e-commerce data from an e-commerce platform based on an online store ID in a non-execution regular queue in accordance with an example embodiment.

Fig. 8 illustrates a flowchart of a method for acquiring multi-vendor big data using multi-threading and priority queues according to an example embodiment.

Fig. 9 is a schematic diagram of a system for acquiring big data of multiple home appliances according to another embodiment of the present application.

FIG. 10 illustrates a block diagram of a computing device according to an example embodiment of the application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, but rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

The terms first, second and the like in the description and in the claims and in the above-described figures are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art will appreciate that the embodiments described herein may be combined with other embodiments.

The online operation mode of the electronic commerce generates a large amount of raw electronic commerce data of a repository, and some electronic commerce platforms open a data interface to acquire data for the electronic commerce. But e-commerce platforms generally have request restrictions on interface access to the same user, e.g., requests within one second cannot exceed a predetermined value, etc. Therefore, an efficient data request mode needs to be designed, so that the acquisition of large data of multiple home appliances can be completed as timely as possible under the condition of acquiring data of multiple online stores of multiple electric appliances.

Therefore, the embodiment of the application provides a method for acquiring big data of multiple home appliances, which can efficiently acquire big data of the electric appliances from an electric appliance platform through multithreading control and other modes.

The following describes the technical scheme of the present application in detail with reference to examples.

Referring to fig. 1, in an e-commerce system, data generated on an e-commerce value chain may be deposited in a database of an e-commerce platform. In order to acquire the precipitation data, the data can be authorized to be used by a data processing system according to the embodiment of the application through an API interface of an e-commerce platform in an e-commerce authorization mode. In addition, the system can also obtain the original electronic commerce data of multiple dimensions (sales, stock, flow, policy violations, logistics, settlement and the like) of the user in real time by carrying out data docking in the form of interfaces and the like with other main participants (including third party payments, logistics merchants and warehouse service merchants) of the electronic commerce, and can use a distributed technical means for storage and calculation.

After receiving the authorization, the data processing system according to the embodiment of the application pulls the original electronic commerce data of the corresponding electronic commerce in the electronic commerce platform to a storage system associated with the data processing system. According to some embodiments, the storage system may be a distributed storage system.

The data processing system processes the data, such as through standardization processing, to obtain data which can be used later, and then, can combine industry experience, business model, financial model and/or statistical model and the like, obtain risk rating results and risk figures of e-commerce enterprises through machine learning and the like, provide the risk rating results and risk figures to financial institutions such as banks and the like as reliable financing basis and the like.

Referring to fig. 2, at S201, a plurality of first threads of N number are simultaneously turned on.

According to an example embodiment, the N first threads are more capable of requesting data from the e-commerce platform than the electrical Shang Ping stations are requesting restrictions on the first type of data, and the N-2 first threads are less capable of requesting data from the e-commerce platform than the electrical Shang Ping stations are requesting restrictions on the first type of data. Thus, by controlling the number of threads, the interface capability provided by the e-commerce platform can be maximized. In addition, by controlling the number of threads, the computational resources and network resources are not wasted as much as possible, and excessive thread idling is avoided.

At S203, first type data of different online stores are respectively requested from the e-commerce platform using a plurality of first threads.

After obtaining the internet shop authorization, the data interface of the e-commerce platform can be accessed. For example, after confirming the authorization of the customer, the e-commerce platform returns a token (token) of the online store, saves the token in the background, and all interfaces need to carry the token to request.

The interfaces of the e-commerce platform may include an order data interface, a sales data interface, a report data interface, a document data interface, and the like. The interfaces provided by different e-commerce platforms may vary. The first type of data may be data acquired from one of these interfaces, as the application is not limited in this regard. It is to be readily understood that the first type of interface is generally referred to herein, and is intended to indicate that the technical solution of the present application may obtain data of any type of interface, and certainly may also obtain data of these interfaces simultaneously by different threads (e.g., 2 nd, n-th, threads).

According to some embodiments, further, the N-1 first threads have less capability to request data from the e-commerce platform than the request limit of the electric Shang Ping stations for the first type of data.

According to some embodiments, the request responses received by the plurality of first threads may also be monitored, thereby dynamically adjusting the number of first threads, and thus making a more accurate dynamic adjustment of the number of threads.

According to some embodiments, the plurality of first threads may run on a plurality of different servers when performance of a single server is not satisfactory.

At S205, the first type of data is stored in the distributed file system.

According to the example embodiment, the acquired first type data is stored in the distributed file system, so that the data acquisition efficiency is improved, and the data security is improved.

For example, the distributed file system may comprise FastDFS systems, but the application is not limited thereto.

At S207, the first type of data in the file system is stored in the distributed database system.

According to an example embodiment, further, the first type of data in the file system may be stored in a distributed database system. Stored in the file system are a large number of small text files that are retrieved. For the convenience of statistical processing and utilization, a large number of small text files are subjected to analysis processing, and data in the small text files are stored in a distributed database system.

For example, the distributed database system may comprise a TiDB system, but the application is not limited thereto.

According to the above-described embodiments, by employing multiple threads whose request capabilities are well beyond the interface limitations of the e-commerce platform, the interface capabilities provided by the e-commerce platform can be maximized while minimizing the waste of computing resources and network resources.

Referring to fig. 3, at S301, each first line Cheng Huoqu is a first distributed lock.

According to some embodiments, distributed lock Redisson may be implemented with an open source, but the application is not limited thereto.

At S303, the first thread that acquired the first distributed lock acquires the online store ID and releases the first distributed lock.

According to some embodiments, the online store ID (identifier) of the online store that needs to obtain the data is stored in the unexecuted queue.

The first thread obtains the online store ID from the unexecuted queue and then the first thread places the online store ID into the in-progress queue. In this way, since the first thread has acquired the first distributed lock, the problem of the online store ID being repeatedly acquired does not occur.

According to some embodiments, a list of online stores authorized by a client may be queried periodically, a set of online store IDs may be obtained, and the obtained set of online store IDs may be added to the unexecuted queue after de-duplication. For example, deduplication may be achieved by removing the online store IDs of the unexecuted queue and the in-execution queue from the acquired set of online store IDs.

At S305, the first thread that acquired the online store ID requests corresponding first type data from the e-commerce platform.

After the online store ID is acquired, the first thread may acquire the token according to the online store ID and parameters such as a time range of data to be acquired, and request first type data of the corresponding online store, such as a single data interface, a sales data interface, a report data interface, or a document data interface, from the e-commerce platform.

According to an example embodiment, after the first thread completes requesting the first type of data corresponding to the online store ID from the e-commerce platform, the online store ID may be removed from the in-execution queue and another online store ID may be obtained from the un-execution queue, and then the above operations may be repeated.

Referring to fig. 4, at S401, a plurality of first threads acquire online store IDs from unexecuted queues, respectively.

According to some embodiments, the online store ID (identifier) of the online store that needs to obtain the data is placed into an unexecuted queue, which is a queue based on a single-threaded memory storage system, which may include, for example, a Redis system. However, the present application is not limited thereto, and other thread-safe queues may be used to store the online store IDs, so as to ensure that the problem of repeatedly acquiring the online store IDs does not occur.

In S403, the first thread that acquired the online store ID puts the acquired online store ID in the in-progress queue.

Thus, through the queue in execution, the visualization and real-time monitoring of the online store in the data grabbing process can be realized.

At S405, the first thread that acquired the online store ID requests corresponding first type data from the e-commerce platform.

Referring to fig. 5, at S501, a queue for storing online store IDs is set. According to an example embodiment, the queues include a normal queue not executing, a normal queue in execution, a priority queue not executing, a priority queue in execution.

By setting different queues, a basis can be provided for preferentially acquiring certain online store data. In addition, through the arrangement of the queues in execution, the visualization and real-time monitoring of online stores in the data grabbing process can be realized, and the preferential acquisition of newly-added online store data is further ensured.

At S503, the online store ID in the queue is updated regularly.

According to some embodiments, customer-authorized online store names may be queried periodically, such as from a database, to obtain an online store ID set. The online store IDs in the queue are then excluded from the set of online store IDs. And adding the reserved online store ID of the newly added online store to the non-execution priority queue, and adding the reserved online store ID of the non-new online store to the non-execution conventional queue.

According to some embodiments, the acquired set of online store IDs may include a first set including online store IDs of newly added online stores and a second set including online store IDs of non-new online stores. Thus, when the online store IDs in the queue are excluded from the online store ID set, the online store IDs of the non-execution priority queue and the in-execution priority queue are excluded from the first set, and the online store IDs of the non-execution regular queue and the in-execution regular queue are excluded from the second set.

In S505, the request for the corresponding e-commerce data from the e-commerce platform is completed preferentially according to the online store ID in the unexecuted priority queue.

When there is a newly added online store authorized to access the data interface, since the newly authorized online store has a large amount of history data, a priority is required to acquire the data of the newly added online store.

According to an example embodiment, requesting the corresponding e-commerce data from the e-commerce platform is accomplished by processing the online store ID in the unexecuted priority queue.

In S507, when there is no data in the priority queue in execution and the priority queue in execution, corresponding e-commerce data is requested to the e-commerce platform according to the online store ID in the regular queue in non-execution.

According to an example embodiment, the acquisition of the data of the online stores in the regular queue is started after the completion of the data acquisition of the online stores in the priority queue. For example, the data of the stock online store is acquired only after the data of the newly added online store is acquired from the e-commerce platform. In this way, not only can the preferential acquisition of specific data be ensured, but also the efficient acquisition of data can be realized in a simple manner, and the switching or waiting process in the data acquisition process is reduced as much as possible.

According to some embodiments, the acquired data may be stored in a distributed file system, improving data acquisition efficiency and increasing data security. For example, the distributed file system may comprise FastDFS systems, but the application is not limited thereto.

According to some embodiments, further, the obtained data in the file system may be stored in a distributed database system. Stored in the file system are a large number of small text files that are retrieved. For the convenience of statistical processing and utilization, a large number of small text files are subjected to analysis processing, and data in the small text files are stored in a distributed database system. For example, the distributed database system may comprise a TiDB system, but the application is not limited thereto.

According to some embodiments, S505 and S507 may be performed in multiple threads. The unexecuted regular queue and unexecuted priority queue are queues based on a single-threaded memory storage system, which may include a Redis system. However, the present application is not limited thereto, and other thread-safe queues may be used to store the online store ID, or the online store ID may be acquired in other thread-safe manners, so as to ensure that the problem that the online store ID is repeatedly acquired does not occur.

Referring to fig. 6, in S601, the online store ID is acquired from the unexecuted priority queue.

According to an example embodiment, each time an online store ID is acquired to request corresponding data from an e-commerce platform, the online store ID is first acquired from an unexecuted priority queue. In this way, the data of some online stores can be preferably acquired, for example, the data of newly added online stores is acquired first.

In S603, the acquired online store ID is put in the execution priority queue.

According to the embodiment, the acquired online store ID is put into the priority queue in execution, so that the online store in the data grabbing process can be visually and real-time monitored. In addition, preferential acquisition of newly added online store data, for example, can be further ensured, see the description below.

At S605, after the request for the corresponding e-commerce data from the e-commerce platform is completed according to the acquired online store ID, the online store ID is removed from the in-execution priority queue.

After the request of the data corresponding to the online store ID from the e-commerce platform is completed, the online store ID may be removed from the in-process priority queue, and then the above operations may be repeatedly performed.

Referring to fig. 7, in S701, the online store ID is acquired from the unexecuted priority queue.

According to an example embodiment, each time an online store ID is acquired to request corresponding data from an e-commerce platform, the online store ID is first acquired from an unexecuted priority queue. In this way, it is ensured that when the newly added online store ID is present in the non-execution priority queue, the online store ID can be processed with priority.

If the online store ID is not acquired from the unexecuted priority queue, but the in-execution priority queue is not empty, the process returns to S701 after a delay. In this way, it is ensured that the data of the online store in the conventional queue starts to be acquired from the e-commerce platform after the data of the online store in the priority queue is acquired from the e-commerce platform is preferentially completed.

If the online store ID is not acquired from the unexecuted priority queue and the in-execution priority queue is empty, the flow goes to S703.

At S703, the online store ID is acquired from the non-execution regular queue.

Since the online store that is preferentially handled is not already needed at this time, the online store in the regular queue, for example, an on-store online store, starts to be handled.

In S705, the acquired online store ID is put in the in-progress regular queue.

As before, by placing the acquired online store ID into the executing regular queue, the online store of the data grabbing process can be visually and real-time monitored.

At S707, after the request for the corresponding e-commerce data from the e-commerce platform is completed according to the acquired online store ID, the online store ID is removed from the in-process regular queue.

After the request of the data corresponding to the online store ID from the e-commerce platform is completed, the online store ID may be removed from the in-process regular queue and then the above operations may be repeatedly performed.

Referring to fig. 8, at S801, a queue for storing online store IDs is set. According to an example embodiment, the queues include a normal queue not executing, a normal queue in execution, a priority queue not executing, a priority queue in execution.

At S803, a number N of first threads are simultaneously started.

According to the example embodiment, the data is acquired in a multithreading mode, so that the data acquisition efficiency is further improved.

According to some embodiments, the ability of the N first threads to request data from the e-commerce platform is greater than the request limit of the electric Shang Ping stations for the first type of data. Further, according to some embodiments, it may be determined that the ability of N-2 or N-1 first threads to request data from the e-commerce platform is less than the request limit of the electricity Shang Ping for the first type of data.

Thus, by controlling the number of threads, the interface capability provided by the e-commerce platform can be maximized. In addition, by controlling the number of threads, computing resources and network resources are not wasted as much as possible.

At S805, the online store ID in the queue is updated regularly.

In S807, first class data of different online stores are respectively requested from the e-commerce platform according to the online store IDs in the unexecuted priority queue using the plurality of first threads.

According to an example embodiment, requesting the corresponding first type of data from the e-commerce platform is accomplished by processing the online store ID in the unexecuted priority queue. By performing the acquisition of the first type of data in a multithreading manner, the data acquisition efficiency can be improved.

Furthermore, by controlling the number of threads, the interface capabilities provided by the e-commerce platform may be maximized and no computing and network resources are wasted as much as possible, according to some embodiments.

In S809, when there is no data in the non-execution priority queue and the in-execution priority queue, the first threads are utilized to request the first type data of different online stores from the e-commerce platform according to the online store IDs in the non-execution regular queue.

As before, by performing the acquisition of the first type of data in a multithreading manner, the data acquisition efficiency may be improved.

The method according to the embodiment shown in fig. 8 may otherwise employ a similar process to the other embodiments described above, and will not be described in detail here.

Referring to fig. 9, according to an example embodiment, a plurality of threads for acquiring online store data are executed on at least two servers a and B.

As shown in fig. 9, the data to be acquired includes four types, which are order data, sales data, report data, and document data, respectively. Each server a or server B creates 3 threads on each class of data to acquire data from the e-commerce platform for online stores. As previously described, the number of threads may be adjusted based on the status of the server and network resources and the limitations of the e-commerce platform on the interface requests.

Referring to fig. 9, for each thread, the online store ID is first acquired, and then the data of the corresponding online store is acquired from the e-commerce platform.

According to an embodiment, each thread first obtains the online store ID from the unexecuted priority queue, and places the obtained online store ID into the in-execution priority queue. And then, according to the acquired online store ID, completing the request of corresponding e-commerce data from the e-commerce platform, and removing the online store ID from the in-process priority queue. Then, another online store ID is acquired from the non-execution priority queue, and the process of acquiring corresponding data is continued.

If the online store ID is not acquired from the non-execution priority queue, but the execution priority queue is not empty, the online store ID is returned after delay, and the online store ID acquisition operation is continued. In this way, it is ensured that the data of the online store in the conventional queue starts to be acquired from the e-commerce platform after the data of the online store in the priority queue is acquired from the e-commerce platform is preferentially completed.

If no online store ID is obtained from the unexecuted priority queue and the in-execution priority queue is empty, indicating that no online stores need to be prioritized at this time, processing of online stores in the regular queue is started, and the online store ID is obtained from the unexecuted regular queue. And putting the acquired online store ID into an executing conventional queue, and completing the request of corresponding e-commerce data from the e-commerce platform according to the acquired online store ID. Then, the online store ID is removed from the executing regular queue, another online store ID is tried to be acquired from the non-executing priority queue, and the process of acquiring data is continued.

As shown in fig. 10, the computing device 30 includes a processor 12 and a memory 14. Computing device 30 may also include a bus 22, a network interface 16, and an I/O interface 18. The processor 12, memory 14, network interface 16, and I/O interface 18 may communicate with each other via a bus 22.

The processor 12 may include one or more general purpose CPUs (Central Processing Unit, central processing units), microprocessors, or application specific integrated circuits, etc. for executing associated program instructions.

Memory 14 may include machine-system-readable media in the form of volatile memory, such as Random Access Memory (RAM), read Only Memory (ROM), and/or cache memory. Memory 14 is used to store one or more programs including instructions as well as data. The processor 12 may read instructions stored in the memory 14 to perform the methods according to embodiments of the application described above.

Computing device 30 may also communicate with one or more networks through network interface 16. The network interface 16 may be a wired network interface or a wireless network interface, or may be a virtual network interface.

Computing device 30 may also communicate with one or more external devices (e.g., audio input devices, audio output devices, cameras, keyboards, mice, displays, various types of sensors, etc.) through input/output (I/O) interface 18.

Bus 22 may include an address bus, a data bus, a control bus, and the like. Bus 22 provides a path for exchanging information between the components.

It should be noted that, in the implementation, the computing device 30 may further include other components necessary to achieve normal operation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.

The present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above method. The computer readable storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, DVDs, CD-ROMs, micro-drives, and magneto-optical disks, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), network storage devices, cloud storage devices, or any type of media or device suitable for storing instructions and/or data.

Embodiments of the present application also provide a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform part or all of the steps of any one of the methods described in the method embodiments above.

It will be clear to a person skilled in the art that the solution according to the application can be implemented by means of software and/or hardware. "Unit" and "module" in this specification refer to software and/or hardware capable of performing a specific function, either alone or in combination with other components, where the hardware may be, for example, a field programmable gate array, an integrated circuit, or the like.

It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, such as the division of the units, merely a logical function division, and there may be additional manners of dividing the actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some service interface, device or unit indirect coupling or communication connection, electrical or otherwise.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable memory. Based on this understanding, the technical solution of the present application may be embodied essentially or partly in the form of a software product, or all or part of the technical solution, which is stored in a memory, and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or part of the steps of the method according to the embodiments of the present application.

The embodiments of the present application have been described and illustrated in detail above. It should be clearly understood that the present application describes how to make and use particular examples, but the present application is not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.

Those skilled in the art will readily appreciate from the description of example embodiments that the risk rating prediction method according to embodiments of the application has at least one or more of the following advantages.

According to some embodiments, by controlling the number of threads, computing resources and network resources are not wasted as much as possible.

According to some embodiments, by employing a thread-safe queue to store online store IDs, it may be ensured that the problem of online store IDs being repeatedly acquired does not occur.

According to some embodiments, the in-execution queue is set, so that visualization and real-time monitoring of an online store in the data grabbing process can be realized.

According to some embodiments, by setting a priority queue, priority acquisition of data, such as newly added online stores, may be achieved in a simple manner.

According to some embodiments, by setting an in-execution priority queue, priority acquisition of newly added online store data, for example, may be further ensured.

According to some embodiments, each time the online store ID is acquired to request corresponding data from the e-commerce platform, the online store ID is acquired from the unexecuted priority queue, so that the unexecuted priority queue can be ensured to be processed preferentially when the online store ID is newly added.

According to some embodiments, when the priority queue is not empty in execution, the online store ID is not acquired from the regular queue, so that it may be ensured that the online store data in the regular queue starts to be acquired from the e-commerce platform after the online store data in the priority queue is preferentially acquired from the e-commerce platform.

According to some embodiments, through setting different queues and combining multithreading to acquire data, priority of acquiring certain online store data can be realized, and meanwhile, the data acquisition efficiency is improved.

The foregoing may be better understood in light of the following clauses:

1. a method for obtaining big data of multiple home appliances, comprising:

simultaneously starting a plurality of first threads with the number N;

respectively requesting first type data of different online stores from an e-commerce platform by utilizing the first threads;

Storing the first type of data in a distributed file system;

Storing said first type of data in said file system in a distributed database system,

The capability of the N first threads for requesting data from the e-commerce platform is larger than the request limit of the electric Shang Ping station for the first type of data, and the capability of the N-2 first threads for requesting data from the e-commerce platform is smaller than the request limit of the electric Shang Ping station for the first type of data.

2. The method of clause 1, wherein the capability of the N-1 first threads to request data from the e-commerce platform is less than the request limit of the electricity Shang Ping for data of the first type.

3. The method of clause 1, further comprising:

the request responses received by the plurality of first threads are monitored to dynamically adjust the number of first threads.

4. The method of clause 1, wherein the plurality of first threads originate from a plurality of servers.

5. The method of clause 1, wherein requesting the first type of data for different online stores from the e-commerce platform using the plurality of first threads, respectively, comprises:

each first line Cheng Huoqu first distributed locks;

A first thread which acquires the first distributed lock acquires an online store ID and releases the first distributed lock;

and the first thread which acquires the online store ID requests corresponding first type data from the e-commerce platform.

6. The method of clause 5, wherein the first thread that acquired the first distributed lock acquires an online store ID, comprising:

The first thread obtains an online store ID from an unexecuted queue;

The first thread places the online store ID into an executing queue.

7. The method of clause 1, wherein requesting the first type of data for different online stores from the e-commerce platform using the plurality of first threads, respectively, comprises:

the first threads respectively acquire online store IDs from unexecuted queues, wherein the unexecuted queues are queues based on a single-thread memory storage system;

the first thread which acquires the online store ID puts the acquired online store ID into an executing queue;

8. The method of clause 7, wherein the single threaded memory storage system comprises a Redis system.

9. The method of clause 6 or 7, further comprising:

inquiring a list of online stores authorized by a client at regular time to acquire an online store ID set;

And adding the acquired online store ID set to the unexecuted queue after de-duplicating the acquired online store ID set.

10. The method of clause 6 or 7, further comprising:

After the first thread which acquires the online store ID finishes requesting corresponding first type data from the e-commerce platform, removing the online store ID from the executing queue, and acquiring another online store ID from the unexecuted queue.

11. The method of clause 1, wherein the distributed file system comprises a FastDFS system.

12. The method of clause 1, wherein the distributed database system comprises a TiDB system.

13. The method of clause 1, further comprising:

Simultaneously starting a plurality of second threads with the number M;

requesting second class data of different online stores from the e-commerce platform by using the plurality of second threads,

The capacity of the M second threads for requesting data from the e-commerce platform is larger than the request limit of the electric Shang Ping stations for the second class of data, and the capacity of the M-2 first threads for requesting data from the e-commerce platform is smaller than the request limit of the electric Shang Ping stations for the second class of data.

14. A computing device, comprising:

A processor;

a memory having a computer program stored thereon;

the method of any one of clauses 1-13 being implemented when the processor executes the computer program.

The exemplary embodiments of the present application have been particularly shown and described above. It is to be understood that the application is not limited to the details of construction, arrangement or method of implementation described herein, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for obtaining big data of multiple home appliances, comprising:

simultaneously starting a plurality of first threads with the number N;

Storing the first type of data in a distributed file system;

2. The method of claim 1, wherein the N-1 first threads have less capability to request data from the e-commerce platform than the electrical Shang Ping stations have a request limit for data of a first type.

3. The method as recited in claim 1, further comprising:

4. The method of claim 1, wherein the plurality of first threads originate from a plurality of servers.

5. The method of claim 1, wherein requesting the first type of data for different online stores from the e-commerce platform using the plurality of first threads, respectively, comprises:

each first line Cheng Huoqu first distributed locks;

6. The method of claim 5, wherein the first thread that acquired the first distributed lock acquires an online store ID, comprising:

The first thread obtains an online store ID from an unexecuted queue;

The first thread places the online store ID into an executing queue.

7. The method of claim 1, wherein requesting the first type of data for different online stores from the e-commerce platform using the plurality of first threads, respectively, comprises:

8. The method of claim 7, wherein the single-threaded memory storage system comprises a Redis system.

9. The method of claim 6 or 7, further comprising:

10. The method of claim 6 or 7, further comprising:

11. The method of claim 1, wherein the distributed file system comprises a FastDFS system.

12. The method of claim 1, wherein the distributed database system comprises a TiDB system.

13. The method as recited in claim 1, further comprising:

Simultaneously starting a plurality of second threads with the number M;

14. A computing device, comprising:

A processor;

a memory having a computer program stored thereon;

the method of any of claims 1-13 being implemented when the processor executes the computer program.