CN118885304B - A GNSS occultation detection ground application system - Google Patents
A GNSS occultation detection ground application system Download PDFInfo
- Publication number
- CN118885304B CN118885304B CN202411365758.5A CN202411365758A CN118885304B CN 118885304 B CN118885304 B CN 118885304B CN 202411365758 A CN202411365758 A CN 202411365758A CN 118885304 B CN118885304 B CN 118885304B
- Authority
- CN
- China
- Prior art keywords
- task
- data
- observation data
- occultation
- execution module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
- G01S19/01—Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
- G01S19/13—Receivers
- G01S19/14—Receivers specially adapted for specific applications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
- G01S19/01—Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
- G01S19/13—Receivers
- G01S19/35—Constructional details or hardware or software details of the signal processing chain
- G01S19/37—Hardware or software details of the signal processing chain
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Hardware Redundancy (AREA)
Abstract
本申请公开了一种GNSS掩星探测地面应用系统，涉及数据处理系统领域，该系统利用任务调度模块进行任务的调度分配，利用分布式的任务执行模块来执行任务，对于最常见的数据反演任务，任务调度模块在解析得到卫星原始载荷数据中的定位观测数据和掩星观测数据后，在检测到分布式文件系统中与每组掩星观测数据匹配的定位观测数据的数据量达到反演要求时才生成对应的数据反演任务，保证了后续数据反演的质量，任务调度模块可以根据不同任务执行模块的资源情况以及不同任务的调度优先级情况，对数量庞大以及要求不同的各种任务进行灵活的任务分配，保证任务执行的有效性，有利于提高任务处理效率。
The present application discloses a GNSS occultation detection ground application system, which relates to the field of data processing systems. The system utilizes a task scheduling module to schedule and allocate tasks, and utilizes a distributed task execution module to execute tasks. For the most common data inversion tasks, after the task scheduling module parses and obtains the positioning observation data and the occultation observation data in the original payload data of the satellite, it generates a corresponding data inversion task only when it detects that the data volume of the positioning observation data matching each group of occultation observation data in the distributed file system reaches the inversion requirement, thereby ensuring the quality of subsequent data inversion. The task scheduling module can flexibly allocate tasks for a large number of tasks with different requirements according to the resource conditions of different task execution modules and the scheduling priorities of different tasks, thereby ensuring the effectiveness of task execution and being conducive to improving task processing efficiency.
Description
Technical Field
The application relates to the field of data processing systems, in particular to a GNSS occultation detection ground application system.
Background
The GNSS (Global Navigation SATELLITE SYSTEM ) occultation detection technology is a new remote sensing technology, can detect the global atmospheric layer and the ionosphere, thereby providing the ionosphere and a neutral atmospheric profile, and has a great effect in space environment tasks. The GNSS occultation detection technology has the advantages of high precision, high vertical resolution, long-term stability, global coverage, all-weather detection and the like, can make up for the defects of the traditional atmosphere detection means, has the advantages of small volume, low power consumption and the like, and is very suitable for carrying microsatellites to carry out large-scale constellation detection.
When the GNSS occultation detection technology is applied, a ground application system is required to receive and process load data acquired by satellites and further generate various levels of data products. With the development and gradual popularization of GNSS occultation detection technology, the number of satellites is increased, and the data volume of load data acquired by each satellite is also increased, which provides new challenges for the performance of ground application systems.
Disclosure of Invention
Aiming at the problems and the technical requirements, the application provides a GNSS occultation detection ground application system, which has the following technical scheme:
The GNSS occultation detection ground application system comprises a task scheduling module, a plurality of task execution modules, a ZooKeeper distributed cache and a distributed file system, wherein a task list to be executed and an execution module state list are maintained in the ZooKeeper distributed cache;
The task scheduling module is used for receiving the original load data sent by each satellite of the GNSS occultation detection system, analyzing the received original load data each time to obtain positioning observation data and occultation observation data, storing the analyzed positioning observation data in the distributed file system, initializing and generating a data inversion task according to each group of occultation observation data and the matched positioning observation data when the data quantity of the positioning observation data matched with each group of occultation observation data in the distributed file system is detected to reach the inversion requirement, and adding the data inversion task into a task list to be executed;
The task scheduling module is also used for respectively distributing each task in the task list to be executed to the corresponding task execution module according to the resource occupation condition of each task execution module recorded in the execution module state list and the scheduling priority of each task in the task list to be executed;
the system comprises a task scheduling module, a task execution module, a distributed file system, a data inversion module and a data processing module, wherein the task scheduling module is used for receiving the task and sending the task execution state to the task execution module;
The task scheduling module is also used for updating the resource occupation condition of each task execution module recorded in the execution module state list according to the task execution state fed back by each task execution module.
The task scheduling module is further technically characterized by detecting whether the data volume of positioning observation data matched with the currently-resolved occultation observation data in the distributed file system meets inversion requirements or not when a group of occultation observation data is obtained through each resolution, initializing and generating a data inversion task according to the currently-resolved occultation observation data and the matched positioning observation data when the data volume of the positioning observation data matched with the currently-resolved occultation observation data in the distributed file system meets inversion requirements, otherwise, correspondingly storing the currently-resolved occultation observation data and satellite identifications of satellites for transmitting the occultation observation data in an unprocessed list maintained by the distributed file system;
the task scheduling module is further used for sequentially traversing each group of occultation observation data of the unprocessed list at preset time intervals, initializing and generating a corresponding data inversion task and deleting the traversed occultation observation data from the unprocessed list when the data quantity of the positioning observation data matched with the traversed occultation observation data in the distributed file system is detected to reach the inversion requirement, otherwise, keeping the traversed occultation observation data in the unprocessed list.
The task scheduling module stores the positioning observation data obtained by analysis in a distributed file system, wherein the task scheduling module correspondingly stores each positioning frame and the positioning time of each positioning observation data obtained by analysis and a satellite identification of a satellite sending the positioning observation data in the distributed file system;
the task scheduling module detecting whether the data volume of the positioning observation data matched with each group of occultation observation data in the distributed file system meets inversion requirements comprises the following steps:
Detecting the number of positioning frames matched with the occultation observation data in the distributed file system, wherein the positioning frames matched with the occultation observation data are positioning frames which correspond to target satellite identifications and the corresponding positioning time is within the duration time of the occultation observation data, and the target satellite identifications are satellite identifications of satellites for transmitting the occultation observation data;
When the number of the positioning frames matched with the occultation observation data is detected to reach a quantity threshold, determining that the data quantity of the positioning observation data matched with the occultation observation data reaches an inversion requirement, extracting all the positioning frames matched with the occultation observation data to obtain the positioning observation data matched with the occultation observation data, otherwise, determining that the data quantity of the positioning observation data matched with the occultation observation data does not reach the inversion requirement.
Each task execution module is integrated with a plurality of CPU cores, each task execution module reserves a plurality of CPU cores as basic cores for operating an operating system of the task execution module, and other CPU cores except the basic cores are isolated by isolcpus commands;
each task execution module is further configured to select one CPU core from the isolated CPU cores when the data inversion task is received, and control the received data inversion task to be executed on the selected CPU core by using taskset commands.
The system is further technically characterized in that each task execution module is used for defining a unique environment variable for the data inversion task and copying the environment variable into each process when a plurality of processes are started to execute one data inversion task in parallel;
Each task execution module is further used for closing the failed process when detecting that any process fails in the process of executing the data inversion task, and closing other processes with the same environment variable as the failed process so as to completely close the processes for executing the same data inversion task.
The task scheduling module is further technically characterized in that the task scheduling modules are respectively deployed on two physical nodes, real IP addresses of the two physical nodes are virtualized into virtual IP addresses of the task scheduling module through VRRP technology, and the task scheduling module provides services to the outside through the virtual IP addresses;
Each physical node sends own node attribute information to the other physical node when starting, and each physical node determines own control priority according to the own node attribute information and determines the control priority of the other physical node according to the received node attribute information of the other physical node;
the master node has the control right of the virtual IP address of the task scheduling module and provides external service of the task scheduling module, and the master node performs heartbeat communication with the backup node;
When any one physical node detects that the duration of disconnecting the heartbeat communication with another physical node reaches a duration threshold value, the physical node detects whether gateway communication can be carried out or not;
when the physical node determines that gateway communication can be performed and the physical node is the main node, the physical node continuously provides external service of the task scheduling module until the heartbeat communication with another physical node is restored after receiving node attribute information sent by the other physical node;
When the physical node determines that gateway communication can be performed and the physical node is a backup node, the physical node is switched to a master node to have the control right of the virtual IP address of the task scheduling module and provide external service of the task scheduling module, and the physical node resumes heartbeat communication with another physical node after receiving node attribute information sent by the other physical node;
And restarting when the physical node determines that gateway communication cannot be performed, and restarting to send own node attribute information to another physical node after restarting.
The method is further technically characterized in that the node attribute information of each physical node comprises the starting time, version number and node host name of the physical node, and the earlier the starting time of the physical node is, the smaller the version number is, the earlier the dictionary ordering of the node host name is, and the higher the control priority of the physical node is.
The task scheduling module is further technically characterized in that the task scheduling module respectively distributes each task in the task list to be executed to the corresponding task execution module, and the task scheduling module comprises:
Tasks in a task list to be executed are sequentially allocated according to the sequence from high to low of scheduling priority, when each task is allocated, idle resources of each task execution module are determined according to the resource occupation condition of each task execution module recorded in an execution module state list, the resource requirements of the tasks to be allocated are determined, and the tasks to be allocated are allocated to the task execution modules of which the idle resources meet the resource requirements of the tasks;
the scheduling priority of each task in the task list to be executed is determined according to the importance degree of the task, the creation time and the waiting scheduling time of the task, wherein the higher the importance degree of the task, the earlier the creation time and the longer the waiting scheduling time, the higher the scheduling priority of the task.
The task scheduling module is further technically characterized in that the task scheduling module is also used for adding the task execution module and the corresponding resource occupation condition thereof into an execution module state list when detecting that a new task execution module is accessed;
And the task scheduling module is also used for re-distributing the task to the task execution module when the waiting time reaches after the task is distributed to the task execution module and the task execution state fed back by the task execution module is not received, and determining that the task execution module is abnormal when the repetition number of distributing the same task to the same task execution module reaches a frequency threshold value, deleting the task execution module from the execution module state list and re-distributing the task to other task execution modules.
The task scheduling module is further technically characterized by being also used for initializing and generating a timing task meeting the timing starting condition and adding the timing task into a task list to be executed when the timing starting condition of any one timing task in the timing task list is detected;
And the task scheduling module is also used for initializing and generating a manual trigger task according to the manual trigger instruction and adding the manual trigger task to a task list to be executed when the manual trigger instruction is received.
The beneficial technical effects of the application are as follows:
The application discloses a GNSS occultation detection ground application system, which utilizes a task scheduling module to initialize and schedule tasks, utilizes a distributed task execution module to execute tasks, and utilizes an execution module state list to dynamically manage the resource occupation condition of different task execution modules, so that after positioning observation data and occultation observation data in satellite original load data are obtained by analysis, the task scheduling module can flexibly allocate various tasks with huge numbers and different requirements according to the resource condition of different task execution modules, and the task execution efficiency is ensured when the data quantity of the positioning observation data matched with each group of occultation observation data in a distributed file system is detected to reach inversion requirements, thereby ensuring the quality of subsequent data inversion.
For a computationally intensive task, such as a data inversion task, which is controlled to run on a separate CPU core, the execution speed is faster and the response time is shorter than if the computational resources were shared between multiple CPU cores. In addition, the CPU scheduling characteristics of Linux and the transmissibility of environmental variables are fully utilized, the utilization rate of the CPU is maximized, all processes belonging to the same data inversion task can be completely closed when the data inversion task fails, idle process carry-over is avoided, and the implementation mode is simple and reliable.
The task scheduling module in the GNSS occultation detection ground application system uses a virtual routing redundancy protocol to realize dynamic switching of the main and standby nodes, and ensures continuous and stable operation of the system through a main and standby mechanism.
The task list to be executed and the execution module state list are maintained in the ZooKeeper distributed cache, so that multiple backups of state data are realized, and the state data can be recovered in time when faults or parameter damages occur. In addition, the Zookeepers distributed cache avoids single-point faults of the Zookeepers, and as long as more than half of Zookeepers nodes survive, normal operation of main and standby hot switching can be ensured.
Drawings
FIG. 1 is a block diagram illustrating a GNSS occultation detection terrestrial application system according to an embodiment of the present application.
FIG. 2 is a flow chart of an initialization generation data inversion task in one embodiment of the application.
Detailed Description
The following describes the embodiments of the present application further with reference to the drawings.
The application discloses a GNSS occultation detection ground application system, which comprises a task scheduling module, a plurality of task execution modules, a ZooKeeper distributed cache and a distributed file system, referring to FIG. 1, the task scheduling module respectively establishes communication connection with the plurality of task execution modules, the task scheduling module also establishes communication connection with the ZooKeeper distributed cache and the distributed file system, and each task execution module also establishes communication connection with the distributed file system. In addition, the task scheduling module is also connected with a plurality of satellites in the GNSS occultation detection system, so that the original load data sent by each satellite can be received.
The GNSS occultation detection ground application system is built by utilizing a plurality of servers, communication connection is established among the servers through a network switch to form a server cluster, and a task scheduling module, a task executing module, a ZooKeeper distributed cache and a distributed file system in the GNSS occultation detection ground application system are respectively deployed on one or more servers.
The GNSS occultation detection ground application system is used for executing a data management task on the GNSS occultation detection system and comprises the following links:
1. Generation of tasks
Among the data management tasks executed on the GNSS occultation detection system, the most important one is the data inversion task, which is driven by the original load data of each satellite, when the original load data of any one satellite arrives at the GNSS occultation detection ground application system, the GNSS occultation detection ground application system needs to perform data inversion to obtain inversion result data, and the description is as follows:
1. data inversion task
The task scheduling module receives the original load data sent by each satellite of the GNSS occultation detection system, and analyzes the received original load data to obtain positioning observation data (called POD package) and occultation observation data (called OCC package). The parsing process of the original payload data includes error detection, verification, unpacking, packetizing, quality verification and rearrangement, and this part may refer to the existing parsing method of the original payload data, which is not described in detail in this embodiment.
The conventional method generally performs data inversion on the positioning observation data and the occultation observation data obtained by analysis, but in practice, because of different sampling frequencies, satellite transit time and data sizes, the starting time and the ending time of the positioning observation data and the occultation observation data contained in the original load data sent by the same satellite are often different, that is, the starting time and the ending time are not aligned in time. When the data inversion is performed, the time aligned data between the positioning observation data and the occultation observation data is mainly needed, and if the time aligned data is less, the data inversion is easy to fail or the accuracy of the data inversion result is lower.
Therefore, the application does not directly carry out data inversion on the positioning observation data and the occultation observation data obtained by analysis, but stores the positioning observation data obtained by analysis in a distributed file system. And when the data quantity of the positioning observation data matched with each group of occultation observation data in the distributed file system reaches the inversion requirement, initializing and generating a data inversion task according to each group of occultation observation data and the matched positioning observation data, and then adding the generated data inversion task into a task list to be executed. The task list to be executed comprises a plurality of tasks to be executed, and the task list to be executed is maintained in a ZooKeeper distributed cache.
When the method is actually executed, after receiving and analyzing the original load data sent by one satellite to obtain a group of occultation observation data, the task scheduling module detects whether the data volume of positioning observation data matched with the occultation observation data obtained by current analysis in the distributed file system reaches the inversion requirement. Two cases are included, please refer to the flow chart shown in fig. 2:
In the first case, when the data volume of the positioning observation data matched with the occultation observation data obtained by current analysis in the distributed file system reaches the inversion requirement, the positioning observation data matched with the occultation observation data obtained by current analysis is directly extracted from the distributed file system, and then the data inversion task is generated according to the occultation observation data obtained by current analysis and the matched positioning observation data thereof in an initialization mode and is added into a task list to be executed.
And in the second case, when the data quantity of the positioning observation data matched with the occultation observation data obtained by current analysis in the distributed file system does not meet the inversion requirement, correspondingly storing the occultation observation data obtained by current analysis and satellite identifications of satellites for transmitting the occultation observation data in an unprocessed list maintained by the distributed file system.
And then the task scheduling module sequentially traverses each group of occultation observation data of the unprocessed list at preset time intervals, and when detecting that the data quantity of positioning observation data matched with the traversed occultation observation data in the distributed file system reaches the inversion requirement, the task scheduling module initiates to generate a corresponding data inversion task and deletes the traversed occultation observation data from the unprocessed list, otherwise, the traversed occultation observation data is kept in the unprocessed list. The predetermined time interval may be custom set.
In either case, when the task scheduling module stores the positioning observation data obtained by analysis in the distributed file system, each positioning frame and the positioning time of each positioning observation data obtained by analysis and the satellite identification of the satellite transmitting the positioning observation data are correspondingly stored in the distributed file system. Each satellite has a unique satellite identification.
The method for detecting whether the data quantity of the positioning observation data matched with each group of the occultation observation data in the distributed file system meets the inversion requirement by the task scheduling module comprises the step of detecting the quantity of the positioning frames matched with the occultation observation data in the distributed file system. The positioning frame matched with the set of occultation observation data is a positioning frame corresponding to a target satellite identifier, the corresponding positioning time is within the duration of the set of occultation observation data, the duration of one set of occultation observation data is the period from the starting time to the ending time of the set of occultation observation data, and the target satellite identifier is the satellite identifier of the satellite transmitting the set of occultation observation data. Therefore, the obtained positioning frame matched with one group of occultation observation data is the positioning frame which comes from the same satellite as the group of occultation observation data and the positioning moment is within the duration time of the group of occultation observation data.
When the number of the positioning frames matched with the occultation observation data is detected to reach a quantity threshold, determining that the data quantity of the positioning observation data matched with the occultation observation data reaches an inversion requirement, extracting the positioning frames matched with the occultation observation data as the positioning observation data matched with the occultation observation data, otherwise, determining that the data quantity of the positioning observation data matched with the occultation observation data does not reach the inversion requirement. The number threshold may be set in a self-defined manner, and the number threshold is generally positively related to the data amount of the occultation observation data, in one embodiment, the total number N of positioning moments included in the duration of a set of occultation observation data is determined, n×η is taken as the number threshold, η is a scaling factor smaller than 1, the closer the value of η is to 1, and the more the number of positioning frames the generated data inversion task matches the occultation observation data. For example, if the positioning time is in seconds, the total number n=7200 of positioning times included in a set of occultation observation data with a duration of 2 hours is taken as η=0.9, and the number threshold is taken as 6480.
2. Other data management tasks
The data management tasks performed on the GNSS occultation detection system include other data management tasks in addition to the data inversion tasks, and another class is more common to be timing tasks, i.e., tasks that need to be performed at timing. According to the actual service scene, the timing tasks comprise a data backup task, a data export task, a data integrity checking task, a data clearing task, a data subscription task and a data pushing task.
The distributed file system is maintained with a timing task list, the timing task list comprises a plurality of timing tasks and respective corresponding timing starting conditions, and the content and the timing starting conditions of each timing task are self-defined.
The task scheduling module generates a timing task meeting the timing starting condition in an initializing mode and adds the timing task to the task list to be executed when detecting that the timing starting condition of any one timing task in the timing task list is met.
In addition to the data inversion task and the timing task, the data management task is manually triggered, and when a manual trigger instruction is received, the task scheduling module generates the manual trigger task according to the manual trigger instruction in an initializing mode and adds the manual trigger task to a task list to be executed. Manually triggered tasks are typically tasks that are particularly urgent and require temporary execution, and some tasks that fail execution may be re-triggered for execution by an attendant.
In practical application, each satellite can send multiple times of original load data, so that multiple data inversion tasks are generated by initializing each satellite, the GNSS occultation detection system comprises multiple satellites, the data size of the data inversion tasks is larger, various timing tasks and manual triggering tasks which cannot be triggered when the tasks are needed to be triggered are added, the task size of data management tasks to be executed is larger, and different tasks are different in light and heavy urgency requirements, so that proper scheduling of the tasks is needed. As can be seen from the above description, each task to be executed generated by the task scheduling module in the initialization is first added to the task list to be executed.
In addition, each task generated by initialization has respective importance degree, the higher the importance degree is, the more the task needs to be preferentially executed, the importance degree of each task can be configured in a self-defined way, and the importance degree of the task triggered by manual operation is always the highest.
2. Task allocation
And then the task scheduling module respectively distributes each task in the task list to be executed to the corresponding task executing module according to the resource occupation condition of each task executing module recorded in the executing module state list and the scheduling priority of each task in the task list to be executed. The state list of the execution modules is maintained in a ZooKeeper distributed cache, and the state list of the execution modules records the resource occupation condition of each task execution module, wherein the resource occupation condition comprises the total resources of the task execution modules and occupied resources, and the occupied resources of one task execution module comprise the resources occupied by the operating system of the task execution module and the resources occupied by other tasks. Each task execution module is integrated with a plurality of CPU cores, and the resources of one task execution module comprise the CPU cores and a memory.
The state list of the execution module is dynamically maintained, when a new task execution module is accessed to the GNSS occultation detection ground application system, the task execution module reports the resource occupation condition to a task scheduling module, and the task scheduling module adds the newly accessed task execution module and the corresponding resource occupation condition to the state list of the execution module. When a task execution module fails or is down, the task execution module deletes the abnormal task execution module and the resource occupation condition of the abnormal task execution module from the execution module state list.
When the task scheduling module distributes the tasks in the task list to be executed to the task execution module, the tasks in the task list to be executed are distributed in sequence according to the order of the scheduling priority from high to low. The scheduling priority of each task in the task list to be executed is determined according to the importance degree of the task, the creation time and the waiting scheduling time of the task. The higher the importance degree of the task, the earlier the creation time and the longer the waiting scheduling time, the higher the scheduling priority of the task, and the specific scheduling priority determining method can be set in a self-defined manner.
When each task is allocated, the idle resources of each task execution module are determined according to the resource occupation condition of each task execution module recorded in the execution module state list, the resource requirements of the tasks to be allocated are determined, and the tasks to be allocated are allocated to the task execution modules of which the idle resources meet the resource requirements of the tasks.
3. Execution of tasks
Each task execution module is used for executing the received task. When the task execution module executes the data inversion task, the data inversion is performed on the occultation observation data and the matched positioning observation data in the data inversion task to obtain inversion result data, and the inversion result data is stored in the distributed file system, and the specific data inversion method can refer to the existing method, and the embodiment is not repeated.
The data inversion task belongs to a computationally intensive task, and in one embodiment, for each task execution module integrated with multiple CPU cores, the task execution module reserves several CPU cores as basic cores for operating an operating system of the task execution module, and isolates other CPU cores except the basic cores by isolcpus commands. When receiving a data inversion task, the task execution module selects one CPU core from the isolated CPU cores, and controls the received data inversion task to be executed on the selected CPU core by utilizing taskset commands. Therefore, the data inversion task can run on an independent CPU core, frequent context switching among a plurality of CPU cores is avoided, and compared with the method of sharing computing resources among the plurality of CPU cores, the data inversion task is faster in execution speed and shorter in response time.
When a task execution module executes a data inversion task, a plurality of processes are started simultaneously to execute the data inversion task in parallel, and in the process of executing the data inversion task, output information of each process for executing the data inversion task is monitored and recorded in a distributed file system. When the output information of the data inversion task is not read beyond a preset time length, the data inversion task may have the faults of locking or unexpected exit and the like, the data inversion task needs to be closed, and as the data inversion task is processed by a plurality of processes in parallel, the closing of one process alone cannot ensure that all processes for executing the data inversion task are closed, the left idle processes on the task execution module can be caused, and the resource waste can be caused.
To solve this problem, the task execution module defines a unique environment variable for the data inversion task and copies it to each process when starting a plurality of processes to execute a data inversion task in parallel. One way to define environment variables is to assign a task identifier JobId to each data inversion task, then when multiple processes are started to execute the data inversion task in parallel, the task identifier JobId of the data inversion task and the start time startTime of the task are transmitted into a command export sky_lucas_env=sky@ JobId @starttime to define environment variables and copy the environment variables into each process.
And the task execution module closes the failed process when detecting that any process fails in the process of executing the data inversion task, and closes other processes with the same environment variable as the failed process so as to completely close the processes for executing the same data inversion task. The method comprises the specific operation that when a fault occurs in a process is detected, a ps command of a Linux system is called, a list of all processes is obtained, each process is traversed, an output of the command is obtained by executing the command of 'cat/proc/$ { pid }/environ', whether SKY_LUCAS_ENV environment variables containing @ $ { JobId } @ exist or not is checked, if yes, the traversed process and the fault process are used for executing the same data inversion task, and the traversed process and the fault process are closed, so that all processes started by the same data inversion task can be closed.
The task execution module operates according to the task content when executing other tasks, and this embodiment is not described in detail.
Each task execution module also periodically feeds back the task execution status to the task scheduling module, regardless of which type of task is executed. The task scheduling module updates the resource occupation condition of each task execution module recorded in the execution module state list according to the task execution state fed back by each task execution module.
And when the task scheduling module reaches the waiting time after the task is allocated to the task executing module and the task executing state fed back by the task executing module is not received, determining that the task executing module is overtime, and allocating the task to the task executing module again. When the number of times of repeating the same task assigned to the same task execution module reaches a threshold number of times, determining that the task execution module is abnormal, deleting the task execution module from the execution module state list, and reassigning the task to other task execution modules.
As can be seen from the above description of the function, the task scheduling module is the key point in the GNSS occultation detection ground application system of the present application, and the task scheduling module is deployed on the physical nodes, where the physical nodes are single or multiple servers, in the real network environment, the physical nodes deploying the task scheduling module will seriously damage the function of the GNSS occultation detection ground application system once the physical nodes fail, in order to improve the reliability of the GNSS occultation detection ground application system, the task scheduling module adopts a primary and backup redundancy design, the task scheduling module is deployed on two physical nodes respectively, the real IP addresses of the two physical nodes are virtualized into the virtual IP addresses of the task scheduling module by the VRRP (Virtual Router Redundancy Protocol, virtual routing redundancy protocol) technology, and the task scheduling module provides services to the outside through the virtual IP addresses. Only one physical node (the physical node is called a master node) at the same time has the control right of the virtual IP, and the master node provides the service of the task scheduling module. The operation process is as follows:
(1) Each physical node respectively sends own node attribute information to the other physical node when starting, and the node attribute information of each physical node comprises the starting time, version number and node host name of the physical node.
(2) Each physical node determines its own control priority according to its own node attribute information, and determines the control priority of another physical node according to the received node attribute information of the other physical node. Each physical node determines the physical node with the highest control priority as a main node, and the other physical node as a backup node. The earlier the starting time of each physical node is, the smaller the version number is, the earlier the dictionary ordering of the node hostnames is, the higher the control priority of the physical node is, and the specific control priority determining method can be set in a self-defined mode.
(3) The master node has control right of the virtual IP address of the task scheduling module and provides external service of the task scheduling module. In addition, the main node and the backup node perform heartbeat communication, which comprises the steps that the main node sends a heartbeat packet to the backup node, the backup node returns the heartbeat packet to the main node in response to the heartbeat packet of the main node, and each heartbeat packet sent by the physical node comprises role information, node attribute information, state information and the like of the physical node, wherein the role information indicates that the physical node is the main node or the backup node.
(4) When any one physical node detects that the duration of disconnecting the heartbeat communication with another physical node reaches a duration threshold, the physical node or another physical node at the opposite end may have network faults, and at this time, the physical node first detects whether the physical node can perform gateway communication.
When the physical node determines that gateway communication is enabled, it may be determined that another physical node has failed. And when the physical node is the main node, continuing to provide the external service of the task scheduling module until the node attribute information sent after the network connection is restored by the other physical node is received, and restoring the heartbeat communication with the other physical node again.
When the physical node determines that gateway communication can be performed and the physical node is a backup node, it can be determined that a network fault occurs in the current main node, and in order to ensure that the function of the task scheduling module is not affected, the physical node is automatically switched from the backup node to the main node so as to have the control right of the virtual IP address of the task scheduling module and provide external service of the task scheduling module. And the physical node resumes the heartbeat communication with the other physical node when receiving the node attribute information sent after the other physical node resumes the network connection again.
When the physical node determines that gateway communication cannot be performed, the physical node is restarted when network failure occurs, and node attribute information of the physical node is sent to another physical node again after the physical node is restarted.
By using the virtual routing redundancy protocol, the dynamic switching of the main node and the standby node is realized, and the whole GNSS occultation detection ground application system can continuously and stably run through a main-standby mechanism.
The above is only a preferred embodiment of the present application, and the present application is not limited to the above examples. It is to be understood that other modifications and variations which may be directly derived or contemplated by those skilled in the art without departing from the spirit and concepts of the present application are deemed to be included within the scope of the present application.
Claims (8)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411365758.5A CN118885304B (en) | 2024-09-29 | 2024-09-29 | A GNSS occultation detection ground application system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411365758.5A CN118885304B (en) | 2024-09-29 | 2024-09-29 | A GNSS occultation detection ground application system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118885304A CN118885304A (en) | 2024-11-01 |
| CN118885304B true CN118885304B (en) | 2025-04-15 |
Family
ID=93229084
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411365758.5A Active CN118885304B (en) | 2024-09-29 | 2024-09-29 | A GNSS occultation detection ground application system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118885304B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949616A (en) * | 2020-09-08 | 2020-11-17 | 天津云遥宇航科技有限公司 | Ground real-time inversion demonstration system for GNSS occultation data |
| CN117930296A (en) * | 2024-03-22 | 2024-04-26 | 中国科学院国家空间科学中心 | Star-masking data processing method and system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101866021A (en) * | 2010-05-20 | 2010-10-20 | 武汉大学 | Parallelized Abel Transformation Atmospheric Parameter Data Processing Method |
| US10505623B2 (en) * | 2018-03-16 | 2019-12-10 | Vector Launch Inc. | Quality of service level selection for peer satellite communications |
| CN112612040B (en) * | 2020-12-11 | 2022-12-09 | 紫光展锐(重庆)科技有限公司 | GNSS starting method and device of global navigation satellite system |
-
2024
- 2024-09-29 CN CN202411365758.5A patent/CN118885304B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949616A (en) * | 2020-09-08 | 2020-11-17 | 天津云遥宇航科技有限公司 | Ground real-time inversion demonstration system for GNSS occultation data |
| CN117930296A (en) * | 2024-03-22 | 2024-04-26 | 中国科学院国家空间科学中心 | Star-masking data processing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118885304A (en) | 2024-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12481531B2 (en) | Hyper-convergence with scheduler extensions for software-defined container storage solutions | |
| US7987467B2 (en) | Scale across in a grid computing environment | |
| Stellner | CoCheck: Checkpointing and process migration for MPI | |
| US20200073552A1 (en) | Dynamic volume replication factor adjustment | |
| US11334372B2 (en) | Distributed job manager for stateful microservices | |
| CN110727653B (en) | Multi-project load balancing method and device | |
| CN106126346A (en) | A kind of large-scale distributed data collecting system and method | |
| CN110247984B (en) | Service processing method, device and storage medium | |
| CN111552496A (en) | A system and method for seamlessly upgrading sidecars based on adding temporary containers | |
| US11354060B2 (en) | Application snapshot for highly available and distributed volumes | |
| US11803448B1 (en) | Faster restart of task nodes using periodic checkpointing of data sources | |
| CN111506388B (en) | Container performance detection method, container management platform and computer storage medium | |
| CN104506643A (en) | Server of distributed storage system and data synchronizing method between servers | |
| CN116668271B (en) | Container migration method, device, equipment and medium of container cluster management system | |
| CN113055203A (en) | Method and device for recovering abnormity of SDN control plane | |
| CN114268593A (en) | SDN-based space satellite network source routing forwarding method and device | |
| CN110971872A (en) | Video image information acquisition method based on distributed cluster | |
| CN114205333A (en) | IP configuration method, cluster construction method, computer device and storage medium | |
| CN118885304B (en) | A GNSS occultation detection ground application system | |
| US11977909B2 (en) | Hardware placement and maintenance scheduling in high availability systems | |
| US6490586B1 (en) | Ordered sub-group messaging in a group communications system | |
| CN114443580B (en) | Data cleaning method, device, medium and computing equipment | |
| CN114051047B (en) | Session message backup method and device, network equipment and storage medium | |
| CN114356214B (en) | Method and system for providing local storage volume for kubernetes system | |
| US12399746B1 (en) | Dynamic task configuration without task restart |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |