CN106933622A - The Hadoop dispositions methods of model-driven in cloud environment - Google Patents

Abstract

The invention discloses a kind of Hadoop dispositions methods of model-driven in cloud environment, including：The Hadoop dispositions methods of model-driven in a kind of cloud environment；Model conversion between the Hadoop demand models and the Hadoop deployment models is realized according to default transformation rule；The information change situation in the Hadoop demand models and the Hadoop deployment models is monitored using synchronization engine, and information in the Hadoop demand models and/or the Hadoop deployment models carries out synchronizing information when changing.The invention has the advantages that：Diversified software and hardware resources in cloud environment can be managed and deployment.

Description

Model-driven Hadoop deployment method in cloud environment

technical field

The invention relates to the field of software engineering, in particular to a model-driven Hadoop deployment method in a cloud environment.

Background technique

In today's society, a large amount of data traffic is generated every day, and 90% of the world's data is generated in the past two years. Massive data processing technology has been widely used in various fields of social production, which also means that the real big data era arrival.

As an open source software framework for distributed processing of big data, Hadoop can process massive amounts of data in a reliable, efficient, and scalable manner. In addition, with the rapid development of the Hadoop ecosystem and the successive completion of a large number of sub-projects, it has been widely used in the processing and storage of big data in various scenarios. Today, Hadoop has become one of the most important software tools for big data processing. As Hadoop is more and more widely deployed in the cloud, administrators need to deploy and configure Hadoop in different ways according to specific needs, which brings two challenges to Hadoop deployment:

(1) Diversity of hardware resources: Hadoop clusters may be deployed on different types of infrastructure, including physical servers, virtual machines, and Docker containers. This heterogeneity brings difficulty and complexity to the management of cluster nodes.

(2) Diversity of software resources: The Hadoop ecosystem includes many different types of computing and storage frameworks, such as HDFS, MapReduce, HBase, Yarn, Spark, etc. Different types of frameworks have specific deployment and configuration methods. In addition, there are dependencies or constraints between some frameworks.

Currently, some management tools exist to help users deploy Hadoop clusters, such as Cloudera Manager and Apache Ambari. In addition, the open source container engine Docker achieves "one package, run anywhere" at the application component level by managing the life cycle of application components such as packaging, distribution, deployment, and operation, reducing the difficulty of Hadoop deployment and operation and maintenance. Although the above-mentioned deployment tools and technologies provide solutions for the deployment and management of Hadoop clusters, most of the research focuses on the configuration of the environment and the setting of parameters, usually providing a fixed deployment mode, without considering the diversity of cloud platforms Due to the limited infrastructure and scalability issues, it cannot meet user-specific Hadoop deployment requirements based on service types, node resources, and scene characteristics.

Contents of the invention

The present invention aims to solve at least one of the above-mentioned technical problems.

Therefore, the object of the present invention is to propose a model-driven Hadoop deployment method in a cloud environment, which can manage and deploy diversified software and hardware resources in the cloud environment.

In order to achieve the above object, an embodiment of the present invention discloses a model-driven Hadoop deployment method in a cloud environment, comprising the following steps: S1: providing a Hadoop demand model and a Hadoop deployment model, wherein the Hadoop demand model is used for Generate corresponding management views according to system requirements, and the Hadoop deployment model is used to describe the node configuration information, running status and software deployment of the management view; S2: realize the Hadoop demand model and the Hadoop according to preset conversion rules Model conversion between deployment models, wherein the preset conversion rules include a node conversion model and a cluster service conversion model, and the node conversion model is used to realize the node between the nodes of the Hadoop demand model and the Hadoop deployment model Between the model conversion, the cluster service conversion model is used to realize the model conversion between the cluster service of the Hadoop demand model and the cluster service of the Hadoop deployment model; S3: use the synchronization engine to monitor the Hadoop demand model and the cluster service Information changes in the Hadoop deployment model, and information synchronization is performed when the information in the Hadoop demand model and/or the Hadoop deployment model changes.

Further, the Hadoop demand model includes: a cluster node module, the cluster node module is provided with infrastructure resources, and the infrastructure resources include corresponding resources and attributes in the node configuration list, node list and container image list; cluster service module, the cluster service module is provided with a service list, and the service list includes various services and attributes of each service.

Further, the Hadoop deployment model includes: a cluster node unit, which stores a virtual machine configuration list, a virtual machine list, and a virtual machine image list; a cluster service unit, which is used to provide cluster services.

Further, the node conversion model realizes model conversion through the element mapping relationship between the node of the Hadoop demand model and the node of the Hadoop deployment model, the element mapping relationship includes a helper label and a mapper label, and the helper The tag is used to describe the mapping relationship between classes and elements between classes, and the helper tag is used to describe the mapping relationship between classes and attributes between classes.

Further, the cluster service conversion model performs automatic conversion of cluster services through a constraint model and a preset conversion algorithm, the constraint model is used to define the association relationship between multiple model elements, and the preset conversion algorithm is based on the The Hadoop requirements model and the constraints model generate a service deployment solution.

Further, the preset deployment algorithm includes the following steps: according to the service units under the service list in the Hadoop demand model, obtain the set of services that need to be deployed; Supplement and sort the services to obtain the ordered set of services that actually need to be deployed; according to the ordered set of services, read each service in reverse order and calculate the deployment plan of the service; according to the node set of the service deployment unit, Deploy the services sequentially.

Further, the Hadoop deployment model is constructed using the SM@RT tool.

According to the model-driven Hadoop deployment method in the cloud environment of the embodiment of the present invention, the runtime architecture model is introduced into the Hadoop deployment process, and the user-specific Hadoop deployment requirements are met through three steps of model proposal, model conversion and model synchronization.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and comprehensible from the description of the embodiments in conjunction with the following drawings, wherein:

Fig. 1 is the flowchart of the model-driven Hadoop deployment method in the cloud environment of the embodiment of the present invention;

Fig. 2 is the schematic diagram of the Hadoop demand metamodel of an embodiment of the present invention;

Fig. 3 is the schematic diagram of the Hadoop deployment metamodel of an embodiment of the present invention;

Fig. 4 is a schematic diagram of a mapping relationship between model elements according to an embodiment of the present invention;

Fig. 5 is a schematic diagram of a constraint model meta-model according to an embodiment of the present invention;

Fig. 6 is a schematic diagram of a constraint model of an embodiment of the present invention;

Fig. 7 is an explanatory diagram of parameter changes during the operation of Hadoop cluster service deployment in one embodiment of the present invention;

Fig. 8 is a schematic diagram of the two-way synchronization of the Hadoop deployment model and the operating system of an embodiment of the present invention;

Fig. 9 is the schematic diagram of Hadoop demand model in the specific embodiment of the present invention;

Fig. 10 is a schematic diagram of a Hadoop deployment model in a specific embodiment of the present invention.

detailed description

Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

These and other aspects of embodiments of the invention will become apparent with reference to the following description and drawings. In these descriptions and drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited by This restriction. On the contrary, the embodiments of the present invention include all changes, modifications and equivalents coming within the spirit and scope of the appended claims.

The present invention is described below in conjunction with accompanying drawing.

FIG. 1 is a schematic diagram of a model-driven Hadoop deployment method in a cloud environment according to an embodiment of the present invention. As shown in Figure 1, the model-driven Hadoop deployment method in the cloud environment of the embodiment of the present invention comprises the following steps:

S1: Provide Hadoop requirement model and Hadoop deployment model. Among them, the Hadoop requirement model is used to generate corresponding management views according to the system requirements, and the Hadoop deployment model is used to describe the node configuration information, running status and software deployment of the management view;

In one embodiment of the present invention, Hadoop demand model includes:

The cluster node module, the cluster node module is set with infrastructure resources, and the infrastructure resources include the corresponding resources and attributes in the node configuration list, node list and container image list; the cluster service module, the cluster service module is set with a service list, and the service list includes Various services and properties of each service.

Specifically, during the deployment of Hadoop clusters, the Hadoop demand model provides administrators with a unified management view of node resources and cluster services. As shown in Figure 2, the requirements meta-model consists of two parts: cluster nodes and cluster services.

In the cluster node part, Infrastructure represents infrastructure resources, including a NodeTypes element, a Nodes element, and an Images element. The NodeTypes element is a list of node configurations, representing a collection of available configuration files, and each NodeType element represents a node configuration, including id, name, ram, disk, cpus and other attributes, which in turn represent identifiers, names, disks, storage, Information such as the number of CPUs; the Nodes element represents a node list, representing a collection of node configurations, and each Node element represents a node, including attributes such as id, name, nodeType, imgeId, Status, etc., which in turn represent the identifier of the node, the name of the node, Node type, container image, node status and other information; Images is a list of container images, representing a collection of available container image files, and each Image element represents a container image, including id, name, size, status, miniDisk, miniRam, etc. Attributes, which represent the image identifier, image name, image size, image status, disk, storage and other information.

In the cluster service part, the Services element represents the list of services provided by Hadoop, including elements such as HDFSService, MapReduceService, HBaseService, YarnService, and SparkService, respectively representing services such as HDFS, MapReduce, HBase, Yarn, and Spark. Take HDFS Service, MapReduce Service, and HBase Service as examples: HDFSService represents the HDFS service of the cluster, which is provided by multiple nodes in the cluster, and each node deploys a corresponding HDFS software module, that is, HDFSAgent; HBaseService represents the HBase of the cluster Service, which is provided by multiple nodes in the cluster, and each node deploys the corresponding HBase software module, that is, HBaseAgent; MapReduce Service represents the MapReduce service of the cluster, which is provided by multiple nodes in the cluster, and each node deploys The corresponding MapReduce software module is MapReduceAgent. All the above-mentioned services contain id, name, version and status attributes, which respectively represent the identifier, name, version and current running status, and its software deployment module (ie Agent) contains health and nodeId attributes, which represent the health status and software deployment Information such as the node location where the module is located.

In one embodiment of the present invention, the Hadoop deployment model includes: a cluster node unit, which stores a virtual machine configuration list, a virtual machine list and a virtual machine image list; a cluster service unit, which is used to provide cluster services.

Specifically, during the Hadoop cluster deployment process, the deployment model provides administrators with a management view of system deployment, describes the cluster node configuration, running status, and software deployment units on it, and is bidirectionally synchronized with the running system. As shown in Figure 3, the deployment metamodel also includes two parts: cluster nodes and cluster services.

In the cluster node part, taking Cloudstack as an example, Project represents a project, including a VMTypes element, a VMs element, and an Images element. The VMTypes element is a list of virtual machine configurations, representing a collection of configuration files that can be used. Each VMType element represents a virtual machine configuration, including id, name, description, guestCpus, memoryMb, imageSpaceGb and other attributes, which in turn represent identifier, name, virtual Information such as machine description, CPU quantity, memory, image space size, etc.; the VMs element represents a list of virtual machines, representing a collection of virtual machine configurations, and each VM element represents a virtual machine, including id, name, imageId, vmtypeId, cpuutiliz, memoryutiliz, etc. Attributes, indicating identifiers, names, virtual machine images, virtual machine types, CPU usage, memory usage and other information; Images is a list of virtual machine images, indicating a collection of available virtual machine image files, each Image element Represents a virtual machine image, including attributes such as id, name, vsize, and description, and represents information such as identifier, name, image size, and image description.

In the cluster service part, the Services element represents the list of services provided by Hadoop, including HDFSService, MapReduceService, HBaseService, YarnService, SparkService and other elements, respectively representing services such as HDFS, MapReduce, HBase, Yarn and Spark, and each service includes service units and roles There are three main model elements: unit and deployment unit. The service unit represents the specific service provided by Hadoop; the role unit represents the different roles contained in the specific Hadoop service; the deployment unit represents the running software modules owned by different roles. Take HDFSService, MapReduceService, and HBaseService as examples. The HDFS service unit includes three types of role units, NameNode, SecondaryNameNode, and DataNode; among them, the role unit NameNode represents the management center of HDFS, which is used to manage the namespace of the file system, cluster configuration information, and the replication of storage blocks. There is a deployment unit DU_NN; the role unit SecondaryNameNode represents the backup node of the NameNode, which is used to back up the metadata saved by the NameNode node, and there is only one deployment unit DU_SNN; the role unit DataNode represents the working node of HDFS, which is used to schedule storage and retrieve data , there can be multiple deployment units DU_DN. The MapReduce service unit includes two types of role units, namely JobTracker and TaskTracker; among them, the role unit JobTracker represents the central service node of MapReduce, which is used to schedule each subtask task of the Job to run on the TaskTracker, and there is only one The deployment unit DU_JT of the JobTracker; TaskTracker represents a sub-service node, which is used to execute the tasks assigned by the JobTracker, and there can be multiple deployment units DU_TT of the TaskTracker. The HBase service unit includes two types of role units, namely HMaster and HRegionServer; among them, HMaster represents the management and scheduling center of HBase, which is used to allocate and manage HRegionServer, and there is only one deployment unit DU_HM of HMaster; HRegionServer represents that HBase runs on each The service on a working node is used to maintain the Region and IO requests allocated by the HMaster. There can be multiple HRegionServer deployment units DU_RS. The above-mentioned deployment units of different roles all include vmId and health attributes, which represent information such as the running health status of the deployment unit and the location of the virtual machine where the software module is located.

S2: Realize the model conversion between the Hadoop requirement model and the Hadoop deployment model according to the preset conversion rules. Among them, the preset conversion rules include a node conversion model and a cluster service conversion model. The node conversion model is used to realize the model conversion between the nodes of the Hadoop demand model and the nodes of the Hadoop deployment model, and the cluster service conversion model is used to realize the Hadoop demand model. Model conversion between cluster services and cluster services for Hadoop deployment models.

In one embodiment of the present invention, the node conversion model realizes model conversion through the element mapping relationship between the nodes of the Hadoop demand model and the nodes of the Hadoop deployment model. The element mapping relationship includes helper tags and mapper tags, and the helper tags are used to describe The mapping relationship between classes and elements between classes, and the helper tag is used to describe the mapping relationship between classes and attributes between classes.

Specifically, in different application scenarios, there are great differences in the cluster node part in the deployment model. For example, in CloudStack, the main management units include VM, Flavor, and Image, which represent virtual machines, configuration files, and images; while in Docker, the main management units are Container, Repository, and Image, which represent containers, warehouses, and images. Therefore, it is necessary to establish the element mapping relationship between the requirement model and the node part of the deployment model to realize model conversion.

Embodiments of the present invention design a set of mapping relationship description rules and model operation conversion methods, and automatically convert the requirement model to the deployment model according to the element mapping relationship between the given models. The element mapping relationship between models is described through an XML file, and the definitions of keywords in the description rules are as follows:

(1) helper is used to describe the mapping relationship between classes and elements between classes. The helper tag contains two attributes: value and key, value indicates the element to be mapped in the requirement model, and key indicates the corresponding element in the deployment model.

(2) mapper is used to describe the mapping relationship between classes and attributes between classes. The mapper tag also contains two attributes, key and value. The key indicates the name of the element attribute that the deployment model should be mapped to, and the value indicates the name of the corresponding attribute of the element in the requirement model. The elements they belong to are respectively defined by the helper tag of the upper layer.

Based on the above keywords, the mapping relationship between the requirements model proposed by the present invention and the elements in the deployment model can be described according to the mapping rules. As shown in Figure 4, the NodeType element in the requirement model is mapped to the VMType element in the deployment model, which is described by a helper tag. The key attribute and value attribute of the helper tag are the names of the corresponding elements of the deployment model and the requirement model, namely VMType and NodeType. Among them, the id of NodeType corresponds to the id of VMType, name corresponds to name, ram corresponds to memoryMb, disk corresponds to imageSpaceGb, and vcpus corresponds to guestCpus.

The management of the system can be realized through the model operation, and the model operation includes five basic types, namely Get, Set, List, Add and Remove. Any model operation on a requirement model element will be converted to a model operation on a deployment model element. As shown in Table 1, the present invention defines conversion rules for model operations and realizes automatic conversion of model operations. For example, the A element in the requirement model is mapped to the B element in the deployment model, then the List, Add, and Remove operations on the A element will be mapped to the same operation on the corresponding B element, and the Get or Set operation on the A attribute will also be will be mapped to the same operation on the corresponding attribute.

Table 1 Mapping rules for model operation conversion

In one embodiment of the present invention, the cluster service conversion model performs automatic conversion of cluster services through a constraint model and a preset conversion algorithm, the constraint model is used to define the association relationship between multiple model elements, and the preset conversion algorithm is based on Hadoop requirements Model and constraint models generate service deployment scenarios.

Specifically, in the Hadoop ecosystem, HDFS, MapReduce, Hbase, Yarn, Spark and other different types of computing and storage frameworks are included. These computing or storage frameworks have specific deployment and configuration methods, and different frameworks There may be dependencies or constraints. For example, deploying MapReduce services needs to depend on HDFS services. Therefore, for different Hadoop cluster services, the model conversion methods are quite different. The invention describes the deployment rules of the cluster service through a constraint model, and realizes the automatic conversion of the cluster service through a general algorithm.

The constraint model describes the deployment rules of a Hadoop service. As shown in Figure 5, the metamodel of the constraint model includes several main model elements such as service unit, role unit, and deployment unit, and describes the relationship between them. Among them, Service represents a service unit, including name and minNodeNum attributes, which in turn represent the name and the minimum number of deployment nodes; Role represents a role unit, including name, excluName, resPriority, deOrder and other attributes, which represent name, exclusive attribute, resource priority and deployment in turn Sequence; DU indicates the deployment unit, including health and other attributes, indicating the health status of the service; Dependency_S indicates the dependency relationship between service units, including the name attribute, indicating the name of the service unit on which it depends; similarly, Dependency_DU indicates the relationship between deployment units Dependencies, the name attribute indicates the name of the deployment unit on which it depends. When there is a dependency relationship between service units, there may not be a dependency relationship between deployment units; however, when there is a dependency relationship between deployment units, there must be a dependency relationship between service units.

As shown in Figure 6, HDFS, MapReduce, HBase and other services are taken as examples to illustrate. HDFS service unit includes three role units, NameNode, SeconderyNameNode and DataNode. Among them, the role unit NameNode has one and only one deployment unit DU_NN, the role unit SeconderyNameNode has one and only one deployment unit DU_SNN, and the NameNode and SeconderyNameNode cannot be deployed on the same node; the role unit DataNode can have multiple deployment units DU_DN, and the DataNode can It is deployed on the same node as NameNode or SecondaryNameNode; in addition, HDFS service units have no dependencies on other service units, and deployment units DU_NN, DU_SNN, and DU_DN have no dependencies on other deployment units. The MapReduce service unit includes two role units, JobTracker and TaskTracker; among them, the role unit JobTracker has one and only one deployment unit DU_JT; the role unit TaskTracker can have multiple deployment units DU_TT, and the JobTracker and TaskTracker cannot be deployed on the same node; , the MapReduce service unit is dependent on the HDFS service unit, the deployment unit DU_JT is dependent on the deployment unit DU_NN, and the deployment unit DU_TT is dependent on the deployment unit DU_DN. The HBase service unit includes two role units, namely HMaster and HRegionServer; among them, the role unit HMaster has one and only one deployment unit DU_HM; the role unit HRegionServer can have multiple deployment units DU_RS, and HMaster and HRegionServer cannot be deployed on the same node; in addition , the HBase service unit is dependent on the HDFS service unit, the deployment unit DU_HM is dependent on the deployment unit DU_NN, and the deployment unit DU_RS is dependent on the deployment unit DU_DN.

In one embodiment of the present invention, the preset deployment algorithm includes the following steps: according to the service units under the service list in the Hadoop demand model, obtain the set of services that need to be deployed; The services in the collection are supplemented and sorted to obtain the ordered collection of services that actually need to be deployed; according to the ordered collection of services, each service is read in reverse order and the deployment plan of the service is calculated; according to the node collection of the service deployment unit, Deploy the services sequentially.

Specifically, the embodiment of the present invention proposes a general algorithm, which can automatically generate a service deployment plan according to a given demand model and constraint model. The basic steps of the algorithm are as follows:

1. According to the service units under the service list services in the demand model, obtain the service set services {S1, S2, S3, ..., Si} that need to be deployed.

2. According to the dependency relationship between the service units in the constraint model, supplement and sort the services in the service set services to obtain the ordered set servicesOrder{S1,S2,S3,...,Sj} of the services that actually need to be deployed; If other services that a service depends on in the service set do not appear in the service set, supplements need to be made; in the ordered set servicesOrder, service S1 does not depend on other services, service S2 can depend on service S1, and service S3 can Depends on service S1 and service S2, and so on.

3. According to the ordered set servicesOrder, read each service in reverse order and calculate the deployment plan of the service: First, get the service according to the information in the software deployment module (ie Agent) of the currently deployed service in the demand model secondly, according to the constraint model, obtain the role types and deployment constraints contained in the service; then, calculate the nodes deployed by each role's deployment unit according to the node information and role deployment constraints; finally, according to the constraint model The dependency relationship of the deployment unit of the role records the deployment node information of the deployment unit it depends on, and obtains the node set of the deployment unit of each service in the ordered set servicesOrder, that is, servicesOrderDeploy{S1{DU1{},DU2{},.. .},S2{DU1{},DU2{},...},...,Sn{DU1{},DU2{},...}}, for example, a deployment scenario that includes HDFS and MapReduce services The node set of the deployment unit is servicesOrderDeploy{HDFSService{DU_NN{1},DU_SNN{2},DU_DN{1,2,3}},MapReduceService{DU_JT{1},DU_TT{2,3}}}.

4. According to the node set servicesOrderDeploy of the service deployment unit, the services are deployed sequentially.

S3: Use the synchronization engine to monitor information changes in the Hadoop demand model and Hadoop deployment model, and perform information synchronization when the information in the Hadoop demand model and/or Hadoop deployment model changes.

In one embodiment of the present invention, the Hadoop deployment model is constructed using the SM@RT tool.

Specifically, the runtime model uses a group of manageable units to represent the overall architecture of the system, and explicitly describes the runtime information such as the structure, status, and configuration hidden inside the system as a standard, manager-oriented structured model. view.

The present invention adopts SM@RT tool to construct Hadoop deployment model. Given the metamodel and access model of the system, the metamodel describes the information of the managed system, and the access model describes the calling method of the management interface. Based on the above inputs, the runtime architecture of the target system can be automatically generated by the SM@RT tool , and the two-way consistency between it and the system can also be guaranteed.

As shown in Figure 7, the present invention provides a set of operations about Hadoop cluster service deployment, summarizes the operations of different types of model elements and classifies the model operations, and for each operation, the present invention specifies the operation name requirements The parameters and operations provided change.

SM@RT supports two-way synchronization between the Hadoop deployment model and the running system. As shown in Figure 8, when the synchronization engine detects that a deployment unit has been added to the role unit of a certain service in the Hadoop deployment model, it automatically adds a virtual machine to the running system for deployment; when the administrator deletes this When running a virtual machine, the synchronization engine can also automatically detect this change, and call the management interface to terminate the corresponding deployment unit.

In order for those skilled in the art to further understand the present invention, the following examples will be described in detail.

In order to verify the feasibility and effectiveness of the method proposed in the present invention, an example of automatic deployment and configuration of Hadoop is realized, and a solution of user-customized Hadoop service is provided, including the deployment of MapReduce service and HBase service.

The experiment deploys a Hadoop cluster on CloudStack, and uses five virtual machines in CloudStack to deploy MapReduce and HBase services. Among them, the MapReduce service is deployed on the three virtual machines Host-1, Host-2, and Host-3; the HBase service is deployed on the three virtual machines Host-1, Host-4, and Host-5. In addition, different virtual machines are assigned different resources such as CPU, memory, and storage. See Table 2 for specific configuration details and deployment conditions.

Table 2 Node deployment

The user only needs to define the demand model of the Hadoop cluster, and the method proposed by the invention can realize the automatic conversion from the demand model to the deployment model, and finally complete the cluster deployment.

As shown in Figure 9, the requirement model includes a cluster node part and a cluster service part.

In the cluster node section, NodeTypes represents a list of node types, including three node types, namely "CPU: 4, Memory: 8, Storage: 400", "CPU: 2, Memory: 8, Storage: 400" and "CPU: 2. Memory: 4, Storage: 800"; Nodes represents the node list, including 5 nodes, among which, the node type of Node with id 1 corresponds to NodeType with id nt1, and the node types of Node with id 2 and 3 correspond to id The NodeType is nt2, and the node types of Node with id 4 and 5 correspond to the NodeType with id nt3.

In the cluster service part, MapReduce and HBase services are deployed on their corresponding three nodes.

The conversion from the demand model to the deployment model is divided into two parts, namely the model conversion of cluster nodes and the model conversion of cluster services.

In the cluster node part, the NodeType element and the Node element in the requirement model are mapped to the VMType element and the VM element in the deployment model respectively. Therefore, there are also three virtual machine types (ie VMType) and 5 virtual machines (ie VM) in the deployment model, as shown in FIG. 10 .

In the cluster service part, according to the constraint model and general algorithm proposed by the present invention, the model conversion process is as follows:

1. According to the service unit under the service list services in the demand model, get the service collection services {MapReduceService, HBaseService} to be deployed;

2. According to the dependency relationship between MapReduceService and HBaseService in the constraint model, there is no dependency relationship between MapReduceService and HBaseService, but both services depend on HDFSService. At this time, HDFSService does not appear in the service collection. Therefore, you need to Add HDFSService to the service collection services and perform deployment sorting to obtain the ordered collection servicesOrder{HDFSService, MapReduceService, HBaseService} of the services that actually need to be deployed;

3. According to the ordered set servicesOrder, read HBaseService, MapReduceService, HDFSService in reverse order and calculate the deployment plan of the service: First, according to the information in the software deployment module (ie Agent) of HBaseService in the demand model, get the HBase service Deployment node list, nodes 1, 4, and 5; Secondly, according to the constraint model, it can be seen that HBaseService includes two role units, HMaster and HRegionServer, HMaster has one and only one deployment unit DU_HM, and HRegionServer can have multiple deployment units DU_RS, and HMaster and HRegionServer HRegionServer cannot be deployed on the same node. In addition, the deployment unit DU_HM has a dependency on the deployment unit DU_NN of the HDFS service, and the deployment unit DU_RS has a dependency on the deployment unit DU_DN of the HDFS service. Then, according to the node information and role deployment constraints, it is calculated that The resource priority and deployment order of HMaster and NameNode are the highest, so these two role units are deployed on the resource-rich node 1, and the role units HRegionServer and DataNode are deployed on nodes 4 and 5; similarly, when deploying MapReduceService When using the demand model and constraint model, it can be calculated that the role units JobTracker and NameNode are deployed on node 1, and the role units TaskTracker and DataNode are deployed on nodes 2 and 3; when HDFSService is deployed, it can be seen from the constraint model that HDFSService does not depend on Other services, and the NameNode and SecondaryNameNode cannot be deployed on the same node. According to the deployment node information of HBaseService and MapReduceService, the NameNode is deployed on node 1, the SecondaryNameNode is deployed on node 2, and the DataNode is deployed on nodes 1-5. Finally, according to the constraints The dependency relationship of the deployment unit in the model records the deployment node information of the deployment unit it depends on, and obtains the node set of the deployment unit of each service in the ordered set servicesOrder, that is, servicesOrderDeploy{HDFSService{DU_NN{1},DU_SNN{2},DU_DN {1,2,3,4,5}},MapReduce Service{DU_JT{1},DU_TT{2,3}},HBaseService{DU_HM{1},DU_RS{4,5}}}.

4. According to the node set servicesOrderDeploy of the service deployment unit, the services are deployed sequentially.

As shown in Figure 10, the specific Hadoop deployment model is obtained from the model transformation of the cluster nodes and the model transformation of the cluster service.

The model-driven Hadoop deployment method in the cloud environment of the present invention aims at the difficulties brought by the diversified software and hardware resources and service on-demand in the cloud environment to deploy Hadoop clusters, and proposes a model-driven Hadoop cluster service deployment method. By proposing Hadoop The conversion method from the demand model to the deployment model and the two-way synchronization between the deployment model and the operating system provide a fast and scalable cluster service deployment method for Hadoop deployment, and verify the feasibility and effectiveness of the present invention in actual scenarios .

In addition, other components and functions of the model-driven Hadoop deployment method in the cloud environment of the embodiment of the present invention are known to those skilled in the art, and will not be repeated in order to reduce redundancy.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions and modifications can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the invention is defined by the claims and their equivalents.

Claims (7)

1. in a kind of cloud environment model-driven Hadoop dispositions methods, it is characterised in that comprise the following steps：

S1：Hadoop demand models and Hadoop deployment models are provided, wherein, the Hadoop demand models are used for basis System requirements generate corresponding administration view, and the Hadoop deployment models are used to describe the node for attempting that manages with confidence Breath, running status and software are disposed；

S2：Realize that the model between the Hadoop demand models and the Hadoop deployment models turns according to default transformation rule Change, wherein, the default transformation rule includes node transformation model and cluster service transformation model, and the node transformation model is used Model conversion between the node of the node for realizing the Hadoop demand models and the Hadoop deployment models, the collection Group's service transformation model is used to realize the cluster service of the Hadoop demand models and the cluster of the Hadoop deployment models Model conversion between service；

S3：The information change feelings in the Hadoop demand models and the Hadoop deployment models are monitored using synchronization engine Condition, and information in the Hadoop demand models and/or the Hadoop deployment models to enter row information when changing same Step.

2. in cloud environment according to claim 1 model-driven Hadoop dispositions methods, it is characterised in that it is described Hadoop demand models include：

Clustered node module, the clustered node module is provided with infrastructure resources, and the infrastructure resources include node Configured list, node listing and corresponding resource and attribute in container image list；

Cluster service module, the cluster service module is provided with service list, the service list include various services and The attribute of every kind of service.

3. in cloud environment according to claim 2 model-driven Hadoop dispositions methods, it is characterised in that it is described Hadoop deployment models include：

Clustered node unit, the clustered node unit is stored with virtual machine configuration list, virtual machine list and virtual machine image List；

Cluster service unit, the cluster service unit is used to provide cluster service.

4. in cloud environment according to claim 3 model-driven Hadoop dispositions methods, it is characterised in that the node Element between node and the node of the Hadoop deployment models that transformation model passes through the Hadoop demand models maps Relation carrys out implementation model conversion, and the element mapping relations include helper labels and mapper labels, the helper labels Mapping relations for describing element between class and class, the mapping that the helper labels are used to describe attribute between class and class is closed System.

5. in cloud environment according to claim 3 model-driven Hadoop dispositions methods, it is characterised in that the cluster Service transformation model carries out the automatic conversion of cluster service by restricted model and default transfer algorithm, and the restricted model is used for Limit the incidence relation between multiple model elements, the default transfer algorithm according to the Hadoop demand models and it is described about Beam model generates service arrangement scheme.

6. in cloud environment according to claim 5 model-driven Hadoop dispositions methods, it is characterised in that it is described default Deployment Algorithm is comprised the following steps：

According to the service unit under service list in the Hadoop demand models, obtain needing the set of service of deployment；

According to the dependence between service unit in restricted model, the service in set of service is supplemented and sorted, obtained To the service ordered set for being actually needed deployment；

According to the service ordered set, each deployment side for servicing and calculating service is successively read according to the mode of backward Case；

According to the node set of service arrangement unit, the deployment for being serviced successively.

7. in cloud environment according to claim 1 model-driven Hadoop dispositions methods, it is characterised in that using SM@ RT instruments construct the Hadoop deployment models.