Ambari
  1. Ambari
  2. AMBARI-1783

Specification for a cluster blueprint consumable by Ambari

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Deployment of a hadoop cluster can be modeled as the mapping of a stack definition to a set of host while satisfying placement constraints. A stack definition refers to the description of various services that comprise a hadoop stack (e.g. HDP-1.2.1), components associated with the services, and configuration properties. A set of available hosts is specified through a host-manifest that lists the available hosts and relevant properties/characteristics of each host. A cluster blueprint is specification of a hadoop stack mapped to a set of hosts. Mapping to a host can be a direct mapping - e.g. deploy this component X on host Y or a host can be specified as a set of requirements which is used to select the most appropriate host(s) from a host-manifest.

      1. Ambari-BluePrint-0.3.docx
        156 kB
        Sumit Mohanty
      2. HostComponentMapping.txt
        2 kB
        Sumit Mohanty
      3. HostManifest.txt
        2 kB
        Sumit Mohanty
      4. PackageConfiguration.txt
        2 kB
        Sumit Mohanty
      5. Schema_HostManifest.txt
        4 kB
        Sumit Mohanty
      6. Schema_PackageConfiguration.txt
        5 kB
        Sumit Mohanty
      7. Sample_Deployment1_HostComponentMapping.txt
        3 kB
        Sumit Mohanty
      8. Sample_Deployment1_HostManifest.txt
        2 kB
        Sumit Mohanty
      9. Sample_Deployment1_HDPStack_1_3_0_Configuration.txt
        1 kB
        Sumit Mohanty
      10. PackageDefinition.txt
        4 kB
        Sumit Mohanty
      11. Schema_HostComponentMapping.txt
        10 kB
        Sumit Mohanty
      12. Sample_HDPStack_1_3_0_Definition.txt
        73 kB
        Sumit Mohanty
      13. Schema_PackageDefinition.txt
        10 kB
        Sumit Mohanty
      14. Ambari-BluePrint-0.4.docx
        158 kB
        Sumit Mohanty

        Issue Links

          Activity

          Hide
          Brian Swan added a comment -

          This is a great start. Some thoughts...

          • Allowing the various sections (Stack Def., Stack Conf., Host Manifest, Host-Component Mapping) to be consumable separately would be helpful for a number of reasons:
          ○ Stack Def will be written by a stack author. A cluster admin (someone deploying a cluster) does not need to be concerned with these. He does need to be concerned with the Host Manifest and Host-Component Mapping (which describe the topology of the cluster).
          ○ In some cases, host details (as specified in the Host Manifest) will be controlled by other tools (such as System Center Virtual Machine Manger or an agent in a Cloud environment).
          ○ In some cases, Host-Component Mapping will be controlled by a tool (such as System Center Virtual Machine Manager, or an agent in a Cloud environment) and will not depend on this content in the blueprint (i.e. it needs to be ignorable).
          ○ Introducing a level of abstraction to the Host Manifest and Host-Component Mapping content that allows a cluster admin to define the topology of a cluster without specifying host details of mapping details would help with the two previous points. For example:

          "Node": [

          { "Name": "Master", "Count": 1, "Component": [ "openjdk7", "core", "namenode", "resourcemanager"], "Type": "HeadNode" }

          ,

          { "Name": "Slave", "Count": 4, "Component": [ "openjdk7", "core", "datanode", "nodemanager", "drawbridge"], "Type": "WorkerNode" }

          ],

          • Providing for an option to instal custom software (such as Python or a specific version of the JDK) across a cluster could be very useful. This could be as simple as a reference to a script that knows how to install software. For example, this could be part of the Stack Definition (though this doesn't adhere exactly to the proposed syntax):

          "Service": [
          {
          "Name": "OracleJDK",
          "Component": [
          {
          "Cardinality": "*",
          "IsRequired": false,
          "IsServerComponent": false,
          "Name": "OracleJDK",
          "Package":

          { "Type": "custom", "Name": "OracleJDK", "ScriptFileName": "OracleJDK.ps1" }

          }
          ]
          },

          • Resource URIs in the Stack Configuration would allow for identification of where distributions (whether they be Hadoop distributions or custom software) could be downloaded. In the example below, the Resources identify where the hdinsight and python packages should be downloaded from:

          "Distros": [

          { "Name": "hdinsight", "Version": "1.0.1", "FileName": "hdp-azure-1.0.1.package.zip" }

          ,

          { "Name": "python", "Version": "2.6", "FileName": "python.zip"}

          ],
          "Resources": [

          {"HDP":"http://mycontainer.blob.core.windows.net/hdp-azure-1.01.package.zip"}

          ,

          {"Python":"http://mycontainer.blob.core.windows.net/python.zip"}

          ]
          ],

          • A Users section in the Stack Configuration would allow for users to be created on machines in the cluster as they are provisioned. Hadoop services would run as these users. This section could be ignored when it isn't needed.
          "Users": [
                 

          {             "Groups": [                 "hadoopadmin"             ],             "Name": "hadoopuser1",             "Password": "pwd",             "Type": "local"         }


              ],

          Show
          Brian Swan added a comment - This is a great start. Some thoughts... • Allowing the various sections (Stack Def., Stack Conf., Host Manifest, Host-Component Mapping) to be consumable separately would be helpful for a number of reasons: ○ Stack Def will be written by a stack author. A cluster admin (someone deploying a cluster) does not need to be concerned with these. He does need to be concerned with the Host Manifest and Host-Component Mapping (which describe the topology of the cluster). ○ In some cases, host details (as specified in the Host Manifest) will be controlled by other tools (such as System Center Virtual Machine Manger or an agent in a Cloud environment). ○ In some cases, Host-Component Mapping will be controlled by a tool (such as System Center Virtual Machine Manager, or an agent in a Cloud environment) and will not depend on this content in the blueprint (i.e. it needs to be ignorable). ○ Introducing a level of abstraction to the Host Manifest and Host-Component Mapping content that allows a cluster admin to define the topology of a cluster without specifying host details of mapping details would help with the two previous points. For example: "Node": [ { "Name": "Master", "Count": 1, "Component": [ "openjdk7", "core", "namenode", "resourcemanager"], "Type": "HeadNode" } , { "Name": "Slave", "Count": 4, "Component": [ "openjdk7", "core", "datanode", "nodemanager", "drawbridge"], "Type": "WorkerNode" } ], • Providing for an option to instal custom software (such as Python or a specific version of the JDK) across a cluster could be very useful. This could be as simple as a reference to a script that knows how to install software. For example, this could be part of the Stack Definition (though this doesn't adhere exactly to the proposed syntax): "Service": [ { "Name": "OracleJDK", "Component": [ { "Cardinality": "*", "IsRequired": false, "IsServerComponent": false, "Name": "OracleJDK", "Package": { "Type": "custom", "Name": "OracleJDK", "ScriptFileName": "OracleJDK.ps1" } } ] }, • Resource URIs in the Stack Configuration would allow for identification of where distributions (whether they be Hadoop distributions or custom software) could be downloaded. In the example below, the Resources identify where the hdinsight and python packages should be downloaded from: "Distros": [ { "Name": "hdinsight", "Version": "1.0.1", "FileName": "hdp-azure-1.0.1.package.zip" } , { "Name": "python", "Version": "2.6", "FileName": "python.zip"} ], "Resources": [ {"HDP":"http://mycontainer.blob.core.windows.net/hdp-azure-1.01.package.zip"} , {"Python":"http://mycontainer.blob.core.windows.net/python.zip"} ] ], • A Users section in the Stack Configuration would allow for users to be created on machines in the cluster as they are provisioned. Hadoop services would run as these users. This section could be ignored when it isn't needed. "Users": [         {             "Groups": [                 "hadoopadmin"             ],             "Name": "hadoopuser1",             "Password": "pwd",             "Type": "local"         }     ],
          Hide
          Sumit Mohanty added a comment -

          Updated blue-prints.

          Show
          Sumit Mohanty added a comment - Updated blue-prints.
          Hide
          Sumit Mohanty added a comment -

          Thanks Brian for the comments. I have modified the blue-print support based on the comments.

          Show
          Sumit Mohanty added a comment - Thanks Brian for the comments. I have modified the blue-print support based on the comments.
          Hide
          Sumit Mohanty added a comment -

          Modified the documents with latest comments.

          Show
          Sumit Mohanty added a comment - Modified the documents with latest comments.
          Hide
          Sumit Mohanty added a comment -

          Incorporated latest comments ...

          Show
          Sumit Mohanty added a comment - Incorporated latest comments ...
          Hide
          Sumit Mohanty added a comment -

          Updated with latest comments as well as schemas and samples.

          Show
          Sumit Mohanty added a comment - Updated with latest comments as well as schemas and samples.
          Hide
          Brian Swan added a comment -

          The latest blueprints look good. I'm still in the process of reviewing them, but I've come across one thing that would be nice to add: In hosted cloud scenarios, there are some services (such as Java or Python), that aren't manageable. It would be nice so have some way to distinguish these services from those that are manageable. I suggest adding an "isManagable":Boolean property to each service in the stack definition. For example:

          ...

          "services": [
          {
          "name": "OpenJDK",
          "isManageable":"false"
          "components": [
          {
          ...

          Show
          Brian Swan added a comment - The latest blueprints look good. I'm still in the process of reviewing them, but I've come across one thing that would be nice to add: In hosted cloud scenarios, there are some services (such as Java or Python), that aren't manageable. It would be nice so have some way to distinguish these services from those that are manageable. I suggest adding an "isManagable":Boolean property to each service in the stack definition. For example: ... "services": [ { "name": "OpenJDK", "isManageable":"false" "components": [ { ...
          Hide
          Sumit Mohanty added a comment -

          Changes:

          • Added support for new flags and service context for service definition
          • Made corresponding changes to the schema and samples
          Show
          Sumit Mohanty added a comment - Changes: Added support for new flags and service context for service definition Made corresponding changes to the schema and samples
          Hide
          Eric Yang added a comment -

          OpenJDK is not a service. It is a package instead of a service. Firewall could be an example of isManageable service. How about simplify the cluster blue print, software stack definition and host manifest? A cleaner version would be:

          Cluster blueprint:

          { 
            "version": "1.3.0",
            "cluster": {
              "name": "example",
              "services": [
                {
                  "name": "filesystem",
                  "components": [ "hdfs" ],
                },
                {
                  "name": "firewall",
                  "components": [ "iptables" ],
                  "isManageable": "false"
                }
              ],
              "applications": [
                {
                  "name": "my hadoop killer app",
                  "components": [ "text-analytics" ],
                  "isManageable": "false"
                }
              ],
              "environment": {
                "fs.default.name": "hdfs://localhost:50070/",
                "HADOOP_NAMENODE_OPTS": { "value": "-Xmx5G", "file": "hadoop-env.sh" }
              }
            }
          }
          

          Software stack example:

          {
            "software": {
              "repository": [
                {
                  "name": "local",
                  "url": "file:///image"
                }
              ],
              "platform": "x86_64",
              "osType": "el6",
              "components": [
                {
                  "name":"glusterfs",
                  "conflicts": [ "hdfs" ],
                  "provides" : "filesystem",
                  "packages": [
                    {
                      "name":"glusterfs",
                      "version":"4.3",
                      "filename":"gluster-4.3.el6.x86_64.rpm",
                      "hostGroup": [ "node" ]
                    }
                  ]
                },
                {
                  "name":"hdfs",
                  "conflicts": [ "glusterfs" ],
                  "provides" : "filesystem",
                  "packages": [
                    {
                      "name":"java",
                      "version":"1.7",
                      "filename":"java-1.7.el6.x86_64.rpm"
                    },
                    {
                      "name":"hadoop",
                      "version":"1.2.0",
                      "filename":"hadoop-1.2.0.el6.x86_64.rpm",
                      "dependency":"java"
                      "hostGroup": [ "namenode", "secondarynamenode", "datanode" ]
                    }
                  ]
                },
                {
                   "name":"text-analytics",
                   "packages": [
                     {
                       "name":"hadoop",
                       "version":"1.2.0",
                       "text-analytics-1.2.0.el6.x86_64.rpm"
                       "dependency":"filesystem"
                     }
                   ]
                }
              ]
            }
          }
          

          Host manifest:

          {
            "namenode": [ "nn1" ],
            "secondarynamenode": [ "snn1" ],
            "datanode": [ "dn1", "dn2", "dn3" ]
          }
          

          Notice in cluster blueprint, the new addition section of applications is designed to deploy application in the hadoop environment (HDFS) or virtual environment, there is no local copy stored in local file system. The running environment is a flatten list of key value pairs, this is designed to make it easier for dev ops to manage the running environment more easily. The additional overwrite to environment allows to set a specific value for a particular config file. Host manifest is a flatten list of hostGroups for user friendliness for dev ops. Conflict, and provide syntax can support swappable component for Hadoop software stack.

          Show
          Eric Yang added a comment - OpenJDK is not a service. It is a package instead of a service. Firewall could be an example of isManageable service. How about simplify the cluster blue print, software stack definition and host manifest? A cleaner version would be: Cluster blueprint: { "version" : "1.3.0" , "cluster" : { "name" : "example" , "services" : [ { "name" : "filesystem" , "components" : [ "hdfs" ], }, { "name" : "firewall" , "components" : [ "iptables" ], "isManageable" : " false " } ], "applications" : [ { "name" : "my hadoop killer app" , "components" : [ "text-analytics" ], "isManageable" : " false " } ], "environment" : { "fs. default .name" : "hdfs: //localhost:50070/" , "HADOOP_NAMENODE_OPTS" : { "value" : "-Xmx5G" , "file" : "hadoop-env.sh" } } } } Software stack example: { "software" : { "repository" : [ { "name" : "local" , "url" : "file: ///image" } ], "platform" : "x86_64" , "osType" : "el6" , "components" : [ { "name" : "glusterfs" , "conflicts" : [ "hdfs" ], "provides" : "filesystem" , "packages" : [ { "name" : "glusterfs" , "version" : "4.3" , "filename" : "gluster-4.3.el6.x86_64.rpm" , "hostGroup" : [ "node" ] } ] }, { "name" : "hdfs" , "conflicts" : [ "glusterfs" ], "provides" : "filesystem" , "packages" : [ { "name" : "java" , "version" : "1.7" , "filename" : "java-1.7.el6.x86_64.rpm" }, { "name" : "hadoop" , "version" : "1.2.0" , "filename" : "hadoop-1.2.0.el6.x86_64.rpm" , "dependency" : "java" "hostGroup" : [ "namenode" , "secondarynamenode" , "datanode" ] } ] }, { "name" : "text-analytics" , "packages" : [ { "name" : "hadoop" , "version" : "1.2.0" , "text-analytics-1.2.0.el6.x86_64.rpm" "dependency" : "filesystem" } ] } ] } } Host manifest: { "namenode" : [ "nn1" ], "secondarynamenode" : [ "snn1" ], "datanode" : [ "dn1" , "dn2" , "dn3" ] } Notice in cluster blueprint, the new addition section of applications is designed to deploy application in the hadoop environment (HDFS) or virtual environment, there is no local copy stored in local file system. The running environment is a flatten list of key value pairs, this is designed to make it easier for dev ops to manage the running environment more easily. The additional overwrite to environment allows to set a specific value for a particular config file. Host manifest is a flatten list of hostGroups for user friendliness for dev ops. Conflict, and provide syntax can support swappable component for Hadoop software stack.
          Hide
          Brian Swan added a comment -

          Eric: In some cases, it makes sense to think of OpenJDK (or Python, or ...) as a service (e.g. deploying Hadoop in a virtual environment or in a cloud environment). In these scenarios, when a "service" is not manageable, it is just software to be installed.

          By suggesting that the "running environment" should just be a flattened list of key-value pairs, are you saying that this would be preferable to grouping them under core-site, hdfs-site, etc. ? The current design is such that it prevents a deployment tool (such as Ambari, but others too) from needing to maintain any sort of configuration mapping. (In general, I think it's important to consider that these manifest files may be consumed by deployment tools other than Ambari.)

          I like your suggestions for being able to add a configuration file or reference one in place of the key-value pairs (if I've interpreted your suggestion correctly). I also like the suggestion for a "conflict" property.

          Show
          Brian Swan added a comment - Eric: In some cases, it makes sense to think of OpenJDK (or Python, or ...) as a service (e.g. deploying Hadoop in a virtual environment or in a cloud environment). In these scenarios, when a "service" is not manageable, it is just software to be installed. By suggesting that the "running environment" should just be a flattened list of key-value pairs, are you saying that this would be preferable to grouping them under core-site, hdfs-site, etc. ? The current design is such that it prevents a deployment tool (such as Ambari, but others too) from needing to maintain any sort of configuration mapping. (In general, I think it's important to consider that these manifest files may be consumed by deployment tools other than Ambari.) I like your suggestions for being able to add a configuration file or reference one in place of the key-value pairs (if I've interpreted your suggestion correctly). I also like the suggestion for a "conflict" property.
          Hide
          Eric Yang added a comment -

          The original design to classify software as a service, it is somewhat simplified the fact that a service is composed of multiple software components. At the same time, the service internal can be replaced by alternative implementation. Such as, file system can be represented by ext3, xfs or ntfs. It is somewhat harder to define substitution services later, if xfs is being promoted as a service rather than file system as a service. However, it is best to describe this part as clear as possible for developer to implement the interface properly.

          For the running environment, it is good to collapse the name space because references can be reused across components. For example, JAVA_HOME reference can be used in hadoop-env.sh, and hbase-env.sh, and several dependent components. Today, hadoop-env.sh has one value for JAVA_HOME, and hbase-env.sh has another value for JAVA_HOME. Both values are not linked together with the JAVA_HOME reference. Operator needs to remember to change each occurrence of JAVA_HOME through all configurations sections. For improve operator experience, collapsing the namespace can avoid searching through files and namespace resolution between services/components.

          The optimization are meant to improve the management ability for both devs and ops.

          Show
          Eric Yang added a comment - The original design to classify software as a service, it is somewhat simplified the fact that a service is composed of multiple software components. At the same time, the service internal can be replaced by alternative implementation. Such as, file system can be represented by ext3, xfs or ntfs. It is somewhat harder to define substitution services later, if xfs is being promoted as a service rather than file system as a service. However, it is best to describe this part as clear as possible for developer to implement the interface properly. For the running environment, it is good to collapse the name space because references can be reused across components. For example, JAVA_HOME reference can be used in hadoop-env.sh, and hbase-env.sh, and several dependent components. Today, hadoop-env.sh has one value for JAVA_HOME, and hbase-env.sh has another value for JAVA_HOME. Both values are not linked together with the JAVA_HOME reference. Operator needs to remember to change each occurrence of JAVA_HOME through all configurations sections. For improve operator experience, collapsing the namespace can avoid searching through files and namespace resolution between services/components. The optimization are meant to improve the management ability for both devs and ops.
          Hide
          Steve Morin added a comment -

          I saw Blueprints on the Hortonworks site, but see the ticket hasn't been updated since last year. Is there a newer ticket that should be followed?

          Show
          Steve Morin added a comment - I saw Blueprints on the Hortonworks site, but see the ticket hasn't been updated since last year. Is there a newer ticket that should be followed?
          Hide
          Jeff Sposetti added a comment -

          Please take a look at AMBARI-5077 and this wiki page for updated info.

          https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

          AMBARI-5077 (and other JIRAs mentioned on the wiki page) are the JIRA to checkout for implementation info.

          Show
          Jeff Sposetti added a comment - Please take a look at AMBARI-5077 and this wiki page for updated info. https://cwiki.apache.org/confluence/display/AMBARI/Blueprints AMBARI-5077 (and other JIRAs mentioned on the wiki page) are the JIRA to checkout for implementation info.

            People

            • Assignee:
              John Speidel
              Reporter:
              Sumit Mohanty
            • Votes:
              3 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development