Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Java 6, Linux

      Description

      The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.

      Prototype demo source code can be obtained from:

      http://github.com/macroadster/hms

        Activity

        Hide
        Allen Wittenauer added a comment -

        Is the intent to provide bindings to pre-existing software or is the intent to re-invent the wheel?

        Show
        Allen Wittenauer added a comment - Is the intent to provide bindings to pre-existing software or is the intent to re-invent the wheel?
        Hide
        Eric Yang added a comment -

        HMS is design to deploy existing Hadoop software stack.

        Show
        Eric Yang added a comment - HMS is design to deploy existing Hadoop software stack.
        Hide
        Eli Collins added a comment -

        Hey Eric, HMS is a separate project headed to the incubator (http://wiki.apache.org/incubator/HMSProposal) right?

        Show
        Eli Collins added a comment - Hey Eric, HMS is a separate project headed to the incubator ( http://wiki.apache.org/incubator/HMSProposal ) right?
        Hide
        Eric Yang added a comment -

        Hi Eli, I am uncertain where is the best place for this project. This is a customized manage and deployment system for Hadoop software stack. Hence, it could be inside Hadoop or in the incubator. Some people have expressed interests for this to be in the contrib project, and others think this should be an incubator project. I am fine either way, but I did drafted the incubator proposal. What do people think? Let me post a architecture diagram and design. Hadoop community can decide the proper course of action.

        Show
        Eric Yang added a comment - Hi Eli, I am uncertain where is the best place for this project. This is a customized manage and deployment system for Hadoop software stack. Hence, it could be inside Hadoop or in the incubator. Some people have expressed interests for this to be in the contrib project, and others think this should be an incubator project. I am fine either way, but I did drafted the incubator proposal. What do people think? Let me post a architecture diagram and design. Hadoop community can decide the proper course of action.
        Hide
        Eric Yang added a comment -

        A brief description of Hadoop Management System design:

        Setup

        HMS Agent is a list of rpm packages which can be deployed as part of OS image through PXE boot. HMS Beacon is a daemon which runs on each zookeeper nodes to broadcast the location of the zookeeper. HMS Agent and controllers are standalone daemons, which resolve zookeeper location through HMS Beacon (zeroconf).

        Operation

        Operator can issue command through HMS client and pass through HMS controller REST API. HMS command is serialized into JSON messages and queued in Zookeeper. Multiple HMS controllers watch the command queue for commands. When a command triggers the controller to execute, HMS controllers compete to create a lock for the command, and corresponding cluster to execute the command. If locks are successfully created, the controller begin to translate the command into a list of actions to perform on the managed nodes. HMS controller watches for the status queues and coordinate actions to perform on HMS agents. HMS managed agents download software through yum repository or bit torrent through peer exchange. HMS agent reports installation status and configuration status back to agent status queue for HMS controller to orchestrate the cluster deployment. Once, all actions are finalized, HMS controller store the deployment command history in the cluster node.

        In the event of node failures (to be implemented), operator can re-image the defected node. When the agent join back, HMS agent can send status to controller to replay the installation and configuration history to recover.

        Monitoring Proposal

        For large clusters deployment, monitoring setup could be complex. HMS can simplify this by orchestrate Hadoop 0.20.2+1 (append branch) + HBase 0.90.3 + Pig 0.8.1 + Chukwa 0.5 deployment using the proposed RPM packages for HADOOP-6255, ZOOKEEPER-999, HBASE-3606, PIG-1857, CHUKWA (HADOOP-5030).

        Show
        Eric Yang added a comment - A brief description of Hadoop Management System design: Setup HMS Agent is a list of rpm packages which can be deployed as part of OS image through PXE boot. HMS Beacon is a daemon which runs on each zookeeper nodes to broadcast the location of the zookeeper. HMS Agent and controllers are standalone daemons, which resolve zookeeper location through HMS Beacon (zeroconf). Operation Operator can issue command through HMS client and pass through HMS controller REST API. HMS command is serialized into JSON messages and queued in Zookeeper. Multiple HMS controllers watch the command queue for commands. When a command triggers the controller to execute, HMS controllers compete to create a lock for the command, and corresponding cluster to execute the command. If locks are successfully created, the controller begin to translate the command into a list of actions to perform on the managed nodes. HMS controller watches for the status queues and coordinate actions to perform on HMS agents. HMS managed agents download software through yum repository or bit torrent through peer exchange. HMS agent reports installation status and configuration status back to agent status queue for HMS controller to orchestrate the cluster deployment. Once, all actions are finalized, HMS controller store the deployment command history in the cluster node. In the event of node failures (to be implemented), operator can re-image the defected node. When the agent join back, HMS agent can send status to controller to replay the installation and configuration history to recover. Monitoring Proposal For large clusters deployment, monitoring setup could be complex. HMS can simplify this by orchestrate Hadoop 0.20.2+1 (append branch) + HBase 0.90.3 + Pig 0.8.1 + Chukwa 0.5 deployment using the proposed RPM packages for HADOOP-6255 , ZOOKEEPER-999 , HBASE-3606 , PIG-1857 , CHUKWA ( HADOOP-5030 ).
        Hide
        Allen Wittenauer added a comment -

        This is a customized manage and deployment system for Hadoop software stack.

        This makes no sense. Rather than re-invent a specific type of wheel, one could generalize the wheel and get a potentially larger audience. The problem of course is that the competition rate goes way up.

        From the above description, I don't see anything here that makes this particularly unique vs. other config systems. (In fact, it could be argued that it is several years behind other systems.)

        Show
        Allen Wittenauer added a comment - This is a customized manage and deployment system for Hadoop software stack. This makes no sense. Rather than re-invent a specific type of wheel, one could generalize the wheel and get a potentially larger audience. The problem of course is that the competition rate goes way up. From the above description, I don't see anything here that makes this particularly unique vs. other config systems. (In fact, it could be argued that it is several years behind other systems.)
        Hide
        Eric Yang added a comment -

        Allen, I did extensive studies on all existing systems including puppet, mcollective, chef, cfengine, controlTier, Bcfg2. Most of the configuration management system focus on generating a set of templates and config parameters and push out changes one node at a time. This works fine in small number of machines, but most of the system fails beyond 1800 nodes or become difficult to maintain. i.e. mcollective uses spamming tree model on puppeteer, hence the puppet master becomes single point of failure. One puppet master failure could take large chunk of the nodes offline. HMS is designed to remove single point of failures in the deployment system, and improve performance. we found it is more reliable to store system state in Zookeeper for HA. Zeroconf is great for resolving service location. Exist config management system requires installation and configuration before it can be deployed. HMS is designed to install and operate without having to configure the management system. Bittorrent is much faster than install software from yum repository for large scale system. Granted that this system started several years behind existing system, but it solves some scalability and reliability issues.

        To summarize, HMS does the following better:

        • Scale
        • Reliability
        • Cross node application orchestration (action dependencies)
        • Speed
        • Sophisticate monitoring system (Reuse Chukwa)
        • Self healing cluster (Ability to replay history to heal nodes)
        Show
        Eric Yang added a comment - Allen, I did extensive studies on all existing systems including puppet, mcollective, chef, cfengine, controlTier, Bcfg2. Most of the configuration management system focus on generating a set of templates and config parameters and push out changes one node at a time. This works fine in small number of machines, but most of the system fails beyond 1800 nodes or become difficult to maintain. i.e. mcollective uses spamming tree model on puppeteer, hence the puppet master becomes single point of failure. One puppet master failure could take large chunk of the nodes offline. HMS is designed to remove single point of failures in the deployment system, and improve performance. we found it is more reliable to store system state in Zookeeper for HA. Zeroconf is great for resolving service location. Exist config management system requires installation and configuration before it can be deployed. HMS is designed to install and operate without having to configure the management system. Bittorrent is much faster than install software from yum repository for large scale system. Granted that this system started several years behind existing system, but it solves some scalability and reliability issues. To summarize, HMS does the following better: Scale Reliability Cross node application orchestration (action dependencies) Speed Sophisticate monitoring system (Reuse Chukwa) Self healing cluster (Ability to replay history to heal nodes)
        Hide
        Rajiv Chittajallu added a comment -

        Allen, I did extensive studies on all existing systems including puppet, mcollective, chef, cfengine, controlTier, Bcfg2. Most of the configuration management system focus on generating a set of templates and config parameters and push out changes one node at a time. This works fine in small number of machines, but most of the system fails beyond 1800 nodes or become difficult to maintain.

        We use tar, ssh, wget, rsync & gpg with custom roles system (https://computing.llnl.gov/linux/genders.html can be an alternative) to manage configuration and packages. Our environment is probably still small to hit the limits of these tools.

        Our challenge with managing hadoop cluster is the lack of standard interfaces to reliably monitor the cluster. Standard unix tools expect process to exit with non zero status on error and counters to be positive numbers.

        IMHO whats needed here are features like HADOOP-6728 & HADOOP-7144, make them consistent across all components and integrate them with existing tools, HADOOP-7324 .

        Zeroconf is great for resolving service location.

        As part of this proposal, are there plans to update how hadoop daemon and client configurations are handled or is this specific to HMS?

        Bittorrent is much faster than install software from yum repository for large scale system.

        Bittorrent is a file sharing protocol and yum is a utility for rpm package management. I guess you mean to say bittorrent is faster to distribute files than http. If RPM is choose as the package format but don't want to use yum, HMS may need to implement another rpm based package management.

        Alternatively, this could just be a yum plugin.

        thats my 0.2 cents. But hey, if you want to invest your time in writing Yet Another Monitoring System , I wish you all the best!

        Show
        Rajiv Chittajallu added a comment - Allen, I did extensive studies on all existing systems including puppet, mcollective, chef, cfengine, controlTier, Bcfg2. Most of the configuration management system focus on generating a set of templates and config parameters and push out changes one node at a time. This works fine in small number of machines, but most of the system fails beyond 1800 nodes or become difficult to maintain. We use tar, ssh, wget, rsync & gpg with custom roles system ( https://computing.llnl.gov/linux/genders.html can be an alternative) to manage configuration and packages. Our environment is probably still small to hit the limits of these tools. Our challenge with managing hadoop cluster is the lack of standard interfaces to reliably monitor the cluster. Standard unix tools expect process to exit with non zero status on error and counters to be positive numbers. IMHO whats needed here are features like HADOOP-6728 & HADOOP-7144 , make them consistent across all components and integrate them with existing tools, HADOOP-7324 . Zeroconf is great for resolving service location. As part of this proposal, are there plans to update how hadoop daemon and client configurations are handled or is this specific to HMS? Bittorrent is much faster than install software from yum repository for large scale system. Bittorrent is a file sharing protocol and yum is a utility for rpm package management. I guess you mean to say bittorrent is faster to distribute files than http. If RPM is choose as the package format but don't want to use yum, HMS may need to implement another rpm based package management. Alternatively, this could just be a yum plugin. thats my 0.2 cents. But hey, if you want to invest your time in writing Yet Another Monitoring System , I wish you all the best!
        Hide
        Allen Wittenauer added a comment -

        The more I think about this and read the comments it is pretty clear what needs to happen. Eli already hinted at it, but I'm going to expand on his suggestion. This needs to get remade as a generic toolset and sent to the incubator as a generic Apache configuration management system. I can't think of any Apache licensed systems and this would be a good thing for the ASF to have in the portfolio. The biggest hurdle it has is the massive reliance on GPL'd bits. (There are so many components that this is a configuration management system that requires a configuration management system to actually configure...)

        As it currently stands, this functionality is completely outside of core Hadoop. So I'm already -1 on this going into common.

        Show
        Allen Wittenauer added a comment - The more I think about this and read the comments it is pretty clear what needs to happen. Eli already hinted at it, but I'm going to expand on his suggestion. This needs to get remade as a generic toolset and sent to the incubator as a generic Apache configuration management system. I can't think of any Apache licensed systems and this would be a good thing for the ASF to have in the portfolio. The biggest hurdle it has is the massive reliance on GPL'd bits. (There are so many components that this is a configuration management system that requires a configuration management system to actually configure...) As it currently stands, this functionality is completely outside of core Hadoop. So I'm already -1 on this going into common.
        Hide
        Eric Yang added a comment -

        As part of this proposal, are there plans to update how hadoop daemon and client configurations are handled or is this specific to HMS?

        It is possible to add zeroconf to hadoop. There were discussion of zeroconf on and off in hadoop jiras and mailing list.

        Bittorrent is a file sharing protocol and yum is a utility for rpm package management. I guess you mean to say bittorrent is faster to distribute files than http. If RPM is choose as the package format but don't want to use yum, HMS may need to implement another rpm based package management.

        HMS is currently designed to install software as a stack for the torrent implementation. If the RPM dependencies are not resolved with in the software stack or the base OS, it will fail the installation instead of resolving the dependency and download from yum. Hadoop software stack is small. Hence, Build time package dependencies declaration and release engineer blessed software stack should be sufficient for now. HMS does not intend to create another package management, but it can utilize platform's package management system. i.e. .rpm/.deb on linux, and .pkg on Mac.

        Alternatively, this could just be a yum plugin.

        There is a yum bittorrent proposal back in 2008, but it has not come to fruit yet. We may utilize it, if this is implemented in the future.

        The biggest hurdle it has is the massive reliance on GPL'd bits. (There are so many components that this is a configuration management system that requires a configuration management system to actually configure...)

        Allen, system call to GPL software does not automatically make caller GPL. See:

        http://www.ifross.org/en/program-forks-gpl-licensed-program-system-or-vice-versa-call-derivative-work

        HMS agent can call Linux utilities as long as the software runs in separated processes. Hence, your concern should be addressed. HMS has already ensure its libraries are Apache Software License or compatible license.

        Show
        Eric Yang added a comment - As part of this proposal, are there plans to update how hadoop daemon and client configurations are handled or is this specific to HMS? It is possible to add zeroconf to hadoop. There were discussion of zeroconf on and off in hadoop jiras and mailing list. Bittorrent is a file sharing protocol and yum is a utility for rpm package management. I guess you mean to say bittorrent is faster to distribute files than http. If RPM is choose as the package format but don't want to use yum, HMS may need to implement another rpm based package management. HMS is currently designed to install software as a stack for the torrent implementation. If the RPM dependencies are not resolved with in the software stack or the base OS, it will fail the installation instead of resolving the dependency and download from yum. Hadoop software stack is small. Hence, Build time package dependencies declaration and release engineer blessed software stack should be sufficient for now. HMS does not intend to create another package management, but it can utilize platform's package management system. i.e. .rpm/.deb on linux, and .pkg on Mac. Alternatively, this could just be a yum plugin. There is a yum bittorrent proposal back in 2008, but it has not come to fruit yet. We may utilize it, if this is implemented in the future. The biggest hurdle it has is the massive reliance on GPL'd bits. (There are so many components that this is a configuration management system that requires a configuration management system to actually configure...) Allen, system call to GPL software does not automatically make caller GPL. See: http://www.ifross.org/en/program-forks-gpl-licensed-program-system-or-vice-versa-call-derivative-work HMS agent can call Linux utilities as long as the software runs in separated processes. Hence, your concern should be addressed. HMS has already ensure its libraries are Apache Software License or compatible license.
        Hide
        Allen Wittenauer added a comment -

        Allen, system call to GPL software does not automatically make caller GPL.

        I'm fully aware of that. The problem is that there isn't much here outside of GPL'ed and other licensed code. This would be much more interesting and useful if it was self-contained. A configuration management system that needs a configuration management system to install sort of defeats the purpose. All of the great ones require a minuscule amount of stub in place before they can install themselves. I don't see that happening here.

        Show
        Allen Wittenauer added a comment - Allen, system call to GPL software does not automatically make caller GPL. I'm fully aware of that. The problem is that there isn't much here outside of GPL'ed and other licensed code. This would be much more interesting and useful if it was self-contained. A configuration management system that needs a configuration management system to install sort of defeats the purpose. All of the great ones require a minuscule amount of stub in place before they can install themselves. I don't see that happening here.
        Hide
        Eric Yang added a comment -

        This would be much more interesting and useful if it was self-contained. A configuration management system that needs a configuration management system to install sort of defeats the purpose. All of the great ones require a minuscule amount of stub in place before they can install themselves. I don't see that happening here.

        There is a difference between configuration management system and package management system. The statements above are mixing package management with configuration management. For package management system, HMS can tap into existing package management system provided by the OS. HMS also supports tarball deployment, and role based software stack installation. Hence, it does not require a package management system to operate, if the customer choose to use tarball only.

        In the base OS, there is nothing manages configuration files for Hadoop. Configuration management is usually operate through template and scripting. Operation can use puppet to configure the system, and commit the configuration files to source tree repository. Chef uses MangoDB to store states. HMS tracks the configuration history and store the state changes in Zookeeper.

        For both package and config management, there are strong cases to use HMS than generic deployment system because HMS can provide better deployment experience and improve deployment efficiency.

        Show
        Eric Yang added a comment - This would be much more interesting and useful if it was self-contained. A configuration management system that needs a configuration management system to install sort of defeats the purpose. All of the great ones require a minuscule amount of stub in place before they can install themselves. I don't see that happening here. There is a difference between configuration management system and package management system. The statements above are mixing package management with configuration management. For package management system, HMS can tap into existing package management system provided by the OS. HMS also supports tarball deployment, and role based software stack installation. Hence, it does not require a package management system to operate, if the customer choose to use tarball only. In the base OS, there is nothing manages configuration files for Hadoop. Configuration management is usually operate through template and scripting. Operation can use puppet to configure the system, and commit the configuration files to source tree repository. Chef uses MangoDB to store states. HMS tracks the configuration history and store the state changes in Zookeeper. For both package and config management, there are strong cases to use HMS than generic deployment system because HMS can provide better deployment experience and improve deployment efficiency.
        Hide
        Jagane Sundar added a comment -

        Let me expand a little bit on Eric's explanation:

        HMS is a full lifecycle management system for Hadoop. It can create hadoop clusters, automatically manage the configuration of hadoop clusters, tear down hadoop clusters and upgrade hadoop clusters.

        HMS consists of highly available HMS Controller(s). A HMS Agent on each Hadoop Node communicates with the HMS Controller(s). HMS Users issue commands to the HMS Controller using a REST API. Example commands can create/teardown/configure/upgrade Hadoop clusters.

        HMS aspires to use OS native packaging mechanisms for installing hadoop, i.e. rpms on RedHat, .debs on Debian/Ubu. tarball installation for unsupported Unix like operating systems is also supported.

        HMS differs from Chef/Puppet and other deployment frameworks in several important ways.
        1. HMS is built to be highly available from day 1. HMS controllers can die and be restarted. HMS Agents and the machines they run on can spontaneously reboot. HMS itself will continue to be available.
        2. HMS is meant to scale to the size of a 20,000 plus server Datacenter.
        3. HMS is designed to be the building block for constructing a self service Hadoop cloud, wherein users can create/configure/teardown Hadoops.

        Show
        Jagane Sundar added a comment - Let me expand a little bit on Eric's explanation: HMS is a full lifecycle management system for Hadoop. It can create hadoop clusters, automatically manage the configuration of hadoop clusters, tear down hadoop clusters and upgrade hadoop clusters. HMS consists of highly available HMS Controller(s). A HMS Agent on each Hadoop Node communicates with the HMS Controller(s). HMS Users issue commands to the HMS Controller using a REST API. Example commands can create/teardown/configure/upgrade Hadoop clusters. HMS aspires to use OS native packaging mechanisms for installing hadoop, i.e. rpms on RedHat, .debs on Debian/Ubu. tarball installation for unsupported Unix like operating systems is also supported. HMS differs from Chef/Puppet and other deployment frameworks in several important ways. 1. HMS is built to be highly available from day 1. HMS controllers can die and be restarted. HMS Agents and the machines they run on can spontaneously reboot. HMS itself will continue to be available. 2. HMS is meant to scale to the size of a 20,000 plus server Datacenter. 3. HMS is designed to be the building block for constructing a self service Hadoop cloud, wherein users can create/configure/teardown Hadoops.
        Hide
        Allen Wittenauer added a comment -

        Something has to maintain the packages and config settings of HMS itself. "Kickstart" is not a valid answer.

        Something else being missed here is that Hadoop is just one component of a working data center. If I have to deploy bcfg2/puppet/chef/cfengine/... to manage my LDAP, Azkaban, Voldemort, ... nodes, why would I have something different for Hadoop when I can just use the pre-existing services? (Most of these systems are HA and can handle 20k nodes if one knows how to deploy them properly.)

        Also, why isn't this just rolled into Whirr? Isn't the whole point of that project software provisioning for "cloud" services? Yet another reason this doesn't belong in Hadoop-proper.

        Show
        Allen Wittenauer added a comment - Something has to maintain the packages and config settings of HMS itself. "Kickstart" is not a valid answer. Something else being missed here is that Hadoop is just one component of a working data center. If I have to deploy bcfg2/puppet/chef/cfengine/... to manage my LDAP, Azkaban, Voldemort, ... nodes, why would I have something different for Hadoop when I can just use the pre-existing services? (Most of these systems are HA and can handle 20k nodes if one knows how to deploy them properly.) Also, why isn't this just rolled into Whirr? Isn't the whole point of that project software provisioning for "cloud" services? Yet another reason this doesn't belong in Hadoop-proper.
        Hide
        Jagane Sundar added a comment -

        The goal of HMS is to provide a Hadoop specific management platform. While chef/puppet/bcfg2 and other scripted deployment tools are useful for simpler software, a complex software platform such as Hadoop could benefit from a management platform that is custom built for it.

        With the hadoop.next platform starting to take shape, the job of installing and configuring Hadoop is going to need even more sophisticated technology. It is critical to look at management of Hadoop from a holistic perspective.

        HMS plays nice with existing packaging mechanisms such as .rpms, .debs and tar files, which are the finest granularity building blocks.

        Alan - I don't understand your comparison of Whirr with HMS. HMS has the potential to provide the ability for cloud service providers to provide a clone of Elastic Map Reduce. Correct me if I am wrong, but isn't Whirr a set of scripts for deploying Hadoop on an IaaS cloud?

        Show
        Jagane Sundar added a comment - The goal of HMS is to provide a Hadoop specific management platform. While chef/puppet/bcfg2 and other scripted deployment tools are useful for simpler software, a complex software platform such as Hadoop could benefit from a management platform that is custom built for it. With the hadoop.next platform starting to take shape, the job of installing and configuring Hadoop is going to need even more sophisticated technology. It is critical to look at management of Hadoop from a holistic perspective. HMS plays nice with existing packaging mechanisms such as .rpms, .debs and tar files, which are the finest granularity building blocks. Alan - I don't understand your comparison of Whirr with HMS. HMS has the potential to provide the ability for cloud service providers to provide a clone of Elastic Map Reduce. Correct me if I am wrong, but isn't Whirr a set of scripts for deploying Hadoop on an IaaS cloud?
        Hide
        Eli Collins added a comment -

        HMS is a separate project for the ecosystem of Hadoop related projects, this is out of scope for Hadoop common. How about moving the discussion of Whirr, HMS etc to the incubator thread.

        Show
        Eli Collins added a comment - HMS is a separate project for the ecosystem of Hadoop related projects, this is out of scope for Hadoop common. How about moving the discussion of Whirr, HMS etc to the incubator thread.
        Hide
        Eric Yang added a comment -

        Echo on Jagane's comments. According to Hadoop mission statement:

        The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing.

        HMS is aligned with Hadoop mission statement. HMS is a tool to build Infrastructure as a Service cloud, and Whirr is a tool to build Platform as a Service. There are differences between them. HMS can provide deep OS integration for Hadoop, i.e. deploy a Kerbrose enabled Hadoop cluster with LDAP integration in cooperate environment. It will also provide hooks to remove defected nodes from operation, and bring up new nodes into the infrastructure.

        Whirr provides the other end of spectrum where it provides a platform for developers to code, test and experiment new software without the complexity of setting up and maintaining test, development and production servers.

        Both model should coexist, but HMS will require tighter integration with Hadoop to enhance Hadoop's presence in the cooperate environment. Additional input from Hadoop PMC can help to decide the right path for HMS.

        Show
        Eric Yang added a comment - Echo on Jagane's comments. According to Hadoop mission statement: The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing. HMS is aligned with Hadoop mission statement. HMS is a tool to build Infrastructure as a Service cloud, and Whirr is a tool to build Platform as a Service. There are differences between them. HMS can provide deep OS integration for Hadoop, i.e. deploy a Kerbrose enabled Hadoop cluster with LDAP integration in cooperate environment. It will also provide hooks to remove defected nodes from operation, and bring up new nodes into the infrastructure. Whirr provides the other end of spectrum where it provides a platform for developers to code, test and experiment new software without the complexity of setting up and maintaining test, development and production servers. Both model should coexist, but HMS will require tighter integration with Hadoop to enhance Hadoop's presence in the cooperate environment. Additional input from Hadoop PMC can help to decide the right path for HMS.
        Hide
        Eli Collins added a comment -

        That's cool, the mailing list is a better forum for discussing HMS and Whirr than a common jira, how about we move this to common-dev?

        Show
        Eli Collins added a comment - That's cool, the mailing list is a better forum for discussing HMS and Whirr than a common jira, how about we move this to common-dev?
        Hide
        Eric Yang added a comment -

        That's cool, the mailing list is a better forum for discussing HMS and Whirr than a common jira, how about we move this to common-dev?

        Sounds good.

        Show
        Eric Yang added a comment - That's cool, the mailing list is a better forum for discussing HMS and Whirr than a common jira, how about we move this to common-dev? Sounds good.
        Hide
        Eric Yang added a comment -

        Eli, we have not started the discussion on common-dev, why was this jira closed?

        Show
        Eric Yang added a comment - Eli, we have not started the discussion on common-dev, why was this jira closed?
        Hide
        Eli Collins added a comment -

        I figured you'd start the discussion on common-dev. This isn't a Hadoop feature, it's a separate project.

        Show
        Eli Collins added a comment - I figured you'd start the discussion on common-dev. This isn't a Hadoop feature, it's a separate project.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            27 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development