Ambari
  1. Ambari
  2. AMBARI-2714

Ability to dynamically add services to a stack.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: controller
    • Labels:
      None

      Description

      Simplifying adding a new service and allowing it to be a simple admin task to add his own service. The first attempt at this will be to define a service outside Ambari, such that it becomes a "zero code" effort.

      1. Stack Definitions.docx
        100 kB
        Nate Cole
      2. ServiceDefinitionDesignandPlan.docx
        74 kB
        Sumit Mohanty
      3. Service Definition Design and Plan.docx
        117 kB
        Mahadev konar

        Issue Links

          Activity

          Hide
          Ed Kohlwey added a comment -

          These design docs seem to focus quite a bit on backend components and reporting - is there any thought going into modularizing the installation wizard GUI's at this point and what that would look like?

          Also, is there any prototyping going on for the designs yet? It seems like many aspects have been fairly well thought out. Would it be beneficial to start looking at the Java classes mentioned in the design docs and prototyping implementations of the spec?

          Show
          Ed Kohlwey added a comment - These design docs seem to focus quite a bit on backend components and reporting - is there any thought going into modularizing the installation wizard GUI's at this point and what that would look like? Also, is there any prototyping going on for the designs yet? It seems like many aspects have been fairly well thought out. Would it be beneficial to start looking at the Java classes mentioned in the design docs and prototyping implementations of the spec?
          Hide
          Jeff Sposetti added a comment -

          In the latest document, it says "Current assumption is that a Custom Command cannot change the state of the Service or Component." But then earlier, it talks about commands start/stop and later it mentions command restart. And it also says Actions do not change state as well. So seems Commands can change state, Custom Commands can't and Actions can't. Not sure why the split, and how someone can change state beyond the well-defined commands for a service. Maybe what I am missing is the context of what state means.

          Show
          Jeff Sposetti added a comment - In the latest document, it says "Current assumption is that a Custom Command cannot change the state of the Service or Component." But then earlier, it talks about commands start/stop and later it mentions command restart. And it also says Actions do not change state as well. So seems Commands can change state, Custom Commands can't and Actions can't. Not sure why the split, and how someone can change state beyond the well-defined commands for a service. Maybe what I am missing is the context of what state means.
          Hide
          Nate Cole added a comment -

          @K. Eric - interesting, I'll have to check it out. Thanks!

          Show
          Nate Cole added a comment - @K. Eric - interesting, I'll have to check it out. Thanks!
          Hide
          K. Eric Harper added a comment -

          @Nate - Supervisor is a daemon implemented in Python (https://pypi.python.org/pypi/supervisor) that we have been using to automatically start and manage services on Linux. If Ambari provides those capabilities so much the better. For the Supervisor configuration you specify the command line that invokes the service in a manner that will send the log to standard output. Supervisor confirms that records are being written to standard out and terminates the process if it hangs, then restarts using the command line. Primitive, but that is what we are using for example to have the Storm UI daemon running when the machine boots. Anyone who is using Supervisor should be able to migrate to Ambari if a similar configuration is expected. What do you think?

          Show
          K. Eric Harper added a comment - @Nate - Supervisor is a daemon implemented in Python ( https://pypi.python.org/pypi/supervisor ) that we have been using to automatically start and manage services on Linux. If Ambari provides those capabilities so much the better. For the Supervisor configuration you specify the command line that invokes the service in a manner that will send the log to standard output. Supervisor confirms that records are being written to standard out and terminates the process if it hangs, then restarts using the command line. Primitive, but that is what we are using for example to have the Storm UI daemon running when the machine boots. Anyone who is using Supervisor should be able to migrate to Ambari if a similar configuration is expected. What do you think?
          Hide
          Jeff Sposetti added a comment -

          Another bit of info to include in the service definition XML is license.

          Each service might/will carry their own license. We should also allow the user to specify the license with the service definition. This includes both a label and a link. For example, if a service was GPL, they would set the <license label="GPL" href="http://www.gnu.org/licenses/gpl-2.0.html">

          With that information in the definition, and available via the REST API, in the UI, after selecting a Stack, when you see the list of services, in addition to the services description, we could show GPL with a link that pops out.

          Show
          Jeff Sposetti added a comment - Another bit of info to include in the service definition XML is license. Each service might/will carry their own license. We should also allow the user to specify the license with the service definition. This includes both a label and a link. For example, if a service was GPL, they would set the <license label="GPL" href="http://www.gnu.org/licenses/gpl-2.0.html"> With that information in the definition, and available via the REST API, in the UI, after selecting a Stack, when you see the list of services, in addition to the services description, we could show GPL with a link that pops out.
          Hide
          Nate Cole added a comment -

          @Erin - initially it will be configuring XML and have the UI recognize it as an addable service. The capability exists to reload stack definitions already, so that's no big deal. Long term I agree - we want to be supply new services using the UI itself. You are also correct in that we want to be able to supply all the artifacts surrounding a new service including tarballs, scripts etc. To start with, at least, we'll probably focus on defining a service without modifying code. Then we can start adding real capability.

          @Abhishek - I think a great use-case will be to try to add Storm without modifying Ambari code.

          @K. Eric - I'm not too familiar with Supervisor - how would you envision that fitting with Ambari (it handles monitoring and control already)?

          Show
          Nate Cole added a comment - @Erin - initially it will be configuring XML and have the UI recognize it as an addable service. The capability exists to reload stack definitions already, so that's no big deal. Long term I agree - we want to be supply new services using the UI itself. You are also correct in that we want to be able to supply all the artifacts surrounding a new service including tarballs, scripts etc. To start with, at least, we'll probably focus on defining a service without modifying code. Then we can start adding real capability. @Abhishek - I think a great use-case will be to try to add Storm without modifying Ambari code. @K. Eric - I'm not too familiar with Supervisor - how would you envision that fitting with Ambari (it handles monitoring and control already)?
          Hide
          Erin A Boyd added a comment -

          Is the idea that the user configure all the xml/json files prior to loading the new service or will the UI provide a way to just load the new service and deploy it accordingly?
          Also, we need to consider what resources need to be loaded with the new service (rpms, tarball, etc..) to enable the new service. Might want to also consider the 'control' from Ambari to the new service. If it's just a 'viewer' or controller of the service itself.

          Show
          Erin A Boyd added a comment - Is the idea that the user configure all the xml/json files prior to loading the new service or will the UI provide a way to just load the new service and deploy it accordingly? Also, we need to consider what resources need to be loaded with the new service (rpms, tarball, etc..) to enable the new service. Might want to also consider the 'control' from Ambari to the new service. If it's just a 'viewer' or controller of the service itself.
          Hide
          Abhishek Aggarwal added a comment -

          I have already installed and configured supervisors and nimbus to run storm. It's running successfully.
          I wanted to monitor these storm nodes through Ambari UI just like we are currently monitoring services like MapReduce , hdfs , hbase etc through Ambari UI.

          One can start this but neeed to understand the complete structure of Ambari project , java files , python files and configuration files and then deployment. It will be lengthy process. I just wanted to check if this enhancement is in-progress , I mean if some one is already working on this task.

          Show
          Abhishek Aggarwal added a comment - I have already installed and configured supervisors and nimbus to run storm. It's running successfully. I wanted to monitor these storm nodes through Ambari UI just like we are currently monitoring services like MapReduce , hdfs , hbase etc through Ambari UI. One can start this but neeed to understand the complete structure of Ambari project , java files , python files and configuration files and then deployment. It will be lengthy process. I just wanted to check if this enhancement is in-progress , I mean if some one is already working on this task.
          Hide
          K. Eric Harper added a comment -

          Currently using Supervisor (http://supervisord.org/) to manage these types of processes. Perhaps a simple start to this activity would be to automatically configure Supervisor on each node, and deploy the necessary java and configuration files, etc. What do you think?

          Show
          K. Eric Harper added a comment - Currently using Supervisor ( http://supervisord.org/ ) to manage these types of processes. Perhaps a simple start to this activity would be to automatically configure Supervisor on each node, and deploy the necessary java and configuration files, etc. What do you think?
          Hide
          Abhishek Aggarwal added a comment -

          Purpose is to enable monitoring capabilities for the new service(e.g Storm) being added

          Show
          Abhishek Aggarwal added a comment - Purpose is to enable monitoring capabilities for the new service(e.g Storm) being added
          Hide
          Abhishek Aggarwal added a comment -

          AMBARI-2813 can be considered as sub-task.

          Show
          Abhishek Aggarwal added a comment - AMBARI-2813 can be considered as sub-task.
          Hide
          Nate Cole added a comment -
          1. Definitely a good idea. Turning off parent services definitely has consequences on their dependent services, so troubleshooting/notification will be key there.
          2. Maybe we can make a "UI display profile" that has that information. Each profile then needs a unique name identifier; "ambari_default" for the built-in UI perhaps. I hesitate to put that info in the existing configuration files because the implication is that the stack is enforcing what is appropriate for display.
          Show
          Nate Cole added a comment - Definitely a good idea. Turning off parent services definitely has consequences on their dependent services, so troubleshooting/notification will be key there. Maybe we can make a "UI display profile" that has that information. Each profile then needs a unique name identifier; "ambari_default" for the built-in UI perhaps. I hesitate to put that info in the existing configuration files because the implication is that the stack is enforcing what is appropriate for display.
          Hide
          Jeff Sposetti added a comment -

          1) Would like to see concept of stack inheritance. A user can define a stack that extends another stack. The resulting stack can add a new service (Stack A is extended by Stack B, which is exactly the same except Stack B has N-new service) and/or turn off a service in the parent stack.
          2) In the service/configurations/*, there should be a way to define how config files + properties show in the UI (including a way to group). For example, how would you describe the Configs tab on HDFS service via meta data (instead of having to code the JS page)? If you can define the UI in meta data, then the UI can read that info and be dynamic.

          Show
          Jeff Sposetti added a comment - 1) Would like to see concept of stack inheritance. A user can define a stack that extends another stack. The resulting stack can add a new service (Stack A is extended by Stack B, which is exactly the same except Stack B has N-new service) and/or turn off a service in the parent stack. 2) In the service/configurations/*, there should be a way to define how config files + properties show in the UI (including a way to group). For example, how would you describe the Configs tab on HDFS service via meta data (instead of having to code the JS page)? If you can define the UI in meta data, then the UI can read that info and be dynamic.

            People

            • Assignee:
              Nate Cole
              Reporter:
              Nate Cole
            • Votes:
              6 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development