Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-780

Support for Docker based application packaging in Slider

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Slider 0.80
    • Component/s: None
    • Labels:
      None
    • Sprint:
      Slider April #1

      Description

      Enable Slider to deploy an application defined as Docker image, monitor its running status, fetching exported configs, and maintain its lifecycle.
      Please find the details of this ticket in the attachment 'SlidersupportingDockersummary(1).pdf'

      1. JIRA-780-001.patch
        49 kB
        thomas liu
      2. JIRA-780-002.patch
        57 kB
        thomas liu
      3. SlidersupportingDockersummary (1).pdf
        279 kB
        thomas liu

        Issue Links

          Activity

          Hide
          thomas_liu thomas liu added a comment -

          This is the code change patch that enables Slider to deploy basic Docker based applications

          Show
          thomas_liu thomas liu added a comment - This is the code change patch that enables Slider to deploy basic Docker based applications
          Hide
          gsaha Gour Saha added a comment -

          Reviewing this now, will provide feedback in the next hour or so.

          Show
          gsaha Gour Saha added a comment - Reviewing this now, will provide feedback in the next hour or so.
          Hide
          gsaha Gour Saha added a comment -

          Thomas, the patch looks good overall. I don't see any tests. Is there a sample package so that users can get a sample metainfo, appconfig and resources example for Docker?

          I did not get a chance to play with the feature to verify the functionality. But here are few comments from just going through the patch.

          main.py

          con = None

          I think we can get rid of variable con and declare controller at global level. Then call controller.actionQueue.dockerManager.stop_container() in signal_handler.

          AgentProviderService.java

          protected void addInstallDockerCommand2(String componentName,
          protected void addStartDockerCommand2(String componentName, String containerId, HeartBeatResponse response,

          Is there a reason to use the digit 2 in the method names? If not, please rename them appropriately.

          return type.toLowerCase().equals("docker");

          Let's define string constant for "docker" in SliderUtils.java just like PYTHON is defined.

          cmdParams.put("script_type", "PYTHON");

          Let's define TYPE_PYTHON in AbstractComponent.java just like you have TYPE_DOCKER.

          Ideally, for the above 2 scenarios, we should create a ScriptType class to encapsulate these, but we can do it later.

          Not required now, but at some point we need to define PythonExecutionCommand and DockerExecutionCommand to modularize the code. This will also pave the way for supporting additional executors/containers in the future, like Flocker (which is picking some steam as well).

          log.info("Docker- value from appconfig component: " + componentName + " configName: " + composedConfigName + " value: " + appConfigValue);
          log.info("Docker- value from metainfo component: " + componentName + " configName: " + configName + " value: " + appConfigValue);
          log.info("Docker- component: " + componentName + " configName: " + configName + " value: " + result);

          Use formatter {} instead of string concat.

          for(String key: compConf.keySet()){

          Seems like we can use entrySet() as you are accessing key and value.

          DockerContainerPort.java, DockerContainerMount.java, and DockerContainer.java

          public void validate(String version) throws SliderException {

          Add @Override to all validate methods.

          DockerContainer.java

            public String toString() {
              return "DockerContainer [name=" + name + ", image=" + image + ", options="
                   + options + ", mounts=" + mounts + ", ports=" + ports
                   + ", statusCommand=" + statusCommand + ", commandPath=" + commandPath
                   + ", additionalParam=" + additionalParam + ", inputFiles=" + inputFiles
                   + "]";
            }
          

          Use StringBuilder

          Some generic formatting comments:

          1. There are several places where the lines should be wrapped as they crosses 80 chars, like in method addStartDockerCommand2. Please revisit all files in the patch and wrap the long lines.
          2. In Slider for Python and Java files we use only 2 spaces for indentation. There are several places where it has 4 spaces (for 1-level indentation), like addContainerDetails in ExecutionCommand.java and ActionQueue.py
          3. New lines should not contain any empty spaces
          4. use space between if and (
          Show
          gsaha Gour Saha added a comment - Thomas, the patch looks good overall. I don't see any tests. Is there a sample package so that users can get a sample metainfo, appconfig and resources example for Docker? I did not get a chance to play with the feature to verify the functionality. But here are few comments from just going through the patch. main.py con = None I think we can get rid of variable con and declare controller at global level. Then call controller.actionQueue.dockerManager.stop_container() in signal_handler. AgentProviderService.java protected void addInstallDockerCommand2(String componentName, protected void addStartDockerCommand2(String componentName, String containerId, HeartBeatResponse response, Is there a reason to use the digit 2 in the method names? If not, please rename them appropriately. return type.toLowerCase().equals("docker"); Let's define string constant for "docker" in SliderUtils.java just like PYTHON is defined. cmdParams.put("script_type", "PYTHON"); Let's define TYPE_PYTHON in AbstractComponent.java just like you have TYPE_DOCKER. Ideally, for the above 2 scenarios, we should create a ScriptType class to encapsulate these, but we can do it later. Not required now, but at some point we need to define PythonExecutionCommand and DockerExecutionCommand to modularize the code. This will also pave the way for supporting additional executors/containers in the future, like Flocker (which is picking some steam as well). log.info("Docker- value from appconfig component: " + componentName + " configName: " + composedConfigName + " value: " + appConfigValue); log.info("Docker- value from metainfo component: " + componentName + " configName: " + configName + " value: " + appConfigValue); log.info("Docker- component: " + componentName + " configName: " + configName + " value: " + result); Use formatter {} instead of string concat. for(String key: compConf.keySet()){ Seems like we can use entrySet() as you are accessing key and value. DockerContainerPort.java, DockerContainerMount.java, and DockerContainer.java public void validate(String version) throws SliderException { Add @Override to all validate methods. DockerContainer.java public String toString() { return "DockerContainer [name=" + name + ", image=" + image + ", options=" + options + ", mounts=" + mounts + ", ports=" + ports + ", statusCommand=" + statusCommand + ", commandPath=" + commandPath + ", additionalParam=" + additionalParam + ", inputFiles=" + inputFiles + "]"; } Use StringBuilder Some generic formatting comments: There are several places where the lines should be wrapped as they crosses 80 chars, like in method addStartDockerCommand2 . Please revisit all files in the patch and wrap the long lines. In Slider for Python and Java files we use only 2 spaces for indentation. There are several places where it has 4 spaces (for 1-level indentation), like addContainerDetails in ExecutionCommand.java and ActionQueue.py New lines should not contain any empty spaces use space between if and (
          Hide
          gsaha Gour Saha added a comment -

          Test TestMain.test_signal_handler is failing. In method signal_handler in main.py you need to add a check -

            if controller is not None:
              tmpdir = controller.actionQueue.dockerManager.stop_container()
          

          It assumes you have got rid of variable con as per comments above.

          Show
          gsaha Gour Saha added a comment - Test TestMain.test_signal_handler is failing. In method signal_handler in main.py you need to add a check - if controller is not None: tmpdir = controller.actionQueue.dockerManager.stop_container() It assumes you have got rid of variable con as per comments above.
          Hide
          thomas_liu thomas liu added a comment -

          Thank you for your comments!
          Now I'm still working on the fun-tests of this patch.
          Given we need to install Docker on the target hosts, writing unit test with minicluster is a little tricker

          I think we can get rid of variable con and declare controller at global level. Then call controller.actionQueue.dockerManager.stop_container() in signal_handler.

          Agree. Also seen your comment below regarding the failing test case.

          Is there a reason to use the digit 2 in the method names? If not, please rename them appropriately.

          No there isn't...It was a result of previous temporary "addStartCommand2" code in the private branches. I just blindly merged them in. Will rename them.

          return type.toLowerCase().equals("docker");
          Let's define string constant for "docker" in SliderUtils.java just like PYTHON is defined.
          cmdParams.put("script_type", "PYTHON");
          Let's define TYPE_PYTHON in AbstractComponent.java just like you have TYPE_DOCKER.

          Agree.

          Not required now, but at some point we need to define PythonExecutionCommand and DockerExecutionCommand to modularize the code. This will also pave the way for supporting additional executors/containers in the future, like Flocker (which is picking some steam as well).

          Agree. Also have this in my radar. Will simplify the params we pass to Agent as well. Besides, PythonExecutionCommand and DockerExecutionCommand, there can be ShellCommandExecutionCommand as well.

          Use formatter {} instead of string concat.

          Agree. Those are old logging code. Will change their level.

          Seems like we can use entrySet() as you are accessing key and value.

          Agree

          DockerContainerPort.java, DockerContainerMount.java, and DockerContainer.java
          public void validate(String version) throws SliderException {
          Add @Override to all validate methods.

          Agree

          Use StringBuilder

          Agree

          Some generic formatting comments:

          Sounds good

          Thank you for your comments!
          Will post my new patch & with test cases tomorrow.

          Show
          thomas_liu thomas liu added a comment - Thank you for your comments! Now I'm still working on the fun-tests of this patch. Given we need to install Docker on the target hosts, writing unit test with minicluster is a little tricker I think we can get rid of variable con and declare controller at global level. Then call controller.actionQueue.dockerManager.stop_container() in signal_handler. Agree. Also seen your comment below regarding the failing test case. Is there a reason to use the digit 2 in the method names? If not, please rename them appropriately. No there isn't...It was a result of previous temporary "addStartCommand2" code in the private branches. I just blindly merged them in. Will rename them. return type.toLowerCase().equals("docker"); Let's define string constant for "docker" in SliderUtils.java just like PYTHON is defined. cmdParams.put("script_type", "PYTHON"); Let's define TYPE_PYTHON in AbstractComponent.java just like you have TYPE_DOCKER. Agree. Not required now, but at some point we need to define PythonExecutionCommand and DockerExecutionCommand to modularize the code. This will also pave the way for supporting additional executors/containers in the future, like Flocker (which is picking some steam as well). Agree. Also have this in my radar. Will simplify the params we pass to Agent as well. Besides, PythonExecutionCommand and DockerExecutionCommand, there can be ShellCommandExecutionCommand as well. Use formatter {} instead of string concat. Agree. Those are old logging code. Will change their level. Seems like we can use entrySet() as you are accessing key and value. Agree DockerContainerPort.java, DockerContainerMount.java, and DockerContainer.java public void validate(String version) throws SliderException { Add @Override to all validate methods. Agree Use StringBuilder Agree Some generic formatting comments: Sounds good Thank you for your comments! Will post my new patch & with test cases tomorrow.
          Hide
          thomas_liu thomas liu added a comment - - edited

          This code patch is to address Gour's comments.
          Will attach another patch only for test cases soon
          I have run all unit test cases and fun-test cases.
          Though there are one or two failing ones, they are not because of our code change

          Show
          thomas_liu thomas liu added a comment - - edited This code patch is to address Gour's comments. Will attach another patch only for test cases soon I have run all unit test cases and fun-test cases. Though there are one or two failing ones, they are not because of our code change
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5a3a64392c51d34c4a4c95bb6047295bc3a6e213 in incubator-slider's branch refs/heads/develop from Gour Saha
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=5a3a643 ]

          SLIDER-780 Support for Docker based application packaging in Slider (Thomas Liu via gourksaha)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5a3a64392c51d34c4a4c95bb6047295bc3a6e213 in incubator-slider's branch refs/heads/develop from Gour Saha [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=5a3a643 ] SLIDER-780 Support for Docker based application packaging in Slider (Thomas Liu via gourksaha)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 49590158167d2dafc96728e7ae007911f8aa12cd in incubator-slider's branch refs/heads/develop from Gour Saha
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=4959015 ]

          SLIDER-780 Support for Docker based application packaging in Slider - add missing license headers (Thomas Liu via gourksaha)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 49590158167d2dafc96728e7ae007911f8aa12cd in incubator-slider's branch refs/heads/develop from Gour Saha [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=4959015 ] SLIDER-780 Support for Docker based application packaging in Slider - add missing license headers (Thomas Liu via gourksaha)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1680364 from Gour Saha in branch 'site/trunk'
          [ https://svn.apache.org/r1680364 ]

          SLIDER-780 Support for Docker based application packaging in Slider - site documentation (Thomas Liu via gourksaha)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1680364 from Gour Saha in branch 'site/trunk' [ https://svn.apache.org/r1680364 ] SLIDER-780 Support for Docker based application packaging in Slider - site documentation (Thomas Liu via gourksaha)
          Hide
          yongw Yong Wang added a comment -

          Hi Thomas, I'm very intersted in Docker based app packaging and am exited to see Slider 0.8.0 is released, I installed this version of Slider and tried to follow the instructions to run the sample application given in SlidersupportingDockersummary.pdf in the attachments of this ticket. However, I have no luck to successfully create the basic application described in Senario 1 in the PDF.
          It report an error indicated that the component 'MEMCACHED' is not a member of application.

          I used the metainfo.json and resources.json provided in the PDF and the command I tried to create app is:
          slider create memcached --metainfo metainfo.json --resources resources.json

          Output Is:
          2015-06-26 17:47:26,296 [main] INFO client.RMProxy - Connecting to ResourceManager at dsj-s6/10.35.22.72:8032
          2015-06-26 17:47:26,895 [main] INFO persist.AppDefinitionPersister - Using default app def path hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef
          2015-06-26 17:47:26,898 [main] INFO persist.AppDefinitionPersister - Setting app package to hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip.
          2015-06-26 17:47:27,288 [main] INFO persist.AppDefinitionPersister - Processing app package/folder /tmp/1435312046889-0/default for appPkg.zip
          2015-06-26 17:47:27,290 [main] INFO tools.SliderUtils - Zipping folder /tmp/1435312046889-0/default to /tmp/1435312047288-0/appPkg.zip
          2015-06-26 17:47:27,328 [main] INFO agent.AgentClientProvider - Validating app definition hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip
          2015-06-26 17:47:27,331 [main] INFO agent.AgentUtils - Reading metainfo at hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip
          2015-06-26 17:47:27,450 [main] ERROR metadata.Metainfo - Malformed app definition: Expect application as the top level element for metainfo
          2015-06-26 17:47:27,451 [main] INFO client.SliderClient - Error Component MEMCACHED is not a member of application. validating application instance definition
          2015-06-26 17:47:27,460 [main] INFO client.SliderClient - {,
          "internal": {
          "schema" : "http://example.org/specification/v2.0.0",
          "metadata" :

          { "create.hadoop.deployed.info" : "branch-2.6.0 @18e43357c8f927c0695f1e9522859d6a", "create.application.build.info" : "Slider Core-0.80.0-incubating Built against commit# d7e3449fa6 on Java 1.7.0_60 by gsaha", "create.hadoop.build.info" : "2.6.0", "create.time.millis" : "1435312046903", "create.time" : "26 Jun 2015 09:47:26 GMT" }

          ,
          "global" :

          { "internal.generated.conf.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/generated", "internal.appdef.dir.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp/appdef", "application.name" : "memcached", "slider.cluster.directory.permissions" : "0770", "internal.addons.dir.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp/addons", "internal.provider.name" : "agent", "internal.data.dir.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/database", "internal.tmp.dir" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp", "internal.snapshot.conf.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/snapshot", "slider.data.directory.permissions" : "0770", "internal.container.failure.shortlife" : "60000", "internal.am.tmp.dir" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp/appmaster", "internal.container.failure.threshold" : "5" }

          ,
          "credentials" : { },
          "components" : { }
          },
          "resources": {
          "schema" : "http://example.org/specification/v2.0.0",
          "metadata" : { },
          "global" : { },
          "credentials" : { },
          "components" : {
          "slider-appmaster" :

          { "yarn.memory" : "1024", "yarn.vcores" : "1", "yarn.component.instances" : "1" }

          ,
          "MEMCACHED" :

          { "yarn.memory" : "512", "yarn.role.priority" : "1", "yarn.component.instances" : "2" }

          }
          },
          "appConf" :{
          "schema" : "http://example.org/specification/v2.0.0",
          "metadata" : { },
          "global" :

          { "site.fs.defaultFS" : "hdfs://dsj-s6:8020", "env.MALLOC_ARENA_MAX" : "4", "site.fs.default.name" : "hdfs://dsj-s6:8020", "zookeeper.path" : "/services/slider/users/hdev/memcached", "zookeeper.quorum" : "localhost:2181", "zookeeper.hosts" : "localhost", "application.def" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip" }

          ,
          "credentials" : { },
          "components" : {
          "slider-appmaster" :

          { "jvm.heapsize" : "256M" }

          }
          }}
          2015-06-26 17:47:27,460 [main] ERROR main.ServiceLauncher - Component MEMCACHED is not a member of application.
          2015-06-26 17:47:27,462 [main] INFO util.ExitUtil - Exiting with status 77

          Show
          yongw Yong Wang added a comment - Hi Thomas, I'm very intersted in Docker based app packaging and am exited to see Slider 0.8.0 is released, I installed this version of Slider and tried to follow the instructions to run the sample application given in SlidersupportingDockersummary.pdf in the attachments of this ticket. However, I have no luck to successfully create the basic application described in Senario 1 in the PDF. It report an error indicated that the component 'MEMCACHED' is not a member of application. I used the metainfo.json and resources.json provided in the PDF and the command I tried to create app is: slider create memcached --metainfo metainfo.json --resources resources.json Output Is: 2015-06-26 17:47:26,296 [main] INFO client.RMProxy - Connecting to ResourceManager at dsj-s6/10.35.22.72:8032 2015-06-26 17:47:26,895 [main] INFO persist.AppDefinitionPersister - Using default app def path hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef 2015-06-26 17:47:26,898 [main] INFO persist.AppDefinitionPersister - Setting app package to hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip. 2015-06-26 17:47:27,288 [main] INFO persist.AppDefinitionPersister - Processing app package/folder /tmp/1435312046889-0/default for appPkg.zip 2015-06-26 17:47:27,290 [main] INFO tools.SliderUtils - Zipping folder /tmp/1435312046889-0/default to /tmp/1435312047288-0/appPkg.zip 2015-06-26 17:47:27,328 [main] INFO agent.AgentClientProvider - Validating app definition hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip 2015-06-26 17:47:27,331 [main] INFO agent.AgentUtils - Reading metainfo at hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip 2015-06-26 17:47:27,450 [main] ERROR metadata.Metainfo - Malformed app definition: Expect application as the top level element for metainfo 2015-06-26 17:47:27,451 [main] INFO client.SliderClient - Error Component MEMCACHED is not a member of application. validating application instance definition 2015-06-26 17:47:27,460 [main] INFO client.SliderClient - {, "internal": { "schema" : "http://example.org/specification/v2.0.0", "metadata" : { "create.hadoop.deployed.info" : "branch-2.6.0 @18e43357c8f927c0695f1e9522859d6a", "create.application.build.info" : "Slider Core-0.80.0-incubating Built against commit# d7e3449fa6 on Java 1.7.0_60 by gsaha", "create.hadoop.build.info" : "2.6.0", "create.time.millis" : "1435312046903", "create.time" : "26 Jun 2015 09:47:26 GMT" } , "global" : { "internal.generated.conf.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/generated", "internal.appdef.dir.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp/appdef", "application.name" : "memcached", "slider.cluster.directory.permissions" : "0770", "internal.addons.dir.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp/addons", "internal.provider.name" : "agent", "internal.data.dir.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/database", "internal.tmp.dir" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp", "internal.snapshot.conf.path" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/snapshot", "slider.data.directory.permissions" : "0770", "internal.container.failure.shortlife" : "60000", "internal.am.tmp.dir" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/tmp/appmaster", "internal.container.failure.threshold" : "5" } , "credentials" : { }, "components" : { } }, "resources": { "schema" : "http://example.org/specification/v2.0.0", "metadata" : { }, "global" : { }, "credentials" : { }, "components" : { "slider-appmaster" : { "yarn.memory" : "1024", "yarn.vcores" : "1", "yarn.component.instances" : "1" } , "MEMCACHED" : { "yarn.memory" : "512", "yarn.role.priority" : "1", "yarn.component.instances" : "2" } } }, "appConf" :{ "schema" : "http://example.org/specification/v2.0.0", "metadata" : { }, "global" : { "site.fs.defaultFS" : "hdfs://dsj-s6:8020", "env.MALLOC_ARENA_MAX" : "4", "site.fs.default.name" : "hdfs://dsj-s6:8020", "zookeeper.path" : "/services/slider/users/hdev/memcached", "zookeeper.quorum" : "localhost:2181", "zookeeper.hosts" : "localhost", "application.def" : "hdfs://dsj-s6:8020/user/hdev/.slider/cluster/memcached/appdef/appPkg.zip" } , "credentials" : { }, "components" : { "slider-appmaster" : { "jvm.heapsize" : "256M" } } }} 2015-06-26 17:47:27,460 [main] ERROR main.ServiceLauncher - Component MEMCACHED is not a member of application. 2015-06-26 17:47:27,462 [main] INFO util.ExitUtil - Exiting with status 77
          Hide
          yongw Yong Wang added a comment -

          I resolved the issue by correcting the metainfo.json. The metainfo content in attached PDF does not work for my slider version. I need use somthing like this:

          {
          "schemaVersion": "2.1",
          "application": {
          "name": "MEMCACHED",
          "components": [
          {
          "name": "MEMCACHED",
          "type": "docker",
          "containers": {
          "container":

          { "name": "memcached", "image": "borja/memcached" }

          }
          }
          ]
          }
          }

          Show
          yongw Yong Wang added a comment - I resolved the issue by correcting the metainfo.json. The metainfo content in attached PDF does not work for my slider version. I need use somthing like this: { "schemaVersion": "2.1", "application": { "name": "MEMCACHED", "components": [ { "name": "MEMCACHED", "type": "docker", "containers": { "container": { "name": "memcached", "image": "borja/memcached" } } } ] } }

            People

            • Assignee:
              thomas_liu thomas liu
              Reporter:
              thomas_liu thomas liu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile