Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Slider 0.91
    • Component/s: app-package
    • Labels:
      None

      Description

      I started working on an app-package for Tomcat.

      https://github.com/apache/incubator-slider/tree/feature/SLIDER-809-tomcat-app-package/app-packages/tomcat

      Some things that need to happen for a generally usable app-package:

      • Maven module in slider for the code
      • Template for server.xml – notably the port to listen on and other options
      • Reference external WAR file
      • Fix exports so allocated port for tomcat is advertised (e.g. can view on appmaster status page)

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 975c2f644050821b18b164bc0d2cf4489bdfa3aa in incubator-slider's branch refs/heads/feature/SLIDER-809-tomcat-app-package from Josh Elser
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=975c2f6 ]

        SLIDER-809 Pulling in initial tomcat app package in main tree.

        What is, hopefully, a working maven module that will build
        the app package for us. These files were enough to get a
        dynamically assigned port from slider and a generated
        server.xml to put that port in!

        Show
        jira-bot ASF subversion and git services added a comment - Commit 975c2f644050821b18b164bc0d2cf4489bdfa3aa in incubator-slider's branch refs/heads/feature/ SLIDER-809 -tomcat-app-package from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=975c2f6 ] SLIDER-809 Pulling in initial tomcat app package in main tree. What is, hopefully, a working maven module that will build the app package for us. These files were enough to get a dynamically assigned port from slider and a generated server.xml to put that port in!
        Hide
        elserj Josh Elser added a comment -

        Just pushed up feature/SLIDER-809-tomcat-app-package so I'm not a hermit in a cave. I got a config template for the server.xml working (as some strife) which was the big hold-back presently. This should allow us to properly advertise where tomcat is running.

        I need to validate that this explicit zip is sufficient to use with Slider (getting late tonight), but the bits inside the war were pulled from the VM I was working on. After that, the goal is probably to figure out how to get the quicklinks to work because I have to use netstat presently to figure out what port Tomcat is listening on

        Show
        elserj Josh Elser added a comment - Just pushed up feature/ SLIDER-809 -tomcat-app-package so I'm not a hermit in a cave. I got a config template for the server.xml working (as some strife) which was the big hold-back presently. This should allow us to properly advertise where tomcat is running. I need to validate that this explicit zip is sufficient to use with Slider (getting late tonight), but the bits inside the war were pulled from the VM I was working on. After that, the goal is probably to figure out how to get the quicklinks to work because I have to use netstat presently to figure out what port Tomcat is listening on
        Hide
        elserj Josh Elser added a comment -

        Found this gem tonight: True Zero Downtime HAProxy Reloads.

        Outlines how Yelp went from about 1s latency with HAProxy to about 20ms latency in when the backend services are changed (granted, I think we'd be happy with 1s latency . IMO, one open problem for Slider specifically for supporting dynamic webservers is how exactly users will transparently use those servers and not have to deal with all the failure handling going on under the hood by the AM and YARN. I remember some chatter about a DNS-based registry (as opposed to ZooKeeper) being suggested which would likely also fit the bill (although I'm not sure at a technical level how DNS copes with service changes), but my hunch was that we could wire up something simple with HAProxy given the information that the AM has about the tomcat containers it's running. If a DNS registry comes about, then we could investigate going that route instead/too.

        Show
        elserj Josh Elser added a comment - Found this gem tonight: True Zero Downtime HAProxy Reloads . Outlines how Yelp went from about 1s latency with HAProxy to about 20ms latency in when the backend services are changed (granted, I think we'd be happy with 1s latency . IMO, one open problem for Slider specifically for supporting dynamic webservers is how exactly users will transparently use those servers and not have to deal with all the failure handling going on under the hood by the AM and YARN. I remember some chatter about a DNS-based registry (as opposed to ZooKeeper) being suggested which would likely also fit the bill (although I'm not sure at a technical level how DNS copes with service changes), but my hunch was that we could wire up something simple with HAProxy given the information that the AM has about the tomcat containers it's running. If a DNS registry comes about, then we could investigate going that route instead/too.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 6d7610238e08868c5cd369091d58e017d45cbb6b in incubator-slider's branch refs/heads/feature/SLIDER-809-tomcat-app-package from Josh Elser
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=6d76102 ]

        SLIDER-809 Fix some discrepancies from the first commit.

        • Fix http_port allocated port definition
        • Use proper file name/paths for tomcat tarball
        • Define the server-xml configuration key
        • Remove unnecessary configuration calls (stop and status)
        Show
        jira-bot ASF subversion and git services added a comment - Commit 6d7610238e08868c5cd369091d58e017d45cbb6b in incubator-slider's branch refs/heads/feature/ SLIDER-809 -tomcat-app-package from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=6d76102 ] SLIDER-809 Fix some discrepancies from the first commit. Fix http_port allocated port definition Use proper file name/paths for tomcat tarball Define the server-xml configuration key Remove unnecessary configuration calls (stop and status)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 940c9a8ad9fc212611a9324c38db4d1d9bc5888f in incubator-slider's branch refs/heads/feature/SLIDER-809-tomcat-app-package from Josh Elser
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=940c9a8 ]

        SLIDER-809 Write up a brief README on what should work for tomcat.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 940c9a8ad9fc212611a9324c38db4d1d9bc5888f in incubator-slider's branch refs/heads/feature/ SLIDER-809 -tomcat-app-package from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=940c9a8 ] SLIDER-809 Write up a brief README on what should work for tomcat.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 61641c45d83a3d41551e8c5f833ee54abaea9c84 in incubator-slider's branch refs/heads/feature/SLIDER-809-tomcat-app-package from Josh Elser
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=61641c4 ]

        SLIDER-809 Mention how to build the app-package

        Show
        jira-bot ASF subversion and git services added a comment - Commit 61641c45d83a3d41551e8c5f833ee54abaea9c84 in incubator-slider's branch refs/heads/feature/ SLIDER-809 -tomcat-app-package from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=61641c4 ] SLIDER-809 Mention how to build the app-package
        Hide
        elserj Josh Elser added a comment -

        Finally got back to this. Spent tonight verifying that the maven build is actually capable of making a functional app package. Good thing I did, because it did need some love.

        Show
        elserj Josh Elser added a comment - Finally got back to this. Spent tonight verifying that the maven build is actually capable of making a functional app package. Good thing I did, because it did need some love.
        Hide
        elserj Josh Elser added a comment - - edited

        Picked this up again. I implemented a new property in the app configuration file which contains a list of HDFS URIs which will get set as localized resources in the container request. After this point, the python scripts in the app package can do whatever it chooses – for Tomcat, we can copy all of these files which end in .war into the webapps directory so that they are automatically deployed.

        I also implemented a single tomcat-users.xml mapping so that a user can specify a username/password and use the built-in Tomcat manager via the browser.

        With the change over in SLIDER-1012, I have also been able to access the URL via the AM's REST service, but it's not displayed on the UI which is frustrating. The information is contained in the exports call. I think this can be a general improvement to the webui in that we can render all exports/quicklinks in the UI and prevent the need for the user to make additional calls.

        Still need to make some tests...

        Show
        elserj Josh Elser added a comment - - edited Picked this up again. I implemented a new property in the app configuration file which contains a list of HDFS URIs which will get set as localized resources in the container request. After this point, the python scripts in the app package can do whatever it chooses – for Tomcat, we can copy all of these files which end in .war into the webapps directory so that they are automatically deployed. I also implemented a single tomcat-users.xml mapping so that a user can specify a username/password and use the built-in Tomcat manager via the browser. With the change over in SLIDER-1012 , I have also been able to access the URL via the AM's REST service, but it's not displayed on the UI which is frustrating. The information is contained in the exports call. I think this can be a general improvement to the webui in that we can render all exports/quicklinks in the UI and prevent the need for the user to make additional calls. Still need to make some tests...
        Hide
        elserj Josh Elser added a comment -

        Ok, only testing to do now.

        https://github.com/joshelser/slider/tree/feature/SLIDER-809-tomcat-app-package

        That branch contains changes here, rebased on develop, includes fengyongshe's patch from SLIDER-1012, and changes for SLIDER-1040 and SLIDER-1041.

        Will try to get some testing in place tomorrow, and put this code to a rest for once.

        Show
        elserj Josh Elser added a comment - Ok, only testing to do now. https://github.com/joshelser/slider/tree/feature/SLIDER-809-tomcat-app-package That branch contains changes here, rebased on develop, includes fengyongshe 's patch from SLIDER-1012 , and changes for SLIDER-1040 and SLIDER-1041 . Will try to get some testing in place tomorrow, and put this code to a rest for once.
        Hide
        elserj Josh Elser added a comment -

        Shoot. It looks like choosing the port for the containers is done per application and not per container.

        When I flexed a cluster from 1 tomcat server to 2 (on my single node yarn "cluster"), the second container came up but tried to bind the same port which failed.

        Show
        elserj Josh Elser added a comment - Shoot. It looks like choosing the port for the containers is done per application and not per container. When I flexed a cluster from 1 tomcat server to 2 (on my single node yarn "cluster"), the second container came up but tried to bind the same port which failed.
        Hide
        jmaron Jonathan Maron added a comment -

        That doesn't seem possible. I believe it's the agent that selects the available port and relates it back to the AM, so that would have to work on a per-container basis. I'll double check when I get a chance.

        Show
        jmaron Jonathan Maron added a comment - That doesn't seem possible. I believe it's the agent that selects the available port and relates it back to the AM, so that would have to work on a per-container basis. I'll double check when I get a chance.
        Hide
        elserj Josh Elser added a comment -

        Ok, it's very possible it's just something dumb I did . Trying to get some tests working before I'll look at this, but I appreciate you looking at it (helps give me some direction).

        Show
        elserj Josh Elser added a comment - Ok, it's very possible it's just something dumb I did . Trying to get some tests working before I'll look at this, but I appreciate you looking at it (helps give me some direction).
        Hide
        jmaron Jonathan Maron added a comment -

        OK - so I'm not certain how you ended up allocating the same port, unless it's a unique timing issue. The code in agent (CustomServiceOrchestrator.allocatePort()) will attempt open a socket from either a provided list or randomly (passing in 0 to socket.bind). If successful, it will close the socket and relate the port back to the AM during its handshake. I suppose it's possible that if two containers are starting up on the same host and and the code is somehow executed sequentially on both agents they may relate back the same port number?

        Show
        jmaron Jonathan Maron added a comment - OK - so I'm not certain how you ended up allocating the same port, unless it's a unique timing issue. The code in agent (CustomServiceOrchestrator.allocatePort()) will attempt open a socket from either a provided list or randomly (passing in 0 to socket.bind). If successful, it will close the socket and relate the port back to the AM during its handshake. I suppose it's possible that if two containers are starting up on the same host and and the code is somehow executed sequentially on both agents they may relate back the same port number?
        Hide
        elserj Josh Elser added a comment -

        I might just be missing {PER_CONTAINER}} in the appConfig.json...

        Show
        elserj Josh Elser added a comment - I might just be missing { PER_CONTAINER }} in the appConfig.json...
        Hide
        jmaron Jonathan Maron added a comment -

        Yeah...or that

        Show
        jmaron Jonathan Maron added a comment - Yeah...or that
        Hide
        elserj Josh Elser added a comment -

        Hahahaha, yup, that was it. Thanks again for looking into my stupidity, Jon. Much appreciated.

        Show
        elserj Josh Elser added a comment - Hahahaha, yup, that was it. Thanks again for looking into my stupidity, Jon. Much appreciated.
        Hide
        elserj Josh Elser added a comment -

        Ran into a failure in org.apache.slider.providers.agent.TestBuildBasicAgent

        testBuildMultipleRoles(org.apache.slider.providers.agent.TestBuildBasicAgent)  Time elapsed: 3.264 sec  <<< ERROR!
        java.io.FileNotFoundException: file:/Users/jelser/.slider/cluster/testbuildmultipleroles
        	at org.apache.slider.core.persist.ConfPersister.acquireReadLock(ConfPersister.java:177)
        	at org.apache.slider.core.persist.ConfPersister.load(ConfPersister.java:277)
        	at org.apache.slider.client.SliderClient.loadPersistedClusterDescription(SliderClient.java:3137)
        	at org.apache.slider.providers.agent.TestBuildBasicAgent.testBuildMultipleRoles(TestBuildBasicAgent.groovy:141)
        

        Not sure why I didn't see this on the 0.90.2-incubating release testing, but I'm getting it on develop and my branch.

        My plan is to investigate this a bit more tonight and then merge this stuff in (includes the other aforementioned related issues)

        Show
        elserj Josh Elser added a comment - Ran into a failure in org.apache.slider.providers.agent.TestBuildBasicAgent testBuildMultipleRoles(org.apache.slider.providers.agent.TestBuildBasicAgent) Time elapsed: 3.264 sec <<< ERROR! java.io.FileNotFoundException: file:/Users/jelser/.slider/cluster/testbuildmultipleroles at org.apache.slider.core.persist.ConfPersister.acquireReadLock(ConfPersister.java:177) at org.apache.slider.core.persist.ConfPersister.load(ConfPersister.java:277) at org.apache.slider.client.SliderClient.loadPersistedClusterDescription(SliderClient.java:3137) at org.apache.slider.providers.agent.TestBuildBasicAgent.testBuildMultipleRoles(TestBuildBasicAgent.groovy:141) Not sure why I didn't see this on the 0.90.2-incubating release testing, but I'm getting it on develop and my branch. My plan is to investigate this a bit more tonight and then merge this stuff in (includes the other aforementioned related issues)
        Hide
        elserj Josh Elser added a comment -

        Filed: SLIDER-1042

        Show
        elserj Josh Elser added a comment - Filed: SLIDER-1042
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit dd121b18c26c7272887ca1e09659adac85916828 in incubator-slider's branch refs/heads/develop from Josh Elser
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=dd121b1 ]

        SLIDER-809 First implementation of an Apache Tomcat app-package

        Uses Apache Tomcat 8.0.30

        • Includes basic documentation
        • Supports basic tomcat-users.xml configuration
        • Implements extra resource localization support into containers
        • Include a basic IT
        • Exports server address to registry
        • Update rat-check for new additions
        Show
        jira-bot ASF subversion and git services added a comment - Commit dd121b18c26c7272887ca1e09659adac85916828 in incubator-slider's branch refs/heads/develop from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=dd121b1 ] SLIDER-809 First implementation of an Apache Tomcat app-package Uses Apache Tomcat 8.0.30 Includes basic documentation Supports basic tomcat-users.xml configuration Implements extra resource localization support into containers Include a basic IT Exports server address to registry Update rat-check for new additions
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 0992e85500239d88f77cbd76f142b1daf02d5219 in incubator-slider's branch refs/heads/develop from Steve Loughran
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=0992e85 ]

        Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop

        Show
        jira-bot ASF subversion and git services added a comment - Commit 0992e85500239d88f77cbd76f142b1daf02d5219 in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=0992e85 ] Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider into develop 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-slider: SLIDER-1042 Use the correct cluster name when fetching the description SLIDER-1041 Fix consistency for exports, enumerate all exports on HTML index. SLIDER-1040 Better formatting on the AM's web ui. SLIDER-1012 Modify server and quicklinks for Tomcat (fengyongshe) SLIDER-809 First implementation of an Apache Tomcat app-package SLIDER-1039 Run python tests from target directory

          People

          • Assignee:
            elserj Josh Elser
            Reporter:
            elserj Josh Elser
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development