Apache S4
  1. Apache S4
  2. S4-60

Development mode for rapid application development and testing

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0
    • Labels:
      None

      Description

      During the development phase of S4 applications, we need to be able to avoid the "build-package-deploy" cycle, which rapidly becomes cumbersome.

      Ideally, we'd be able to start S4 nodes and directly run the application from the classpath.

      This should be possible by providing a "DevelopmentDeploymentManager", that simply reads an application classname as a parameter, and directly loads this class and therefore the application. The only requirement is to have application classes and dependencies in the classpath.

        Activity

        Hide
        Matthieu Morel added a comment -

        Currently, this development mode is already available by running the org.apache.s4.core.Main class, and requires:

        • the app class to be in the classpath
        • a subcluster to be already defined
        • main parameters
          • -appClass=<full.name.of.class>
          • -c=<name of aforementioned subcluster>
          • any other relevant parameter, e.g. -namedStringParameter=output.stream.name:MyOutputStream
        Show
        Matthieu Morel added a comment - Currently, this development mode is already available by running the org.apache.s4.core.Main class, and requires: the app class to be in the classpath a subcluster to be already defined main parameters -appClass=<full.name.of.class> -c=<name of aforementioned subcluster> any other relevant parameter, e.g. -namedStringParameter=output.stream.name:MyOutputStream
        Hide
        Matthieu Morel added a comment -

        The current mode still requires a developer to:

        • start a Zookeeper instance (s4 zkServer)
        • define a subcluster in Zookeeper

        We could simplify the process by merging these operations with defaults in an s4 task such as: s4 zkServer -testMode. It would automatically create 2 subclusters testCluster1 and testCluster2 with 1 node each, listening on predefined ports. (Defaults would be overridable.)

        Show
        Matthieu Morel added a comment - The current mode still requires a developer to: start a Zookeeper instance ( s4 zkServer ) define a subcluster in Zookeeper We could simplify the process by merging these operations with defaults in an s4 task such as: s4 zkServer -testMode . It would automatically create 2 subclusters testCluster1 and testCluster2 with 1 node each, listening on predefined ports. (Defaults would be overridable.)
        Hide
        Daniel Gómez Ferro added a comment -

        Thanks for the pointer Matthieu, this greatly improves the development speed!

        Making it easier to start a ZK server and defining clusters is also a good idea. Instead of a -testMode option we could let the user specify the clusters inline with sensible defaults (1 task each, etc): s4 zkServer -c=cluster1,cluster2

        Show
        Daniel Gómez Ferro added a comment - Thanks for the pointer Matthieu, this greatly improves the development speed! Making it easier to start a ZK server and defining clusters is also a good idea. Instead of a -testMode option we could let the user specify the clusters inline with sensible defaults (1 task each, etc): s4 zkServer -c=cluster1,cluster2
        Hide
        Matthieu Morel added a comment -

        I uploaded a modification in branch S4-60 that allows to specify clusters configuration inline when starting a Zookeeper instance. You would do something like:

        s4 zkServer -clusters={c=cluster1:flp=13000:nbTasks=1},{c=cluster2:flp=14000:nbTasks=10} 

        And there is also a test mode with by default 2 clusters named "testCluster1" and "testCluster2" with 1 task each, that you may start like this:

        s4 zkServer -t

        Please let me know if you have other suggestions/improvements, otherwise I'll integrate that into the main branch.

        Commit diff is : https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commitdiff;h=e438fbff

        Show
        Matthieu Morel added a comment - I uploaded a modification in branch S4-60 that allows to specify clusters configuration inline when starting a Zookeeper instance. You would do something like: s4 zkServer -clusters={c=cluster1:flp=13000:nbTasks=1},{c=cluster2:flp=14000:nbTasks=10} And there is also a test mode with by default 2 clusters named "testCluster1" and "testCluster2" with 1 task each, that you may start like this: s4 zkServer -t Please let me know if you have other suggestions/improvements, otherwise I'll integrate that into the main branch. Commit diff is : https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commitdiff;h=e438fbff
        Hide
        Daniel Gómez Ferro added a comment -

        As I said before, I think this change is great. I tried the patch and it works perfectly. The only confusing thing is that ZkServer requires the parameter -s4ScriptPath, even though it doesn't use it. It's only used by createApp if my understanding is correct.

        Show
        Daniel Gómez Ferro added a comment - As I said before, I think this change is great. I tried the patch and it works perfectly. The only confusing thing is that ZkServer requires the parameter -s4ScriptPath , even though it doesn't use it. It's only used by createApp if my understanding is correct.
        Hide
        Matthieu Morel added a comment -

        set the -s4ScriptPath option as optional (so that there is no confusion when call the Main class from an IDE) and merged into piper https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=6f872878b4bb1976303ee52c95425e00419b3dcf

        Show
        Matthieu Morel added a comment - set the -s4ScriptPath option as optional (so that there is no confusion when call the Main class from an IDE) and merged into piper https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=6f872878b4bb1976303ee52c95425e00419b3dcf

          People

          • Assignee:
            Matthieu Morel
            Reporter:
            Matthieu Morel
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development