Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Labels:
      None

      Description

      We want to be able to load and unload applications dynamically. Here is some background information:

      Once a server is started, we expect it to run indefinitely until there is a failure or we need to upgrade the framework software or we change the cluster configuration. In S4, the cluster is symmetric, that is, all the nodes are identical, with the same code base and the same configuration.

      Applications are loaded dynamically after the cluster is started. If app1 depends on app2, then app2 needs to use an EventSource to publish events. Apps must be able to find EventSource and subscribe to it during initialization.

      When an app is unloaded, it will close its EventSource(s). This in turn, will close all the streams that are subscribed. The apps that own the streams are responsible for taking action once their dependencies are gone.

      Possible roadmap:

      M1:

      • Server starts with no apps.
      • Apps are loaded during initialization, for example by searching an
        app directory.
      • Apps are initialized (no dependencies among apps).
      • Apps are started.

      M2:

      • Add dependencies among apps using EventSource.

      M3:

      • Add runtime loading/unloading functionality

      Leo implemented a solution using JBOSS Modules which seems simpler than using OSGI. Modules seems to work but is not well documented which makes it hard to use without understanding the low level details. See the commented out code here: https://github.com/leoneu/s4-piper/blob/master/subprojects/s4-core/src/main/java/io/s4/core/Server.java

      From Adam:

      Yes I have a few ideas but I need couple days to implement them.
      First, is to build S4 and examples as proper OSGi bundles using gradle osgi plugin http://gradle.org/osgi_plugin.
      Its using bnd tool http://www.aqute.biz/Bnd/Bnd.
      Then embed osgi container (using standard osgi api) in Controller instead of JBoss modules see http://felix.apache.org/site/apache-felix-framework-launching-and-embedding.html.
      and configure directory from which felix should auto deploy our examples http://felix.apache.org/site/apache-felix-framework-usage-documentation.html#ApacheFelixFrameworkUsageDocumentation-autodeploy. For dynamic deployment we can use file install bundle http://felix.apache.org/site/apache-felix-file-install.html, it will watch directory from which it loads new bundles.
      To configure properly examples we can use ipojo http://felix.apache.org/site/apache-felix-ipojo.html or declarative services http://felix.apache.org/site/apache-felix-service-component-runtime.html, they both using service component model and are not intrusive in the code.
      That will be enough to load properly the apps.

      Later we have to think how we want to use osgi in s4, just to load apps or properly use service management. Bundles are hiding the internals from other bundles and should communicate through well defined services. And also I don't have clear picture how s4 is going to be distributed (have to look more at https://github.com/s4/s4). Maybe you can write me some small description of s4 concept.

      ...

      I've implemented ideas which I sent you before. Now I can load the apps using embedded osgi container but there is still class loading issue (com.esotericsoftware.kryo.SerializationException: Unable to find class: io.s4.example.counter.UserEvent). To be able to run s4 I had to put plenty system packages in configuration file for felix.
      Around 15 jars defined in libraries are not osgi bundles (don't have proper manifest file). The biggest issue are dependencies and classloading (for example Kryo is trying to load UserEvent class but that class is not visible for him so Class.forName is not going to work).I will try to fix these problems, in the meantime you can have a look at my repo https://github.com/adamwojtuniak/s4-piper. I added task createRepo, it will create bundles directory with all dependencies and project jars. When you launch Main class from s4-core, it launches embedded felix which will auto deploy all bundles located in bundles directory. In the console you will see felix gogo shell ( http://www.packtpub.com/article/apache-felix-gogo ).

      From Henry:

      Ugh, I was actually trying to avoid OSGI if I can =)

        Issue Links

          Activity

          Hide
          Adam Wojtuniak added a comment -

          Hi Leo,

          We can't avoid class loading issue even if we want only to load apps dynamically. Every app is going to have dependencies to s4 (plus s4 dependencies) and also can happen that app can have their own dependencies, all these dependencies must be visible to app classloader. One solution is to expose all these dependencies as a system packages then they will be visible to all bundles in the osgi container but that is a hack which are going to lead to maintenance nightmare (every time is new dependency it must be added to system package). Class.forName is another problem, for example kryo lib is loading classes which in the container aren't visible to kryo until kryo will be bundlized with DynamicImport-Package: *. These problems can't be avoided and is no matter what solution is used (JBossModules or Felix etc.). One solution is to find proper bundlized version of some s4 dependencies or wrap them using http://www.aqute.biz/Code/Bnd and put for some of them DynamicImport-Package: *.

          Cheers,
          Adam

          Show
          Adam Wojtuniak added a comment - Hi Leo, We can't avoid class loading issue even if we want only to load apps dynamically. Every app is going to have dependencies to s4 (plus s4 dependencies) and also can happen that app can have their own dependencies, all these dependencies must be visible to app classloader. One solution is to expose all these dependencies as a system packages then they will be visible to all bundles in the osgi container but that is a hack which are going to lead to maintenance nightmare (every time is new dependency it must be added to system package). Class.forName is another problem, for example kryo lib is loading classes which in the container aren't visible to kryo until kryo will be bundlized with DynamicImport-Package: *. These problems can't be avoided and is no matter what solution is used (JBossModules or Felix etc.). One solution is to find proper bundlized version of some s4 dependencies or wrap them using http://www.aqute.biz/Code/Bnd and put for some of them DynamicImport-Package: *. Cheers, Adam
          Hide
          Adam Wojtuniak added a comment -

          Hi Leo,

          Maybe it is possible with JBossModules, I haven't noticed that they are not really OSGi container.
          "JBoss Modules is designed to work with any existing library or application without changes, and its simple naming and resolution strategy is what makes that possible. Unlike OSGi, JBoss Modules does not implement a container; rather, it is a thin bootstrap wrapper for executing an application in a modular environment. The moment your application takes control, the modular environment is ready to load and link modules as needed. Furthermore, modules are never loaded (not even for resolution purposes) until required by a dependency, meaning that the performance of a modular application depends only on the number of modules actually used (and when they are used), rather than the total number of modules in the system. And, they may be unloaded by the user at any time."

          I am going to investigate that what I can do with that solution.

          Regards,
          Adam

          On Thu, Oct 13, 2011 at 4:44 PM, Leo Neumeyer wrote:
          When I tried jboss modules, I didn't have to do any of this. Maybe it
          is worth investing more times in getting jboss modules to work.

          I started a jira for this topic here:
          https://issues.apache.org/jira/browse/S4-4

          please add to this issue.

          -leo

          Show
          Adam Wojtuniak added a comment - Hi Leo, Maybe it is possible with JBossModules, I haven't noticed that they are not really OSGi container. "JBoss Modules is designed to work with any existing library or application without changes, and its simple naming and resolution strategy is what makes that possible. Unlike OSGi, JBoss Modules does not implement a container; rather, it is a thin bootstrap wrapper for executing an application in a modular environment. The moment your application takes control, the modular environment is ready to load and link modules as needed. Furthermore, modules are never loaded (not even for resolution purposes) until required by a dependency, meaning that the performance of a modular application depends only on the number of modules actually used (and when they are used), rather than the total number of modules in the system. And, they may be unloaded by the user at any time." I am going to investigate that what I can do with that solution. Regards, Adam On Thu, Oct 13, 2011 at 4:44 PM, Leo Neumeyer wrote: When I tried jboss modules, I didn't have to do any of this. Maybe it is worth investing more times in getting jboss modules to work. I started a jira for this topic here: https://issues.apache.org/jira/browse/S4-4 please add to this issue. -leo
          Hide
          Leo Neumeyer added a comment -

          Great, I was able to get the counter example to work once I figured out how to set up the XML descriptor. The only error was related to logging but the application worked fine. Check the history in the Server class.

          Show
          Leo Neumeyer added a comment - Great, I was able to get the counter example to work once I figured out how to set up the XML descriptor. The only error was related to logging but the application worked fine. Check the history in the Server class.
          Hide
          Leo Neumeyer added a comment -

          I just saw this. An onlien chat on mudularity. Chance to ask questions and share ideas:
          http://planet.jboss.org/post/javaone_jigsaw_jboss_modules_and_the_community_you

          The chat is scheduled to be held at 10:00 A.M. US Pacific time on Friday, October 21, 2011 in the ##java-modularity channel at irc.freenode.net (web chat link). If you have any experience you'd like to share, either as a user or a developer of an existing build or modularity system, or you just want to see what happens, please join in the discussion. It's undecided at the moment but the discussion may be moderated, depending on attendance. Note! Make sure you have a registered nickname ahead of time so you can join in without any issues when the time arrives.

          Show
          Leo Neumeyer added a comment - I just saw this. An onlien chat on mudularity. Chance to ask questions and share ideas: http://planet.jboss.org/post/javaone_jigsaw_jboss_modules_and_the_community_you The chat is scheduled to be held at 10:00 A.M. US Pacific time on Friday, October 21, 2011 in the ##java-modularity channel at irc.freenode.net (web chat link). If you have any experience you'd like to share, either as a user or a developer of an existing build or modularity system, or you just want to see what happens, please join in the discussion. It's undecided at the moment but the discussion may be moderated, depending on attendance. Note! Make sure you have a registered nickname ahead of time so you can join in without any issues when the time arrives.
          Hide
          Leo Neumeyer added a comment -

          I'm looking at using classloaders directly rather than rely on a framework. One option is to use a MultiClassLoader:

          http://stackoverflow.com/questions/148681/unloading-classes-in-java

          Apache openjpa uses this approach. See:
          http://openjpa.apache.org/builds/1.2.2/apidocs/org/apache/openjpa/lib/util/MultiClassLoader.html
          Source: openjpa-lib/src/main/java/org/apache/openjpa/lib/util/MultiClassLoader.java

          Show
          Leo Neumeyer added a comment - I'm looking at using classloaders directly rather than rely on a framework. One option is to use a MultiClassLoader: http://stackoverflow.com/questions/148681/unloading-classes-in-java Apache openjpa uses this approach. See: http://openjpa.apache.org/builds/1.2.2/apidocs/org/apache/openjpa/lib/util/MultiClassLoader.html Source: openjpa-lib/src/main/java/org/apache/openjpa/lib/util/MultiClassLoader.java
          Hide
          Leo Neumeyer added a comment -

          I played with MultiClassLoader and ran a HelloApp test. Seems like a good approach at least for milestone 1:

          https://github.com/leoneu/s4-piper/blob/modules/subprojects/s4-core/src/main/java/io/s4/core/Server.java

          To run the test you need to create a HelloApp class with a default package. The class should extend class and print something in the start() method. Move HelloApp.class to /tmp/HelloApp.impl for testing. (Different extension to make sure it is not a standard class found in the classpath.)

          I propose using this approach for now. What we need to do is pass an app path in the properties file and read apps from there every X seconds. When the app is removed, then we should stop it and unload.

          Jars loaded by the framework will be available to apps.

          Show
          Leo Neumeyer added a comment - I played with MultiClassLoader and ran a HelloApp test. Seems like a good approach at least for milestone 1: https://github.com/leoneu/s4-piper/blob/modules/subprojects/s4-core/src/main/java/io/s4/core/Server.java To run the test you need to create a HelloApp class with a default package. The class should extend class and print something in the start() method. Move HelloApp.class to /tmp/HelloApp.impl for testing. (Different extension to make sure it is not a standard class found in the classpath.) I propose using this approach for now. What we need to do is pass an app path in the properties file and read apps from there every X seconds. When the app is removed, then we should stop it and unload. Jars loaded by the framework will be available to apps.
          Hide
          Leo Neumeyer added a comment -

          My notes from today's discussion:

          • Matthieu will look at app loading
          • Our problem is constrained because the entry point is always a class of type App. All interaction with the app is via the App methods: ini,t start, close, etc and the APi only uses known framework classes: Event, EventSource, etc.)
          • Each will have its own classloader to make it easy to release and GC.
          • we will test using Matthieu's forked node unit tests to validate
          • create a separate minimal project as template for creating apps. Requirement: output should be a single JAR file (we could call it S4R: HelloApp.s4r to indicate that it has valid s4 attributes in the manifest).
          • The S4R file is dropped in a dir. Server will load/unload based on what files are in the dir.
          • Loaded apps are tracked by Server. Apps must must have a unique ID. Here is an idea: S4R includes properties to identify app: for example:

          S4-App-Description: An app to count words.
          S4-App-OwnerId: 79f0eea01c
          ...

          Server (or ZK) will assign a unique ID (0,1,2,...). Server will map AppIDs to AppModules. An AppModule has a reference to the loaded app, the S4-App- properties, deployment time, and any other administrative info. In this way we can always identify the loaded app and guarantee that there are unique IDs in the cluster.

          • No app to app communication in first milestone.

          For later:

          • Test app to app communication using EventSource.
          • Coordinate load/unload using ZK and a deployment server.
          • Restrict loaded App so they cannot execute System.exit(), they cannot create threads, etc.
          • Test app loading by creating lots of apps and releasing them, memory should not grow indefinitely and we can probably track that apps get garbage collected.
          • what happens if jar in app is diff than framework jar. It should use its own jar without interfering with system jar.
          Show
          Leo Neumeyer added a comment - My notes from today's discussion: Matthieu will look at app loading Our problem is constrained because the entry point is always a class of type App. All interaction with the app is via the App methods: ini,t start, close, etc and the APi only uses known framework classes: Event, EventSource, etc.) Each will have its own classloader to make it easy to release and GC. we will test using Matthieu's forked node unit tests to validate create a separate minimal project as template for creating apps. Requirement: output should be a single JAR file (we could call it S4R: HelloApp.s4r to indicate that it has valid s4 attributes in the manifest). The S4R file is dropped in a dir. Server will load/unload based on what files are in the dir. Loaded apps are tracked by Server. Apps must must have a unique ID. Here is an idea: S4R includes properties to identify app: for example: S4-App-Description: An app to count words. S4-App-OwnerId: 79f0eea01c ... Server (or ZK) will assign a unique ID (0,1,2,...). Server will map AppIDs to AppModules. An AppModule has a reference to the loaded app, the S4-App- properties, deployment time, and any other administrative info. In this way we can always identify the loaded app and guarantee that there are unique IDs in the cluster. No app to app communication in first milestone. For later: Test app to app communication using EventSource. Coordinate load/unload using ZK and a deployment server. Restrict loaded App so they cannot execute System.exit(), they cannot create threads, etc. Test app loading by creating lots of apps and releasing them, memory should not grow indefinitely and we can probably track that apps get garbage collected. what happens if jar in app is diff than framework jar. It should use its own jar without interfering with system jar.
          Hide
          Matthieu Morel added a comment -

          I pushed some improvements to the current multiclassloader approach to leo's master branch for s4-piper.

          We are now able to load applications from a given application directory, and the code of an application runs in isolation from other applications, including dynamically generated classes.

          I included some regression tests and I also checked that the mechanism works correctly for packaging and deploying an existing app from s4-examples.

          In addition, we also have a basis for s4r application archives: jar file with a manifest identifying app entry point (application class).

          This corresponds to M1 above, but there are more improvements to be included:

          Next steps:
          1. provide a mechanism for users to generate s4r packages by encapsulating both application code and related dependencies
          2. define an API for fetching s4r packages from remote repositories (with implementation using different underlying technologies: file system etc... and provide some security features, though maybe not in a first version)
          3. synchronize deployment (and startup of applications) through zookeeper

          Show
          Matthieu Morel added a comment - I pushed some improvements to the current multiclassloader approach to leo's master branch for s4-piper. We are now able to load applications from a given application directory, and the code of an application runs in isolation from other applications, including dynamically generated classes. I included some regression tests and I also checked that the mechanism works correctly for packaging and deploying an existing app from s4-examples. In addition, we also have a basis for s4r application archives: jar file with a manifest identifying app entry point (application class). This corresponds to M1 above, but there are more improvements to be included: Next steps: 1. provide a mechanism for users to generate s4r packages by encapsulating both application code and related dependencies 2. define an API for fetching s4r packages from remote repositories (with implementation using different underlying technologies: file system etc... and provide some security features, though maybe not in a first version) 3. synchronize deployment (and startup of applications) through zookeeper
          Hide
          Leo Neumeyer added a comment -

          Nice progress!

          • Moved apps path configuration to Module.
          • Moves Sender/Receiver outside the apps loop because they are singletons.
          • In AppLoadingTest, something is breaking, seems to be related to using System.getProperty("user.dir") which is a relative dir. When running from Eclipse it gets the ../build path. Because I didn't run Gradle, the dir build doesn't exist and the test crashes with a NullPointerE. I'm not a 100% sure but ran out of time. I will push to a branch called broken so you can merge some of my changes and throw others.
          Show
          Leo Neumeyer added a comment - Nice progress! Moved apps path configuration to Module. Moves Sender/Receiver outside the apps loop because they are singletons. In AppLoadingTest, something is breaking, seems to be related to using System.getProperty("user.dir") which is a relative dir. When running from Eclipse it gets the ../build path. Because I didn't run Gradle, the dir build doesn't exist and the test crashes with a NullPointerE. I'm not a 100% sure but ran out of time. I will push to a branch called broken so you can merge some of my changes and throw others.
          Hide
          Matthieu Morel added a comment -

          I fixed the issue you mentioned in apploading test and pushed it to your master branch (includes your own changes).

          This was due to gradle and eclipse using different output directories for compiled classes. This is actually deliberately configured by gradle, and it seems a good idea to leave it like that. The issue only affected the apploading test, and is fixed by resolving the output directory for test files at runtime, taking into account the execution environment (see my comments in TestUtils#findDirForCompiledTestClasses() )

          Show
          Matthieu Morel added a comment - I fixed the issue you mentioned in apploading test and pushed it to your master branch (includes your own changes). This was due to gradle and eclipse using different output directories for compiled classes. This is actually deliberately configured by gradle, and it seems a good idea to leave it like that. The issue only affected the apploading test, and is fixed by resolving the output directory for test files at runtime, taking into account the execution environment (see my comments in TestUtils#findDirForCompiledTestClasses() )
          Hide
          Henry Saputra added a comment -

          Has module branch being merged back to master? Was wondering if fix for S4-20 is push to master or module.

          Show
          Henry Saputra added a comment - Has module branch being merged back to master? Was wondering if fix for S4-20 is push to master or module.
          Hide
          Matthieu Morel added a comment -

          @Henry : S4-20 is currently in a separate project https://github.com/leoneu/s4-piper-app

          Show
          Matthieu Morel added a comment - @Henry : S4-20 is currently in a separate project https://github.com/leoneu/s4-piper-app
          Hide
          Matthieu Morel added a comment - - edited

          I added tests for the deployment of multiple applications, on top of the deployment manager updates:
          https://github.com/matthieumorel/s4-piper/tree/deployment-manager

          Classes with same signatures in different applications are correctly independently loaded, even when referenced through reflection.

          --> it works well !

          We still have to :

          • test loading of classes dynamically generated from independent applications
          • test library incompatibilities between libraries from S4 application packages and libraries from the S4 platform
          Show
          Matthieu Morel added a comment - - edited I added tests for the deployment of multiple applications, on top of the deployment manager updates: https://github.com/matthieumorel/s4-piper/tree/deployment-manager Classes with same signatures in different applications are correctly independently loaded, even when referenced through reflection. --> it works well ! We still have to : test loading of classes dynamically generated from independent applications test library incompatibilities between libraries from S4 application packages and libraries from the S4 platform
          Hide
          Matthieu Morel added a comment -

          At this point I'm not sure we really want to be able to deploy multiple applications on the same S4 node. It only adds isolation issues, and also adds a lot of complexity for correctly implementing a system with multiple classloader. And it is not needed for dynamic app loading.

          So I would suggest to only allow 1 app to be deployed on 1 node. At least for now.
          Of course we still use the same packaging mechanism for bundling S4 apps into s4r archives.

          And for avoiding dependency issues between S4 apps and S4 platform, we could look into building a "shaded" s4 archive, for instance using the maven shade plugin http://maven.apache.org/plugins/maven-shade-plugin/

          Show
          Matthieu Morel added a comment - At this point I'm not sure we really want to be able to deploy multiple applications on the same S4 node. It only adds isolation issues, and also adds a lot of complexity for correctly implementing a system with multiple classloader. And it is not needed for dynamic app loading. So I would suggest to only allow 1 app to be deployed on 1 node. At least for now. Of course we still use the same packaging mechanism for bundling S4 apps into s4r archives. And for avoiding dependency issues between S4 apps and S4 platform, we could look into building a "shaded" s4 archive, for instance using the maven shade plugin http://maven.apache.org/plugins/maven-shade-plugin/
          Hide
          Flavio Junqueira added a comment -

          So I would suggest to only allow 1 app to be deployed on 1 node.

          +1, I agree.

          Show
          Flavio Junqueira added a comment - So I would suggest to only allow 1 app to be deployed on 1 node. +1, I agree.
          Hide
          Matthieu Morel added a comment -

          If we choose to avoid conflicting dependencies between apps and platform through shading, we could actually decide to "shade" the platform itself, instead of the apps. Might be easier from a user point of view.

          And of course, shading would also probably simplify the classloading mechanism (we'd only need to fetch libraries from a remote location, actual classloading could be handled by the parent classloader).

          Show
          Matthieu Morel added a comment - If we choose to avoid conflicting dependencies between apps and platform through shading, we could actually decide to "shade" the platform itself, instead of the apps. Might be easier from a user point of view. And of course, shading would also probably simplify the classloading mechanism (we'd only need to fetch libraries from a remote location, actual classloading could be handled by the parent classloader).
          Hide
          Matthieu Morel added a comment -

          S4-22 implemented a suitable deployment model for S4, as well as inter-app communication
          S4-72 simplified and improved classloading and application packaging

          As a result, dynamic app loading is effectively implemented, so I'm marking this ticket as resolved.

          Note that S4-80 will address shading

          Show
          Matthieu Morel added a comment - S4-22 implemented a suitable deployment model for S4, as well as inter-app communication S4-72 simplified and improved classloading and application packaging As a result, dynamic app loading is effectively implemented, so I'm marking this ticket as resolved. Note that S4-80 will address shading

            People

            • Assignee:
              Matthieu Morel
              Reporter:
              Leo Neumeyer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development