Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0
    • Labels:
      None

      Description

      S4RLoader should load resources from jars. This could be accomplished overriding findResource() or directly getResource()/getResourceAsStream().

      User resources should also take precedence over dependencies resources. Right now dependencies are exploded and packaged on a single JAR causing all resources to be at the same level, possibly conflicting with each other. There are a couple of options to solve this:

      • Specify a directory for user resources which will be searched before any other directory
      • Don't explode dependencies, packaging them as JARs. Only user resources would be at top level

        Activity

        Hide
        Matthieu Morel added a comment -

        I uploaded a patch in branch S4-72.

        • It replaces the "multiple classloaders" scheme with a mechanism inspired from Hadoop's classloading scheme.
        • S4R are generated differently: dependencies are not exploded then gathered again. Instead, we place them in a /lib directory in the S4R, whereas the app jar is placed in /app. After retrieval, nodes unzip the S4R in a given directory and load class/resources from there.

        Benefits:

        • user resources take precedence over dependencies resources: upon class/resource loading, the /app dir takes precedence over the /lib dir (which fulfills the requirements for this ticket)
        • S4R packaging is much faster, in particular with large dependencies: there is no need to unzip/copy/zip all dependencies
        • simpler and more efficient classloading
        Show
        Matthieu Morel added a comment - I uploaded a patch in branch S4-72 . It replaces the "multiple classloaders" scheme with a mechanism inspired from Hadoop's classloading scheme. S4R are generated differently: dependencies are not exploded then gathered again. Instead, we place them in a /lib directory in the S4R, whereas the app jar is placed in /app. After retrieval, nodes unzip the S4R in a given directory and load class/resources from there. Benefits: user resources take precedence over dependencies resources: upon class/resource loading, the /app dir takes precedence over the /lib dir (which fulfills the requirements for this ticket) S4R packaging is much faster, in particular with large dependencies: there is no need to unzip/copy/zip all dependencies simpler and more efficient classloading
        Hide
        Matthieu Morel added a comment -

        Note that you'll need to modify existing build.gradle files for existing projects, by applying the following diff:

        +// external dependencies will be available in the /lib directory of the s4r
        +task copyDependenciesToLib(type: Copy) {
        +    into project.libsDir.path+"/lib"
        +    from configurations.runtime
        +}
        +
        +// app jar will be available from the /app directory of the s4r
        +task buildProjectJar() {
        +       dependsOn jar {
        +               destinationDir file(project.libsDir.path + "/app")
        +               from sourceSets.main.output
        +       }
        +}
        +
        +
         task s4r(type: Jar) {
        -
        -   if (rootProject.hasProperty("appName")) {
        +       if (rootProject.hasProperty("appName")) {
                archiveName= "$appName"+".s4r"
            }
        -   dependsOn jar
        -   from { appDependencies.collect { it.isDirectory() ? it : zipTree(it) } }
        -   from { configurations.archives.allArtifacts.files.collect { zipTree(it) } }
        +   dependsOn cleanCopyDependenciesToLib, copyDependenciesToLib, cleanBuildProjectJar, buildProjectJar
        +   from { project.libsDir }
            manifest = project.manifest
        +   extension = 's4r'
        +   exclude '*.s4r'
        
        Show
        Matthieu Morel added a comment - Note that you'll need to modify existing build.gradle files for existing projects, by applying the following diff: + // external dependencies will be available in the /lib directory of the s4r +task copyDependenciesToLib(type: Copy) { + into project.libsDir.path+ "/lib" + from configurations.runtime +} + + // app jar will be available from the /app directory of the s4r +task buildProjectJar() { + dependsOn jar { + destinationDir file(project.libsDir.path + "/app" ) + from sourceSets.main.output + } +} + + task s4r(type: Jar) { - - if (rootProject.hasProperty( "appName" )) { + if (rootProject.hasProperty( "appName" )) { archiveName= "$appName" + ".s4r" } - dependsOn jar - from { appDependencies.collect { it.isDirectory() ? it : zipTree(it) } } - from { configurations.archives.allArtifacts.files.collect { zipTree(it) } } + dependsOn cleanCopyDependenciesToLib, copyDependenciesToLib, cleanBuildProjectJar, buildProjectJar + from { project.libsDir } manifest = project.manifest + extension = 's4r' + exclude '*.s4r'
        Hide
        Matthieu Morel added a comment -

        I added a commit in S4-72 branch for making better use of optional s4.tmp.dir option for exploding S4R archives and loading resources/classes from there.

        Show
        Matthieu Morel added a comment - I added a commit in S4-72 branch for making better use of optional s4.tmp.dir option for exploding S4R archives and loading resources/classes from there.
        Hide
        Daniel Gómez Ferro added a comment -

        Really nice job. Resource loading works perfectly now and the deployment time has decreased a lot, from up to 2 minutes down to 20 seconds.

        Show
        Daniel Gómez Ferro added a comment - Really nice job. Resource loading works perfectly now and the deployment time has decreased a lot, from up to 2 minutes down to 20 seconds.
        Show
        Matthieu Morel added a comment - Merged in piper branch https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=commit;h=341f4bf97f1e7821c6f3df402acbd983f3b84b40

          People

          • Assignee:
            Matthieu Morel
            Reporter:
            Daniel Gómez Ferro
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development