Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None

      Description

      When using hadoop inside an OSGI environment one needs to change the META-INF/MANIFEST.MF file to include OSGI headers (version and symbolic name). It would be convenient to do this in the default build.xml.

      There are no runtime dependencies.

      An easy way of doing this is to use the bnd ant task: http://www.aqute.biz/Code/Bnd

      <target name="build">
      <taskdef resource="aQute/bnd/ant/taskdef.properties"
      classpath="bnd.jar"/>
      <bnd
      classpath="src"
      eclipse="true"
      failok="false"
      exceptions="true"
      files="test.bnd"/>
      </target>

      1. HADOOP-6484.patch
        19 kB
        Jean-Baptiste Onofré

        Issue Links

          Activity

          Leen Toelen created issue -
          Hide
          Leen Toelen added a comment -

          ivy.xml:
          <dependency org="biz.aQute" name="bnd" rev="0.0.169" conf="common->default"></dependency>

          conf/hadoop-core.bnd.template:
          Bundle-Version:$

          {version}

          Bundle-SymbolicName:org.apache.hadoop.common

          Added to the jar section in build.xml:
          <taskdef name="bndwrap" classname="aQute.bnd.ant.WrapTask">
          <classpath refid="classpath" />
          </taskdef>

          <copy file="$

          {conf.dir}

          /hadoop-core.bnd.template" tofile="$

          {build.dir}/${final.name}.bnd" overwrite="true"/>

          <bndwrap output="${build.dir}

          " definitions="$

          {build.dir}">
          <fileset dir="${build.dir}

          " includes="*.jar"/>
          </bndwrap>
          <move file="$

          {build.dir}/${final.name}.bar" tofile="${build.dir}

          /$

          {final.name}

          .jar" overwrite="true"/>

          This generates an OSGI compatible manifest

          Show
          Leen Toelen added a comment - ivy.xml: <dependency org="biz.aQute" name="bnd" rev="0.0.169" conf="common->default"></dependency> conf/hadoop-core.bnd.template: Bundle-Version:$ {version} Bundle-SymbolicName:org.apache.hadoop.common Added to the jar section in build.xml: <taskdef name="bndwrap" classname="aQute.bnd.ant.WrapTask"> <classpath refid="classpath" /> </taskdef> <copy file="$ {conf.dir} /hadoop-core.bnd.template" tofile="$ {build.dir}/${final.name}.bnd" overwrite="true"/> <bndwrap output="${build.dir} " definitions="$ {build.dir}"> <fileset dir="${build.dir} " includes="*.jar"/> </bndwrap> <move file="$ {build.dir}/${final.name}.bar" tofile="${build.dir} /$ {final.name} .jar" overwrite="true"/> This generates an OSGI compatible manifest
          Hide
          Steve Loughran added a comment -

          This could be useful

          1. It needs a proper patch file
          2. that new dependency would need to be build time only, property driven, etc
          3. I wouldn't do a <copy>/<move> that way as it effectively ignores timestamps and would take place every build. The manifest operation should only take place if the bar is older than the source jar.
          4. Is there a way for a unit test to check the manifest? Some test case that looks in the classpath for a resource?
          Show
          Steve Loughran added a comment - This could be useful It needs a proper patch file that new dependency would need to be build time only, property driven, etc I wouldn't do a <copy>/<move> that way as it effectively ignores timestamps and would take place every build. The manifest operation should only take place if the bar is older than the source jar. Is there a way for a unit test to check the manifest? Some test case that looks in the classpath for a resource?
          Hide
          Steve Loughran added a comment -

          the mechanism for doing this will have changed with the move to maven -but that does not eliminate the need.

          Show
          Steve Loughran added a comment - the mechanism for doing this will have changed with the move to maven -but that does not eliminate the need.
          Steve Loughran made changes -
          Field Original Value New Value
          Link This issue blocks HADOOP-7977 [ HADOOP-7977 ]
          Hide
          Jean-Baptiste Onofré added a comment -

          Hi all,

          related to that I have a patch ready (I'm gonna submit it) to turn Hadoop artifacts as OSGi bundles and providing a Karaf feature for an easy installation in the Karaf OSGi Container.

          Show
          Jean-Baptiste Onofré added a comment - Hi all, related to that I have a patch ready (I'm gonna submit it) to turn Hadoop artifacts as OSGi bundles and providing a Karaf feature for an easy installation in the Karaf OSGi Container.
          Hide
          Jean-Baptiste Onofré added a comment -

          Is someone can assign this Jira to me ?

          Show
          Jean-Baptiste Onofré added a comment - Is someone can assign this Jira to me ?
          Hide
          Steve Loughran added a comment -
          1. The patch sounds good; Karaf + Hadoop good too.
          2. What is the test plan? I could imagine an existing in-junit test that looks for the manifest entries, while presumably there is some karaf+hadoop thing too.
          3. Be aware that from my own experience what Hadoop does with security is strongly incompatible with other java container security, which could cause problems, but you will need a security manager to catch System.exit() calls.
          4. Regarding the JIRA assignment, I tried by giving you developer rights on the project, and, when that didn't seem to do anything, put you in the all-developers group. I still can't seem to assign it to you -perhaps you could log out, log in and try to assign it to yourself. Otherwise, well, you are JIRA administrator so may stand a better chance of fixing this than I can.
          Show
          Steve Loughran added a comment - The patch sounds good; Karaf + Hadoop good too. What is the test plan? I could imagine an existing in-junit test that looks for the manifest entries, while presumably there is some karaf+hadoop thing too. Be aware that from my own experience what Hadoop does with security is strongly incompatible with other java container security, which could cause problems, but you will need a security manager to catch System.exit() calls. Regarding the JIRA assignment, I tried by giving you developer rights on the project, and, when that didn't seem to do anything, put you in the all-developers group. I still can't seem to assign it to you -perhaps you could log out, log in and try to assign it to yourself. Otherwise, well, you are JIRA administrator so may stand a better chance of fixing this than I can.
          Hide
          Jean-Baptiste Onofré added a comment -

          Hi Steve,

          1/ patch is on the way
          2/ for the tests, I plan to provide itests using Karaf pax-exam. Thanks to Karaf pax-exam, we are able to bootstrap a Karaf container, execute some commands in the Karaf shell console, and deploy some features (like the Hadoop one)
          3/ right, we have to setup a security manager in Karaf. I could enhance the Karaf security framework to handle that in an elegant way.
          4/ ok, let me try on my side, else it's not a big deal

          Show
          Jean-Baptiste Onofré added a comment - Hi Steve, 1/ patch is on the way 2/ for the tests, I plan to provide itests using Karaf pax-exam. Thanks to Karaf pax-exam, we are able to bootstrap a Karaf container, execute some commands in the Karaf shell console, and deploy some features (like the Hadoop one) 3/ right, we have to setup a security manager in Karaf. I could enhance the Karaf security framework to handle that in an elegant way. 4/ ok, let me try on my side, else it's not a big deal
          Hide
          Jean-Baptiste Onofré added a comment -

          First patch for the hadoop-common-project, and basement of the other following patches.

          It includes:

          • turn all Hadoop common modules as OSGi bundles
          • add a Karaf features ready to be deployed
          Show
          Jean-Baptiste Onofré added a comment - First patch for the hadoop-common-project, and basement of the other following patches. It includes: turn all Hadoop common modules as OSGi bundles add a Karaf features ready to be deployed
          Jean-Baptiste Onofré made changes -
          Attachment HADOOP-6484.patch [ 12515182 ]
          Hide
          Sanjay Radia added a comment -

          I am missing the context here. Part of this might be my ignorance about OSGI and Karaf
          Can you please create a umbrella jira that describes the benefits of using OSGI and Karaf and the scope of the work. Link all related jiras in this umbrella jira.

          • Benefits and target audience: Is this work targeted for managing/running hadoop for developers or for production use? Briefly describe the benefits.
          • Besides adding the manifests to jar files will it require adding more invasive changes such as special interfaces for stopping and starting hadoop daemons?
          • Will this be used for management after deployment has been done through some other mechanism or will this work also enable the deployment in a cluster.
          Show
          Sanjay Radia added a comment - I am missing the context here. Part of this might be my ignorance about OSGI and Karaf Can you please create a umbrella jira that describes the benefits of using OSGI and Karaf and the scope of the work. Link all related jiras in this umbrella jira. Benefits and target audience: Is this work targeted for managing/running hadoop for developers or for production use? Briefly describe the benefits. Besides adding the manifests to jar files will it require adding more invasive changes such as special interfaces for stopping and starting hadoop daemons? Will this be used for management after deployment has been done through some other mechanism or will this work also enable the deployment in a cluster.
          Hide
          Steve Loughran added a comment -

          Sanjay, HADOOP-7977 is the umbrella JIRA -answering your questions there

          Show
          Steve Loughran added a comment - Sanjay, HADOOP-7977 is the umbrella JIRA -answering your questions there
          Hide
          Steve Loughran added a comment -
          1. I don't know enough about M2 to review the POM files -what does the switch from a jar target to a bundle mean in terms of generation and output?
          2. what does resolution:=optional mean for dependencies. Particulary for things like commons-logging that are very much non-optional in the codebase?
          Show
          Steve Loughran added a comment - I don't know enough about M2 to review the POM files -what does the switch from a jar target to a bundle mean in terms of generation and output? what does resolution:=optional mean for dependencies. Particulary for things like commons-logging that are very much non-optional in the codebase?
          Kihwal Lee made changes -
          Link This issue relates to HADOOP-8446 [ HADOOP-8446 ]
          Gavin made changes -
          Link This issue blocks HADOOP-7977 [ HADOOP-7977 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-7977 [ HADOOP-7977 ]
          Steve Loughran made changes -
          Link This issue is duplicated by HADOOP-8446 [ HADOOP-8446 ]
          Allen Wittenauer made changes -
          Link This issue relates to HADOOP-8446 [ HADOOP-8446 ]
          Hide
          Allen Wittenauer added a comment -

          Ping!

          Show
          Allen Wittenauer added a comment - Ping!

            People

            • Assignee:
              Unassigned
              Reporter:
              Leen Toelen
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development