Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: tools
    • Labels:
      None

      Description

      To follow the discussion about OSGi, and in order to move forward, I propose the following hadoop-karaf bundle.

      1. HADOOP-9706.patch
        25 kB
        Jean-Baptiste Onofré
      2. Karaf-HDFS-client.pdf
        76 kB
        Jean-Baptiste Onofré

        Activity

        Hide
        jbonofre Jean-Baptiste Onofré added a comment -

        First article of a serie to explain what we can do with that:

        http://blog.nanthrax.net/2013/07/apache-hadoop-and-karaf-article-1-karaf-as-hdfs-client/

        Show
        jbonofre Jean-Baptiste Onofré added a comment - First article of a serie to explain what we can do with that: http://blog.nanthrax.net/2013/07/apache-hadoop-and-karaf-article-1-karaf-as-hdfs-client/
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12591173/HADOOP-9706.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The applied patch generated 1160 javac compiler warnings (more than the trunk's current 1153 warnings).

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-karaf.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2748//testReport/
        Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2748//artifact/trunk/patchprocess/diffJavacWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2748//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12591173/HADOOP-9706.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The applied patch generated 1160 javac compiler warnings (more than the trunk's current 1153 warnings). +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-karaf. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2748//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2748//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2748//console This message is automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -
        1. is there any reason why this couldn't fit under hadoop-tools?
        2. there's a lot of new version numberings for various libraries -i'd have expected these all to be picked up from the main hadoop-project/pom.xml unless there's some problem.
        1. how would we go about adding tests for this? Is there an OSGi container that can some up inside JUnit, host say a miniDFS cluster- and then let us talk to the filesystem? Tests inside the hadoop codebase are a key way to stop regressions.
        Show
        stevel@apache.org Steve Loughran added a comment - is there any reason why this couldn't fit under hadoop-tools ? there's a lot of new version numberings for various libraries -i'd have expected these all to be picked up from the main hadoop-project/pom.xml unless there's some problem. how would we go about adding tests for this? Is there an OSGi container that can some up inside JUnit, host say a miniDFS cluster- and then let us talk to the filesystem? Tests inside the hadoop codebase are a key way to stop regressions.
        Hide
        jbonofre Jean-Baptiste Onofré added a comment -

        Hi Steve,

        1. For the "client" part (the current patch), you are right, it could be in hadoop-tools.
        2. I defined the dependencies version because I didn't find the version of these artifacts in other hadoop-common modules. For instance:
        fgrep -r -i jzlib *
        I don't find any definition of the jzlib artifact. I can move the dependencyManagement section in hadoop-project, let me know.
        3. The test part is on my TODO list: I plan to use karaf-pax-exam/pax-exam to implement utest/itest. I plan to bootstrap a miniDFS cluster and a Karaf instance.

        Show
        jbonofre Jean-Baptiste Onofré added a comment - Hi Steve, 1. For the "client" part (the current patch), you are right, it could be in hadoop-tools. 2. I defined the dependencies version because I didn't find the version of these artifacts in other hadoop-common modules. For instance: fgrep -r -i jzlib * I don't find any definition of the jzlib artifact. I can move the dependencyManagement section in hadoop-project, let me know. 3. The test part is on my TODO list: I plan to use karaf-pax-exam/pax-exam to implement utest/itest. I plan to bootstrap a miniDFS cluster and a Karaf instance.
        Hide
        jbonofre Jean-Baptiste Onofré added a comment -

        PDF export of the blog post.

        Show
        jbonofre Jean-Baptiste Onofré added a comment - PDF export of the blog post.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12591397/Karaf-HDFS-client.pdf
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2753//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12591397/Karaf-HDFS-client.pdf against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2753//console This message is automatically generated.
        Hide
        thierry_c Thierry Caminel added a comment -

        Excellent initiative.
        I hability to deploy MR jobs in Karaf, export Hadoop services as OSGi services and thus inheriting the full power of OSGi might ease a lot the integration of Hadoop with ESB.
        Continue!

        Show
        thierry_c Thierry Caminel added a comment - Excellent initiative. I hability to deploy MR jobs in Karaf, export Hadoop services as OSGi services and thus inheriting the full power of OSGi might ease a lot the integration of Hadoop with ESB. Continue!
        Hide
        stevel@apache.org Steve Loughran added a comment -
        1. we need a patch for the hadoop-tools section
        2. all the versions need to go into the hadoop-project pom.xml. If there are new artifacts, they should still be declared there (including the actual <dependency> value). this gives us one central place for managing the artifacts, both version and exclusions.
        Show
        stevel@apache.org Steve Loughran added a comment - we need a patch for the hadoop-tools section all the versions need to go into the hadoop-project pom.xml. If there are new artifacts, they should still be declared there (including the actual <dependency> value). this gives us one central place for managing the artifacts, both version and exclusions.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12591397/Karaf-HDFS-client.pdf
        against trunk revision 14e2639.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4722//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12591397/Karaf-HDFS-client.pdf against trunk revision 14e2639. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4722//console This message is automatically generated.
        Hide
        aw Allen Wittenauer added a comment -

        Cancelling patch since it no longer applies.

        Show
        aw Allen Wittenauer added a comment - Cancelling patch since it no longer applies.

          People

          • Assignee:
            Unassigned
            Reporter:
            jbonofre Jean-Baptiste Onofré
          • Votes:
            1 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:

              Development