HBase
  1. HBase
  2. HBASE-9003

TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 0.99.2
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is the problem: TableMapReduceUtil#addDependencyJars relies on org.apache.hadoop.util.JarFinder if available to call getJar(). However getJar() uses File.createTempFile() to create a temporary file under hadoop.tmp.dir/target/test-dir. Due HADOOP-9737 the created jar and its content is not purged after the JVM is destroyed. Since most configurations point hadoop.tmp.dir under /tmp the generated jar files get purged by tmpwatch or a similar tool, but boxes that have hadoop.tmp.dir pointing to a different location not monitored by tmpwatch will pile up a collection of jars causing all kind of issues. Since JarFinder#getJar is not a public API from Hadoop (see Alejandro Abdelnur comment on HADOOP-9737) we shouldn't use that as part of TableMapReduceUtil in order to avoid this kind of issues.

      1. HBASE-9003.v3.patch
        7 kB
        Nick Dimiduk
      2. HBASE-9003.v2.patch
        26 kB
        stack
      3. HBASE-9003.v2.patch
        26 kB
        Esteban Gutierrez
      4. HBASE-9003.v1.patch
        25 kB
        Esteban Gutierrez
      5. HBASE-9003.v0.patch
        2 kB
        Esteban Gutierrez

        Activity

        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.2 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.2 release.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-1.0 #398 (See https://builds.apache.org/job/HBase-1.0/398/)
        HBASE-9003 TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar (Esteban Gutierrez) (stack: rev b2d1b154bb299a68565c1fb1e692cf8b0b3ca101)

        • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestJarFinder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/JarFinder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/JarFinder.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestJarFinder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-1.0 #398 (See https://builds.apache.org/job/HBase-1.0/398/ ) HBASE-9003 TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar (Esteban Gutierrez) (stack: rev b2d1b154bb299a68565c1fb1e692cf8b0b3ca101) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestJarFinder.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/JarFinder.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/JarFinder.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestJarFinder.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #5728 (See https://builds.apache.org/job/HBase-TRUNK/5728/)
        HBASE-9003 TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar (Esteban Gutierrez) (stack: rev a71dd16a70e9a594e6aa24f69eb7b1d279273ca8)

        • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestJarFinder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/JarFinder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/JarFinder.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestJarFinder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5728 (See https://builds.apache.org/job/HBase-TRUNK/5728/ ) HBASE-9003 TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar (Esteban Gutierrez) (stack: rev a71dd16a70e9a594e6aa24f69eb7b1d279273ca8) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TestJarFinder.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/JarFinder.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/JarFinder.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestJarFinder.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        Hide
        stack added a comment -

        Committed to branch-1+ Nice team effort there lads. Thanks Esteban Gutierrez

        Show
        stack added a comment - Committed to branch-1+ Nice team effort there lads. Thanks Esteban Gutierrez
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12678379/HBASE-9003.v3.patch
        against trunk revision .
        ATTACHMENT ID: 12678379

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678379/HBASE-9003.v3.patch against trunk revision . ATTACHMENT ID: 12678379 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11532//console This message is automatically generated.
        Hide
        Esteban Gutierrez added a comment -

        That works stack. And yes, the tempJar.deleteOnExit() is mainly what we need to clean up the temp files. Thanks Nick Dimiduk!

        Show
        Esteban Gutierrez added a comment - That works stack . And yes, the tempJar.deleteOnExit() is mainly what we need to clean up the temp files. Thanks Nick Dimiduk !
        Hide
        stack added a comment -

        Good by you Esteban Gutierrez?

        Show
        stack added a comment - Good by you Esteban Gutierrez ?
        Hide
        Nick Dimiduk added a comment -

        The patch doesn't apply cleanly for me and the renames are masking whatever content changes were made. I took a shot at cleaning it up, and generated it with git diff -M90% so that the renames wouldn't clutter the patch. I also cleaned up some whitespace changes I found in the original patch and removed a log debug line. This is what I ended up with. Is it inline with what you intended?

        I'm +1 for this one.

        Show
        Nick Dimiduk added a comment - The patch doesn't apply cleanly for me and the renames are masking whatever content changes were made. I took a shot at cleaning it up, and generated it with git diff -M90% so that the renames wouldn't clutter the patch. I also cleaned up some whitespace changes I found in the original patch and removed a log debug line. This is what I ended up with. Is it inline with what you intended? I'm +1 for this one.
        Hide
        stack added a comment -

        Back to the point at hand +1 for fixing the accumulation problem.

        That is +1 on patch Nick Dimiduk?

        Show
        stack added a comment - Back to the point at hand +1 for fixing the accumulation problem. That is +1 on patch Nick Dimiduk ?
        Hide
        Nick Dimiduk added a comment -

        IIRC, I introduced JarFinder for the purpose of launching jobs from the output committer of a running job. In this context, the dependency jars have been unpacked, so to launch the job, JarFinder is used to re-pack the class files into a jar.

        Which raises an interesting point: you're seeing this accumulation of files under hadoop.tmp.dir even for regular jobs? Should be that nothing is created unless the requested class is found to exist on the class path outside of a jar.

        I don't remember the details; let me look into the code when I have a few minutes.

        Back to the point at hand +1 for fixing the accumulation problem.

        Show
        Nick Dimiduk added a comment - IIRC, I introduced JarFinder for the purpose of launching jobs from the output committer of a running job. In this context, the dependency jars have been unpacked, so to launch the job, JarFinder is used to re-pack the class files into a jar. Which raises an interesting point: you're seeing this accumulation of files under hadoop.tmp.dir even for regular jobs? Should be that nothing is created unless the requested class is found to exist on the class path outside of a jar. I don't remember the details; let me look into the code when I have a few minutes. Back to the point at hand +1 for fixing the accumulation problem.
        Hide
        Esteban Gutierrez added a comment -

        I don't think we can get rid of JarFinder as long as we have an option to use the distributed cache in initTable* I remember we used to ship the jars in similar way back in 0.90.x but we cleaned up the temporary jar. Here the only problem we have is that we don't clean up.

        Show
        Esteban Gutierrez added a comment - I don't think we can get rid of JarFinder as long as we have an option to use the distributed cache in initTable* I remember we used to ship the jars in similar way back in 0.90.x but we cleaned up the temporary jar. Here the only problem we have is that we don't clean up.
        Hide
        Nick Dimiduk added a comment -

        Makes sense. Let's get your fix in. What do you think about removing JarFinder all together for 1.0?

        Show
        Nick Dimiduk added a comment - Makes sense. Let's get your fix in. What do you think about removing JarFinder all together for 1.0?
        Hide
        Esteban Gutierrez added a comment -

        The files end up building even in non test environments when hadoop.tmp.dir points to a different location and that should be affecting any user that uses initTableMapperJob or initTableReducerJob and submits jobs using the distributed cache.

        Show
        Esteban Gutierrez added a comment - The files end up building even in non test environments when hadoop.tmp.dir points to a different location and that should be affecting any user that uses initTableMapperJob or initTableReducerJob and submits jobs using the distributed cache.
        Hide
        Nick Dimiduk added a comment -

        In HBASE-12008 I removed the test from IntegrationTestImportTsv. The HCat feature our users wanted supported has since been deprecated from HCat. The feature never really worked on that side anyway. The test was removed because it was failing with security-enabled installs due to flummoxed credential passing. Since the one user of the feature was gone, I pulled it out. I should have included this whole code path as well.

        Esteban Gutierrez are you seeing folks using this feature outside of the HCat scenario? Or are the files building up in a test env?

        Show
        Nick Dimiduk added a comment - In HBASE-12008 I removed the test from IntegrationTestImportTsv. The HCat feature our users wanted supported has since been deprecated from HCat. The feature never really worked on that side anyway. The test was removed because it was failing with security-enabled installs due to flummoxed credential passing. Since the one user of the feature was gone, I pulled it out. I should have included this whole code path as well. Esteban Gutierrez are you seeing folks using this feature outside of the HCat scenario? Or are the files building up in a test env?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12678245/HBASE-9003.v2.patch
        against trunk revision .
        ATTACHMENT ID: 12678245

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 10 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11522//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678245/HBASE-9003.v2.patch against trunk revision . ATTACHMENT ID: 12678245 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11522//console This message is automatically generated.
        Hide
        stack added a comment -

        Try to see if needs rebasing.

        Show
        stack added a comment - Try to see if needs rebasing.
        Hide
        stack added a comment -

        Nick Dimiduk How does the IT test relate? OK I commit this?

        Show
        stack added a comment - Nick Dimiduk How does the IT test relate? OK I commit this?
        Hide
        Nick Dimiduk added a comment -

        I found the integration test in question: IntegrationTestImportTsv#runFromOutputCommitter().

        Show
        Nick Dimiduk added a comment - I found the integration test in question: IntegrationTestImportTsv#runFromOutputCommitter().
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12642563/HBASE-9003.v2.patch
        against trunk revision .
        ATTACHMENT ID: 12642563

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 10 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10785//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642563/HBASE-9003.v2.patch against trunk revision . ATTACHMENT ID: 12642563 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10785//console This message is automatically generated.
        Hide
        Enis Soztutar added a comment -

        Nick Dimiduk, Stack do you guys want to commit this?

        Show
        Enis Soztutar added a comment - Nick Dimiduk , Stack do you guys want to commit this?
        Hide
        Esteban Gutierrez added a comment -

        New patch with Nick Dimiduk suggestions.

        Show
        Esteban Gutierrez added a comment - New patch with Nick Dimiduk suggestions.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12642412/HBASE-9003.v1.patch
        against trunk revision .
        ATTACHMENT ID: 12642412

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 10 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642412/HBASE-9003.v1.patch against trunk revision . ATTACHMENT ID: 12642412 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified tests. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9423//console This message is automatically generated.
        Hide
        Nick Dimiduk added a comment -

        Looks like a good fix to me. How about we keep the name JarFinder and move it to o.a.h.h.mapreduce package? Lineage is maintained via the class comment.

        I thought we had an integration test to cover the jar-creation code path but IntegrationTestTableMapReduceUtil doesn't fit the bill.

        Show
        Nick Dimiduk added a comment - Looks like a good fix to me. How about we keep the name JarFinder and move it to o.a.h.h.mapreduce package? Lineage is maintained via the class comment. I thought we had an integration test to cover the jar-creation code path but IntegrationTestTableMapReduceUtil doesn't fit the bill.
        Hide
        Esteban Gutierrez added a comment -

        New version renames JarFinder to HBaseJarFinder I don't think supporting the old class should be required for now, but if anybody thinks we need to support it for deprecation please let me know.

        Show
        Esteban Gutierrez added a comment - New version renames JarFinder to HBaseJarFinder I don't think supporting the old class should be required for now, but if anybody thinks we need to support it for deprecation please let me know.
        Hide
        stack added a comment -

        Yes to renaming.

        Show
        stack added a comment - Yes to renaming.
        Hide
        Esteban Gutierrez added a comment -

        I think there is conflict with this approach when hadoop-common-tests.jar is in the classpath. One approach might be to rename org.apache.hadoop.hbase.mapreduce.hadoopbackport.JarFinder to something like org.apache.hadoop.hbase.mapreduce.hadoopbackport.HBaseJarFinder or we could only change:

        String hadoopJarFinder = "org.apache.hadoop.util.JarFinder";
        

        to

        String hadoopJarFinder = "org.apache.hadoop.hbase.mapreduce.hadoopbackport.JarFinder";
        

        I think renaming is the best approach. Any thoughts?

        Show
        Esteban Gutierrez added a comment - I think there is conflict with this approach when hadoop-common-tests.jar is in the classpath. One approach might be to rename org.apache.hadoop.hbase.mapreduce.hadoopbackport.JarFinder to something like org.apache.hadoop.hbase.mapreduce.hadoopbackport.HBaseJarFinder or we could only change: String hadoopJarFinder = "org.apache.hadoop.util.JarFinder" ; to String hadoopJarFinder = "org.apache.hadoop.hbase.mapreduce.hadoopbackport.JarFinder" ; I think renaming is the best approach. Any thoughts?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12642385/HBASE-9003.v0.patch
        against trunk revision .
        ATTACHMENT ID: 12642385

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The patch appears to cause mvn compile goal to fail.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9417//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9417//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642385/HBASE-9003.v0.patch against trunk revision . ATTACHMENT ID: 12642385 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc . The javadoc tool did not generate any warning messages. -1 javac . The patch appears to cause mvn compile goal to fail. -1 findbugs . The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9417//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9417//console This message is automatically generated.
        Hide
        stack added a comment -

        lgtm Nick Dimiduk you good w/ this?

        Show
        stack added a comment - lgtm Nick Dimiduk you good w/ this?
        Hide
        Esteban Gutierrez added a comment -

        Attaching patch that defaults now to the backported JarFinder and sets tempJar.deleteOnExit() too.

        Show
        Esteban Gutierrez added a comment - Attaching patch that defaults now to the backported JarFinder and sets tempJar.deleteOnExit() too.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4672 (See https://builds.apache.org/job/HBase-TRUNK/4672/)
        HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599)

        • /hbase/trunk/dev-support/findbugs-exclude.xml
        • /hbase/trunk/dev-support/test-patch.properties
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4672 (See https://builds.apache.org/job/HBase-TRUNK/4672/ ) HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599) /hbase/trunk/dev-support/findbugs-exclude.xml /hbase/trunk/dev-support/test-patch.properties
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #830 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/830/)
        HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599)

        • /hbase/trunk/dev-support/findbugs-exclude.xml
        • /hbase/trunk/dev-support/test-patch.properties
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #830 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/830/ ) HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599) /hbase/trunk/dev-support/findbugs-exclude.xml /hbase/trunk/dev-support/test-patch.properties
        Hide
        Nick Dimiduk added a comment -

        I'm not aware of any other projects making use of it. The feature has only been available since 0.94.6 and it's a relatively esoteric feature of job submission. Maybe send a note to the user list asking if anyone depends on it? If we're removing support, better sooner than later.

        On the other hand, Hadoop not cleaning up after itself is a Hadoop bug, not HBase. HBase isn't the only project interested in seeing JarFinder promoted to a public API.

        Show
        Nick Dimiduk added a comment - I'm not aware of any other projects making use of it. The feature has only been available since 0.94.6 and it's a relatively esoteric feature of job submission. Maybe send a note to the user list asking if anyone depends on it? If we're removing support, better sooner than later. On the other hand, Hadoop not cleaning up after itself is a Hadoop bug, not HBase. HBase isn't the only project interested in seeing JarFinder promoted to a public API.
        Hide
        Esteban Gutierrez added a comment -

        Nick Dimiduk besides HCatalog is there any other consumer of JarFinder by the ecosystem? So far I researched is the only one around and for HBase I think it causes unnecessary headaches since nobody expects having 10s of GBs of jars under hadoop.tmp.dir after few days of submitting MR jobs using TableMapReduceUtil

        Show
        Esteban Gutierrez added a comment - Nick Dimiduk besides HCatalog is there any other consumer of JarFinder by the ecosystem? So far I researched is the only one around and for HBase I think it causes unnecessary headaches since nobody expects having 10s of GBs of jars under hadoop.tmp.dir after few days of submitting MR jobs using TableMapReduceUtil
        Hide
        Nick Dimiduk added a comment -

        HADOOP-9426 is intended to expose JarFinder as public API. It was included in addDependencyJars in order to facilitate launching MapReduce jobs from within other MapReduce jobs. This was consumed as part of a now deprecated HCatalog API.

        Show
        Nick Dimiduk added a comment - HADOOP-9426 is intended to expose JarFinder as public API. It was included in addDependencyJars in order to facilitate launching MapReduce jobs from within other MapReduce jobs. This was consumed as part of a now deprecated HCatalog API.

          People

          • Assignee:
            Esteban Gutierrez
            Reporter:
            Esteban Gutierrez
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development