Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5097

Job.addArchiveToClassPath is ignored when running job with LocalJobRunner

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: test
    • Labels:
      None

      Description

      Using external dependency jar in mr job. Adding it to the job classpath via Job.addArchiveToClassPath(...) doesn't work when running with LocalJobRunner (i.e. in unit test). This makes it harder to unit-test such jobs (with third-party runtime dependencies).

      1. MAPREDUCE-5097.patch
        1 kB
        Alex Baranau
      2. MAPREDUCE-5097-ugly-test.patch
        7 kB
        Alex Baranau

        Activity

        Hide
        Alex Baranau added a comment -

        Attached tiny patch (MAPREDUCE-5097.patch).

        Also tried to show the test case scenario that fails without the patch (MAPREDUCE-5097-ugly-test.patch). It is ugly: it actually just shows that when job classes are not in classpath then adding the jar using addArchiveToClassPath has no effect. The test should be better. I will try to figure one out when I have more time.. if this is really needed: the fix is pretty simple (if I got it right).

        Show
        Alex Baranau added a comment - Attached tiny patch ( MAPREDUCE-5097 .patch). Also tried to show the test case scenario that fails without the patch ( MAPREDUCE-5097 -ugly-test.patch). It is ugly: it actually just shows that when job classes are not in classpath then adding the jar using addArchiveToClassPath has no effect. The test should be better. I will try to figure one out when I have more time.. if this is really needed: the fix is pretty simple (if I got it right).
        Hide
        Alex Baranau added a comment -

        oh, and to try test you need to build examples jar after applying patch, so that you have jar to test on. It is ugly, I know and is not meant for committing. Only the patch with the fix (unless I add good test).

        Show
        Alex Baranau added a comment - oh, and to try test you need to build examples jar after applying patch, so that you have jar to test on. It is ugly, I know and is not meant for committing. Only the patch with the fix (unless I add good test).
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12575117/MAPREDUCE-5097-ugly-test.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 one of tests included doesn't have a timeout.

        -1 javac. The applied patch generated 1363 javac compiler warnings (more than the trunk's current 1362 warnings).

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples:

        org.apache.hadoop.mapred.TestMiniMRClassPathWithExternalJar

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3461//testReport/
        Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3461//artifact/trunk/patchprocess/diffJavacWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3461//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575117/MAPREDUCE-5097-ugly-test.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. -1 javac . The applied patch generated 1363 javac compiler warnings (more than the trunk's current 1362 warnings). +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples: org.apache.hadoop.mapred.TestMiniMRClassPathWithExternalJar +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3461//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3461//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3461//console This message is automatically generated.

          People

          • Assignee:
            Alex Baranau
            Reporter:
            Alex Baranau
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development