Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None

      Description

      When running KVTest and LargeKVTest, if the job failed for some reason(lack libhadoop.so etc), both native and normal job failed, and both compare empty output directory, so the test passes without noticing failure.

      1. MAPREDUCE-6058.v1.patch
        4 kB
        Binglin Chang
      2. MAPREDUCE-6058.v2.patch
        22 kB
        Binglin Chang
      3. MAPREDUCE-6058.v3.patch
        22 kB
        Binglin Chang

        Activity

        Hide
        Todd Lipcon added a comment -

        Looks good. Should we add some kind of JUnit assumeTrue() that the native code is available, so that the test gets marked skipped if we are testing without -Pnative?

        Show
        Todd Lipcon added a comment - Looks good. Should we add some kind of JUnit assumeTrue() that the native code is available, so that the test gets marked skipped if we are testing without -Pnative?
        Hide
        Binglin Chang added a comment -

        If we are testing without -Pnative in pom.xml, those tests are skipped, I encounter the error mainly because I run test inside hadoop-mapreduce-client-nativetask(on mac mvn test -Pnative will fail), anyway I think it's good to add assumeTrue, will update the patch soon.

        Show
        Binglin Chang added a comment - If we are testing without -Pnative in pom.xml, those tests are skipped, I encounter the error mainly because I run test inside hadoop-mapreduce-client-nativetask(on mac mvn test -Pnative will fail), anyway I think it's good to add assumeTrue, will update the patch soon.
        Hide
        Binglin Chang added a comment -

        Changes:
        1. add assumeTrue to skip tests when native library is not available
        2. add more run job checks
        3. since test can be automatically skipped, remove test configs in pom.xml

        Show
        Binglin Chang added a comment - Changes: 1. add assumeTrue to skip tests when native library is not available 2. add more run job checks 3. since test can be automatically skipped, remove test configs in pom.xml
        Hide
        Sean Zhong added a comment -

        looks good,

        Should we move the following lines to a startUp() function so that when adding new test function, we don't need to ad these two lines.
        @Test
        public void testKVCompability() {
        + Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
        + Assume.assumeTrue(NativeRuntime.isNativeLibraryLoaded());

        Show
        Sean Zhong added a comment - looks good, Should we move the following lines to a startUp() function so that when adding new test function, we don't need to ad these two lines. @Test public void testKVCompability() { + Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded()); + Assume.assumeTrue(NativeRuntime.isNativeLibraryLoaded());
        Hide
        Binglin Chang added a comment -

        Thanks for the review and comments Sean, update the patch addressing the comments.

        Show
        Binglin Chang added a comment - Thanks for the review and comments Sean, update the patch addressing the comments.
        Hide
        Sean Zhong added a comment -

        +1

        Show
        Sean Zhong added a comment - +1
        Hide
        Binglin Chang added a comment -

        I have committed this, thanks Sean and Todd.

        Show
        Binglin Chang added a comment - I have committed this, thanks Sean and Todd.
        Hide
        Todd Lipcon added a comment -

        Hey Binglin Chang and [~seanzhong] – after this patch, it seems like the tests are always getting skipped for me. Is something wrong about the assumeTrue calls?

        Show
        Todd Lipcon added a comment - Hey Binglin Chang and [~seanzhong] – after this patch, it seems like the tests are always getting skipped for me. Is something wrong about the assumeTrue calls?
        Hide
        Todd Lipcon added a comment -

        Oh, I see the problem. I hadn't built the hadoop-common native code (libhadoop.so). Is there a reason that we require libhadoop.so to run the nativetask tests? Seems that the two .sos should be independent, right?

        Show
        Todd Lipcon added a comment - Oh, I see the problem. I hadn't built the hadoop-common native code (libhadoop.so). Is there a reason that we require libhadoop.so to run the nativetask tests? Seems that the two .sos should be independent, right?
        Hide
        Manu Zhang added a comment -

        I think that's required to load in codecs for the original job to run ?

        Show
        Manu Zhang added a comment - I think that's required to load in codecs for the original job to run ?
        Hide
        Sean Zhong added a comment -

        Agree with Manu

        Show
        Sean Zhong added a comment - Agree with Manu
        Hide
        Binglin Chang added a comment -

        Yes, the test requires compression codec to run, which requires libhadoop.so. If the whole project is not compiled with Pnative, I thinks those tests should be skipped.

        Show
        Binglin Chang added a comment - Yes, the test requires compression codec to run, which requires libhadoop.so. If the whole project is not compiled with Pnative, I thinks those tests should be skipped.
        Hide
        Todd Lipcon added a comment -

        ah, I see you're right. Thanks and sorry for the noise

        Show
        Todd Lipcon added a comment - ah, I see you're right. Thanks and sorry for the noise
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
        MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/ ) MAPREDUCE-6058 . native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
        MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/ ) MAPREDUCE-6058 . native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
        MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/ ) MAPREDUCE-6058 . native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt

          People

          • Assignee:
            Binglin Chang
            Reporter:
            Binglin Chang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development