Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None

      Description

      When running KVTest and LargeKVTest, if the job failed for some reason(lack libhadoop.so etc), both native and normal job failed, and both compare empty output directory, so the test passes without noticing failure.

      1. MAPREDUCE-6058.v3.patch
        22 kB
        Binglin Chang
      2. MAPREDUCE-6058.v2.patch
        22 kB
        Binglin Chang
      3. MAPREDUCE-6058.v1.patch
        4 kB
        Binglin Chang

        Activity

        Binglin Chang created issue -
        Binglin Chang made changes -
        Field Original Value New Value
        Component/s task [ 12312920 ]
        Binglin Chang made changes -
        Attachment MAPREDUCE-6058.v1.patch [ 12664961 ]
        Hide
        Todd Lipcon added a comment -

        Looks good. Should we add some kind of JUnit assumeTrue() that the native code is available, so that the test gets marked skipped if we are testing without -Pnative?

        Show
        Todd Lipcon added a comment - Looks good. Should we add some kind of JUnit assumeTrue() that the native code is available, so that the test gets marked skipped if we are testing without -Pnative?
        Hide
        Binglin Chang added a comment -

        If we are testing without -Pnative in pom.xml, those tests are skipped, I encounter the error mainly because I run test inside hadoop-mapreduce-client-nativetask(on mac mvn test -Pnative will fail), anyway I think it's good to add assumeTrue, will update the patch soon.

        Show
        Binglin Chang added a comment - If we are testing without -Pnative in pom.xml, those tests are skipped, I encounter the error mainly because I run test inside hadoop-mapreduce-client-nativetask(on mac mvn test -Pnative will fail), anyway I think it's good to add assumeTrue, will update the patch soon.
        Hide
        Binglin Chang added a comment -

        Changes:
        1. add assumeTrue to skip tests when native library is not available
        2. add more run job checks
        3. since test can be automatically skipped, remove test configs in pom.xml

        Show
        Binglin Chang added a comment - Changes: 1. add assumeTrue to skip tests when native library is not available 2. add more run job checks 3. since test can be automatically skipped, remove test configs in pom.xml
        Binglin Chang made changes -
        Attachment MAPREDUCE-6058.v2.patch [ 12665260 ]
        Hide
        Sean Zhong added a comment -

        looks good,

        Should we move the following lines to a startUp() function so that when adding new test function, we don't need to ad these two lines.
        @Test
        public void testKVCompability() {
        + Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
        + Assume.assumeTrue(NativeRuntime.isNativeLibraryLoaded());

        Show
        Sean Zhong added a comment - looks good, Should we move the following lines to a startUp() function so that when adding new test function, we don't need to ad these two lines. @Test public void testKVCompability() { + Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded()); + Assume.assumeTrue(NativeRuntime.isNativeLibraryLoaded());
        Hide
        Binglin Chang added a comment -

        Thanks for the review and comments Sean, update the patch addressing the comments.

        Show
        Binglin Chang added a comment - Thanks for the review and comments Sean, update the patch addressing the comments.
        Binglin Chang made changes -
        Attachment MAPREDUCE-6058.v3.patch [ 12665832 ]
        Hide
        Sean Zhong added a comment -

        +1

        Show
        Sean Zhong added a comment - +1
        Hide
        Binglin Chang added a comment -

        I have committed this, thanks Sean and Todd.

        Show
        Binglin Chang added a comment - I have committed this, thanks Sean and Todd.
        Binglin Chang made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Todd Lipcon added a comment -

        Hey Binglin Chang and [~seanzhong] – after this patch, it seems like the tests are always getting skipped for me. Is something wrong about the assumeTrue calls?

        Show
        Todd Lipcon added a comment - Hey Binglin Chang and [~seanzhong] – after this patch, it seems like the tests are always getting skipped for me. Is something wrong about the assumeTrue calls?
        Todd Lipcon made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Todd Lipcon added a comment -

        Oh, I see the problem. I hadn't built the hadoop-common native code (libhadoop.so). Is there a reason that we require libhadoop.so to run the nativetask tests? Seems that the two .sos should be independent, right?

        Show
        Todd Lipcon added a comment - Oh, I see the problem. I hadn't built the hadoop-common native code (libhadoop.so). Is there a reason that we require libhadoop.so to run the nativetask tests? Seems that the two .sos should be independent, right?
        Hide
        Manu Zhang added a comment -

        I think that's required to load in codecs for the original job to run ?

        Show
        Manu Zhang added a comment - I think that's required to load in codecs for the original job to run ?
        Hide
        Sean Zhong added a comment -

        Agree with Manu

        Show
        Sean Zhong added a comment - Agree with Manu
        Hide
        Binglin Chang added a comment -

        Yes, the test requires compression codec to run, which requires libhadoop.so. If the whole project is not compiled with Pnative, I thinks those tests should be skipped.

        Show
        Binglin Chang added a comment - Yes, the test requires compression codec to run, which requires libhadoop.so. If the whole project is not compiled with Pnative, I thinks those tests should be skipped.
        Hide
        Todd Lipcon added a comment -

        ah, I see you're right. Thanks and sorry for the noise

        Show
        Todd Lipcon added a comment - ah, I see you're right. Thanks and sorry for the noise
        Todd Lipcon made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
        MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/ ) MAPREDUCE-6058 . native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
        MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/ ) MAPREDUCE-6058 . native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
        MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java
        • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/ ) MAPREDUCE-6058 . native-task: KVTest and LargeKVTest should check mr job is sucessful (Contributed by Binglin Chang) (bchang: rev 17cd0faaceb8f9ce00b8c2c200e810f78f36c671) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/OldAPICombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/LargeKVCombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/compresstest/CompressTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/combinertest/CombinerTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/LargeKVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/kvtest/KVTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/org/apache/hadoop/mapred/nativetask/nonsorttest/NonSortTest.java hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        4d 15h 8m 1 Binglin Chang 02/Sep/14 04:04
        Resolved Resolved Reopened Reopened
        1d 16h 4m 1 Todd Lipcon 03/Sep/14 20:08
        Reopened Reopened Resolved Resolved
        1d 9h 16m 1 Todd Lipcon 05/Sep/14 05:24

          People

          • Assignee:
            Binglin Chang
            Reporter:
            Binglin Chang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development