Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-6006.v3.patch
      6 kB
      Binglin Chang
    2. MAPREDUCE-6006.v2.patch
      6 kB
      Binglin Chang
    3. MAPREDUCE-6006.v1.patch
      4 kB
      Binglin Chang

      Activity

      Binglin Chang created issue -
      Binglin Chang made changes -
      Field Original Value New Value
      Component/s task [ 12312920 ]
      Hide
      Binglin Chang added a comment -

      Attach first version of the patch, changes:
      1. bugfix use hadoop-mapreduce-client-common normal jar rather than test-jar
      2. invoke nttest in maven native build
      3. remove copy test.sh in pom.xml, use cmake configure_file instead, which can pass libjvm.so path to test script.

      Note that the run mvn test -Pnative still fail(or OOM) cause some java tests are still failing, I skip all java tests by running:
      mvn test -Pnative -Dtest=ddd
      Manually tested on macos and ubuntu, native tests run successfully.

      Show
      Binglin Chang added a comment - Attach first version of the patch, changes: 1. bugfix use hadoop-mapreduce-client-common normal jar rather than test-jar 2. invoke nttest in maven native build 3. remove copy test.sh in pom.xml, use cmake configure_file instead, which can pass libjvm.so path to test script. Note that the run mvn test -Pnative still fail(or OOM) cause some java tests are still failing, I skip all java tests by running: mvn test -Pnative -Dtest=ddd Manually tested on macos and ubuntu, native tests run successfully.
      Binglin Chang made changes -
      Attachment MAPREDUCE-6006.v1.patch [ 12660119 ]
      Hide
      Manu Zhang added a comment -

      Hi Binglin, do we need to revert changes to hadoop-maprecuce-client-common and hadoop-project brought in by MAPREDUCE-5991 ?
      Meanwhile, which java tests are still failing, could you give a list or open a jira for that ?

      Show
      Manu Zhang added a comment - Hi Binglin, do we need to revert changes to hadoop-maprecuce-client-common and hadoop-project brought in by MAPREDUCE-5991 ? Meanwhile, which java tests are still failing, could you give a list or open a jira for that ?
      Hide
      Binglin Chang added a comment -

      Chages:
      Revert some changes done by MAPREDUCE-5991.

      To Manu Zhang: generally the tests(LargeKVTest, KVTest) takes too much memory and time to run, in my default env it produces OOM, (those tests are important, but we may need to make it smaller when run as unit test), and the test output are written to project sources directory rather than target directory, which pollutes project structure.

      Show
      Binglin Chang added a comment - Chages: Revert some changes done by MAPREDUCE-5991 . To Manu Zhang : generally the tests(LargeKVTest, KVTest) takes too much memory and time to run, in my default env it produces OOM, (those tests are important, but we may need to make it smaller when run as unit test), and the test output are written to project sources directory rather than target directory, which pollutes project structure.
      Binglin Chang made changes -
      Attachment MAPREDUCE-6006.v2.patch [ 12660314 ]
      Hide
      Binglin Chang added a comment -

      update the patch, forget to pass skipTests env to nttest.

      Show
      Binglin Chang added a comment - update the patch, forget to pass skipTests env to nttest.
      Binglin Chang made changes -
      Attachment MAPREDUCE-6006.v3.patch [ 12660317 ]
      Hide
      Todd Lipcon added a comment -

      +1, looks good to me. I'll commit this to the branch momentarily after giving it a try on my system.

      Show
      Todd Lipcon added a comment - +1, looks good to me. I'll commit this to the branch momentarily after giving it a try on my system.
      Hide
      Todd Lipcon added a comment -

      Worked great here. Committed to branch.

      Show
      Todd Lipcon added a comment - Worked great here. Committed to branch.
      Todd Lipcon made changes -
      Status Open [ 1 ] Resolved [ 5 ]
      Hadoop Flags Reviewed [ 10343 ]
      Resolution Fixed [ 1 ]
      Hide
      Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
      MAPREDUCE-6006. native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh
      • hadoop-project/pom.xml
      • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      Show
      Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/ ) MAPREDUCE-6006 . native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh hadoop-project/pom.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
      MAPREDUCE-6006. native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560)

      • hadoop-project/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh
      • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Show
      Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/ ) MAPREDUCE-6006 . native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560) hadoop-project/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Hide
      Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
      MAPREDUCE-6006. native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
      • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh
      • hadoop-project/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Show
      Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/ ) MAPREDUCE-6006 . native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh hadoop-project/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Transition Time In Source Status Execution Times Last Executer Last Execution Date
      Open Open Resolved Resolved
      19d 20h 37m 1 Todd Lipcon 14/Aug/14 05:47

        People

        • Assignee:
          Binglin Chang
          Reporter:
          Binglin Chang
        • Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development