Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-6006.v3.patch
      6 kB
      Binglin Chang
    2. MAPREDUCE-6006.v2.patch
      6 kB
      Binglin Chang
    3. MAPREDUCE-6006.v1.patch
      4 kB
      Binglin Chang

      Activity

      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/)
      MAPREDUCE-6006. native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
      • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh
      • hadoop-project/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/ ) MAPREDUCE-6006 . native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh hadoop-project/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/)
      MAPREDUCE-6006. native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560)

      • hadoop-project/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh
      • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/ ) MAPREDUCE-6006 . native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560) hadoop-project/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/)
      MAPREDUCE-6006. native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560)

      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh
      • hadoop-project/pom.xml
      • hadoop-mapreduce-project/CHANGES.MAPREDUCE-2841.txt
      • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/679/ ) MAPREDUCE-6006 . native-task: add native tests to maven and fix bug in pom.xml. Contributed by Binglin Chang. (todd: rev 886338ff80438b2daea3755f165cc411e39ef560) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test.sh hadoop-project/pom.xml hadoop-mapreduce-project/CHANGES. MAPREDUCE-2841 .txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt
      Hide
      tlipcon Todd Lipcon added a comment -

      Worked great here. Committed to branch.

      Show
      tlipcon Todd Lipcon added a comment - Worked great here. Committed to branch.
      Hide
      tlipcon Todd Lipcon added a comment -

      +1, looks good to me. I'll commit this to the branch momentarily after giving it a try on my system.

      Show
      tlipcon Todd Lipcon added a comment - +1, looks good to me. I'll commit this to the branch momentarily after giving it a try on my system.
      Hide
      decster Binglin Chang added a comment -

      update the patch, forget to pass skipTests env to nttest.

      Show
      decster Binglin Chang added a comment - update the patch, forget to pass skipTests env to nttest.
      Hide
      decster Binglin Chang added a comment -

      Chages:
      Revert some changes done by MAPREDUCE-5991.

      To Manu Zhang: generally the tests(LargeKVTest, KVTest) takes too much memory and time to run, in my default env it produces OOM, (those tests are important, but we may need to make it smaller when run as unit test), and the test output are written to project sources directory rather than target directory, which pollutes project structure.

      Show
      decster Binglin Chang added a comment - Chages: Revert some changes done by MAPREDUCE-5991 . To Manu Zhang : generally the tests(LargeKVTest, KVTest) takes too much memory and time to run, in my default env it produces OOM, (those tests are important, but we may need to make it smaller when run as unit test), and the test output are written to project sources directory rather than target directory, which pollutes project structure.
      Hide
      mauzhang Manu Zhang added a comment -

      Hi Binglin, do we need to revert changes to hadoop-maprecuce-client-common and hadoop-project brought in by MAPREDUCE-5991 ?
      Meanwhile, which java tests are still failing, could you give a list or open a jira for that ?

      Show
      mauzhang Manu Zhang added a comment - Hi Binglin, do we need to revert changes to hadoop-maprecuce-client-common and hadoop-project brought in by MAPREDUCE-5991 ? Meanwhile, which java tests are still failing, could you give a list or open a jira for that ?
      Hide
      decster Binglin Chang added a comment -

      Attach first version of the patch, changes:
      1. bugfix use hadoop-mapreduce-client-common normal jar rather than test-jar
      2. invoke nttest in maven native build
      3. remove copy test.sh in pom.xml, use cmake configure_file instead, which can pass libjvm.so path to test script.

      Note that the run mvn test -Pnative still fail(or OOM) cause some java tests are still failing, I skip all java tests by running:
      mvn test -Pnative -Dtest=ddd
      Manually tested on macos and ubuntu, native tests run successfully.

      Show
      decster Binglin Chang added a comment - Attach first version of the patch, changes: 1. bugfix use hadoop-mapreduce-client-common normal jar rather than test-jar 2. invoke nttest in maven native build 3. remove copy test.sh in pom.xml, use cmake configure_file instead, which can pass libjvm.so path to test script. Note that the run mvn test -Pnative still fail(or OOM) cause some java tests are still failing, I skip all java tests by running: mvn test -Pnative -Dtest=ddd Manually tested on macos and ubuntu, native tests run successfully.

        People

        • Assignee:
          decster Binglin Chang
          Reporter:
          decster Binglin Chang
        • Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development