Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9459

hadoop-hdfs-native-client fails test build on Windows after transition to ctest.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-9369 transitioned to usage of ctest for running the HDFS native tests. This broke the mvn test build on Windows.

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          There are multiple problems:

          1. The Ant XML for calling ctest in the native-win profile is invalid, because there are nested <target> declarations. This causes the mvn test to fail.
          2. On Windows, the native bits are built into target/native. The working directory for the ctest call was just target, so it couldn't find the test configuration file and actually run the tests.
          3. The environment settings appear to be copy-pasted from the *nix native profile, setting LD_LIBRARY_PATH. Instead, we need to set HADOOP_HOME to find winutils.exe and PATH to find hadoop.dll.
          Show
          cnauroth Chris Nauroth added a comment - There are multiple problems: The Ant XML for calling ctest in the native-win profile is invalid, because there are nested <target> declarations. This causes the mvn test to fail. On Windows, the native bits are built into target/native. The working directory for the ctest call was just target, so it couldn't find the test configuration file and actually run the tests. The environment settings appear to be copy-pasted from the *nix native profile, setting LD_LIBRARY_PATH . Instead, we need to set HADOOP_HOME to find winutils.exe and PATH to find hadoop.dll.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user cnauroth opened a pull request:

          https://github.com/apache/hadoop/pull/58

          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows afte…

          …r transition to ctest.

          I verified that the hadoop-hdfs-native-client tests build correctly and run successfully after applying this patch.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cnauroth/hadoop-1 HDFS-9459

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/58.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #58


          commit d583992956eb65fc4d1ba93c5c81da85c4592778
          Author: cnauroth <cnauroth@apache.org>
          Date: 2015-11-24T19:54:51Z

          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after transition to ctest.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user cnauroth opened a pull request: https://github.com/apache/hadoop/pull/58 HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows afte… …r transition to ctest. I verified that the hadoop-hdfs-native-client tests build correctly and run successfully after applying this patch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cnauroth/hadoop-1 HDFS-9459 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/58.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #58 commit d583992956eb65fc4d1ba93c5c81da85c4592778 Author: cnauroth <cnauroth@apache.org> Date: 2015-11-24T19:54:51Z HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after transition to ctest.
          Hide
          wheat9 Haohui Mai added a comment -

          +1 pending jenkins

          Show
          wheat9 Haohui Mai added a comment - +1 pending jenkins
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 10m 43s trunk passed
          +1 compile 0m 31s trunk passed with JDK v1.8.0_66
          +1 compile 0m 32s trunk passed with JDK v1.7.0_85
          +1 mvnsite 0m 18s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 javadoc 0m 14s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.8.0_66
          +1 javac 0m 31s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.7.0_85
          +1 javac 0m 31s the patch passed
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 14s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_85
          +1 unit 1m 0s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 56s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 31s Patch does not generate ASF License warnings.
          21m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Issue HDFS-9459
          GITHUB PR https://github.com/apache/hadoop/pull/58
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux e60d4b7ac5ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 177975e
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13659/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13659/console
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13659/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 43s trunk passed +1 compile 0m 31s trunk passed with JDK v1.8.0_66 +1 compile 0m 32s trunk passed with JDK v1.7.0_85 +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 16s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 16s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_66 +1 javac 0m 31s the patch passed +1 compile 0m 31s the patch passed with JDK v1.7.0_85 +1 javac 0m 31s the patch passed +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 14s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_85 +1 unit 1m 0s hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_66. +1 unit 0m 56s hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 21m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Issue HDFS-9459 GITHUB PR https://github.com/apache/hadoop/pull/58 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux e60d4b7ac5ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 177975e JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13659/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13659/console Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13659/console This message was automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          I've committed the patch to trunk and branch-2. Thanks Chris Nauroth for the contribution.

          Show
          wheat9 Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks Chris Nauroth for the contribution.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8891 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8891/)
          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8891 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8891/ ) HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2) hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          Haohui Mai, thanks for the commit.

          Show
          cnauroth Chris Nauroth added a comment - Haohui Mai , thanks for the commit.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1455 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1455/)
          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1455 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1455/ ) HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #734 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/734/)
          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #734 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/734/ ) HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2) hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #723 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/723/)
          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #723 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/723/ ) HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2666 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2666/)
          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2666 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2666/ ) HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2) hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #644 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/644/)
          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #644 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/644/ ) HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2) hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/)
          HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2)

          • hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/ ) HDFS-9459 . hadoop-hdfs-native-client fails test build on Windows after (wheat9: rev 95d5227c75f430da7c77847f31734b34b36157d2) hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aajisaka commented on the issue:

          https://github.com/apache/hadoop/pull/58

          Now HDFS-9459 is fixed.
          @cnauroth Would you close this pull request?

          Show
          githubbot ASF GitHub Bot added a comment - Github user aajisaka commented on the issue: https://github.com/apache/hadoop/pull/58 Now HDFS-9459 is fixed. @cnauroth Would you close this pull request?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cnauroth closed the pull request at:

          https://github.com/apache/hadoop/pull/58

          Show
          githubbot ASF GitHub Bot added a comment - Github user cnauroth closed the pull request at: https://github.com/apache/hadoop/pull/58

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development