Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6051

HDFS cannot run on Windows since short-circuit shared memory segment changes.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Patches recently committed in HDFS-5950 and HDFS-6040 for short-circuit shared memory segments cause the HDFS client to fail on Windows.

      1. HDFS-6051.003.patch
        4 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Hide
          Chris Nauroth added a comment -

          This stack trace shows what happen when you try to launch a MiniDFSCluster on Windows.

          Running org.apache.hadoop.cli.TestAclCLI
          Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.156 sec <<< FAILURE! - in org.apache.hadoop.cli.TestAclCLI
          testAll(org.apache.hadoop.cli.TestAclCLI)  Time elapsed: 2.922 sec  <<< ERROR!
          java.lang.IllegalArgumentException: null
                  at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
                  at org.apache.hadoop.io.nativeio.SharedFileDescriptorFactory.<init>(SharedFileDescriptorFactory.java:60)
                  at org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.<init>(ShortCircuitRegistry.java:171)
                  at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:583)
                  at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:771)
                  at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:316)
                  at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1868)
                  at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1768)
                  at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1282)
                  at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:718)
                  at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
                  at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
                  at org.apache.hadoop.cli.TestAclCLI.setUp(TestAclCLI.java:40)
          
          Show
          Chris Nauroth added a comment - This stack trace shows what happen when you try to launch a MiniDFSCluster on Windows. Running org.apache.hadoop.cli.TestAclCLI Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.156 sec <<< FAILURE! - in org.apache.hadoop.cli.TestAclCLI testAll(org.apache.hadoop.cli.TestAclCLI) Time elapsed: 2.922 sec <<< ERROR! java.lang.IllegalArgumentException: null at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72) at org.apache.hadoop.io.nativeio.SharedFileDescriptorFactory.<init>(SharedFileDescriptorFactory.java:60) at org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.<init>(ShortCircuitRegistry.java:171) at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:583) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:771) at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:316) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1868) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1768) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1282) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:718) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355) at org.apache.hadoop.cli.TestAclCLI.setUp(TestAclCLI.java:40)
          Hide
          Chris Nauroth added a comment -

          Thanks for taking this Colin. I'm reassigning the issue to you. I had a similar patch in testing. I'll take yours for a test run in my Windows environment now.

          Show
          Chris Nauroth added a comment - Thanks for taking this Colin. I'm reassigning the issue to you. I had a similar patch in testing. I'll take yours for a test run in my Windows environment now.
          Hide
          Chris Nauroth added a comment -

          Oops, compilation error in ShortCircuitRegistry, because there is no getLoadingFailureReason. I assume you wanted to call the method in SharedFileDescriptorFactory, so here is patch v2 with that minor change. This is what I'm testing in my Windows environment now.

          Show
          Chris Nauroth added a comment - Oops, compilation error in ShortCircuitRegistry , because there is no getLoadingFailureReason . I assume you wanted to call the method in SharedFileDescriptorFactory , so here is patch v2 with that minor change. This is what I'm testing in my Windows environment now.
          Hide
          Colin Patrick McCabe added a comment -

          try this patch out

          Show
          Colin Patrick McCabe added a comment - try this patch out
          Hide
          Colin Patrick McCabe added a comment -

          re-uploading correct version

          Show
          Colin Patrick McCabe added a comment - re-uploading correct version
          Hide
          Chris Nauroth added a comment -

          +1 for the patch. I ran a subset of the HDFS tests on Windows, and it looks good now. Thanks again, Colin.

          Show
          Chris Nauroth added a comment - +1 for the patch. I ran a subset of the HDFS tests on Windows, and it looks good now. Thanks again, Colin.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12632581/HDFS-6051.003.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6299//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6299//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632581/HDFS-6051.003.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6299//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6299//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.

          This is acceptable, because we've tested the patch independently on Windows.

          Show
          Chris Nauroth added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. This is acceptable, because we've tested the patch independently on Windows.
          Hide
          Chris Nauroth added a comment -

          I'm going to commit this in the next few minutes.

          Show
          Chris Nauroth added a comment - I'm going to commit this in the next few minutes.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5262 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5262/)
          HDFS-6051. HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5262 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5262/ ) HDFS-6051 . HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Hide
          Colin Patrick McCabe added a comment -

          committed; thanks for the reviews & testing

          Show
          Colin Patrick McCabe added a comment - committed; thanks for the reviews & testing
          Hide
          Chris Nauroth added a comment -

          Cool, you beat me to it. Thanks again!

          Show
          Chris Nauroth added a comment - Cool, you beat me to it. Thanks again!
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #500 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/500/)
          HDFS-6051. HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #500 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/500/ ) HDFS-6051 . HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1692/)
          HDFS-6051. HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1692/ ) HDFS-6051 . HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1717 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1717/)
          HDFS-6051. HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1717 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1717/ ) HDFS-6051 . HDFS cannot run on Windows since short-circuit memory segment changes (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574246 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development