Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13586

Fsync fails on directories on Windows

    Details

    • Hadoop Flags:
      Reviewed

      Description

      HDFS-11915 added a fsync call on DataNode's rbw directory on the first hsync() call. IOUtils.fsync first tries to get a FileChannel on the directory using FileChannel.open(READ). This call fails on Windows for any directory and throws an AccessDeniedException, see discussion here: http://mail.openjdk.java.net/pipermail/nio-dev/2015-May/003140.html

       

      java.io.IOException: Failed to sync E:\workspace\OSSHadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-571178992-10.123.152.148-1526591934139\current\rbw
      at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.fsyncDirectory(DatanodeUtil.java:160)
      at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.flushOrSync(BlockReceiver.java:430)
      at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:807)
      at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:873)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:291)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.nio.file.AccessDeniedException: E:\workspace\OSSHadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-571178992-10.123.152.148-1526591934139\current\rbw
      at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
      at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
      at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
      at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
      at java.nio.channels.FileChannel.open(FileChannel.java:287)
      at java.nio.channels.FileChannel.open(FileChannel.java:335)
      at org.apache.hadoop.io.IOUtils.fsync(IOUtils.java:405)
      at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.dirSync(FileIoProvider.java:169)
      at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.fsyncDirectory(DatanodeUtil.java:158)
      ... 8 more
      

       

        Attachments

        1. HDFS-13586.000.patch
          1.0 kB
          Lukas Majercak
        2. HDFS-13586.001.patch
          1.0 kB
          Lukas Majercak

          Issue Links

            Activity

              People

              • Assignee:
                lukmajercak Lukas Majercak
                Reporter:
                lukmajercak Lukas Majercak
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: