[HDFS-13586] Fsync fails on directories on Windows

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3
• Component/s: datanode, hdfs
• Labels: Windows

Environment

JDK 1.8.0_144
Hadoop 2.9+

• Hadoop Flags: Reviewed
• Assignee: Lukas Majercak
• Reporter: Lukas Majercak
• Created: 17/May/18 21:04
• Updated: 02/Oct/19 17:15
• Resolved: 18/May/18 02:29

Description

HDFS-11915 added a fsync call on DataNode's rbw directory on the first hsync() call. IOUtils.fsync first tries to get a FileChannel on the directory using FileChannel.open(READ). This call fails on Windows for any directory and throws an AccessDeniedException, see discussion here: http://mail.openjdk.java.net/pipermail/nio-dev/2015-May/003140.html. 

 

java.io.IOException: Failed to sync E:\workspace\OSSHadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-571178992-10.123.152.148-1526591934139\current\rbw
at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.fsyncDirectory(DatanodeUtil.java:160)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.flushOrSync(BlockReceiver.java:430)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:807)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:873)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:291)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.AccessDeniedException: E:\workspace\OSSHadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-571178992-10.123.152.148-1526591934139\current\rbw
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.hadoop.io.IOUtils.fsync(IOUtils.java:405)
at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.dirSync(FileIoProvider.java:169)
at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.fsyncDirectory(DatanodeUtil.java:158)
... 8 more

 

Issue Links

• is broken by: HDFS-11915 Sync rbw dir on the first hsync() to avoid file lost on power failure
• is related to: HADOOP-15475 Fix broken unit tests on Windows