Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12293

DataNode should log file name on disk error

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: datanode
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Found the following error message in precommit build https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/488/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/

      2017-08-10 09:36:53,619 [DataXceiver for client DFSClient_NONMAPREDUCE_670847838_18 at /127.0.0.1:55851 [Receiving block BP-219227751-172.17.0.2-1502357801473:blk_1073741829_1005]] WARN  datanode.DataNode (BlockReceiver.java:<init>(287)) - IOException in BlockReceiver constructor. Cause is 
      java.io.IOException: Not a directory
      	at java.io.UnixFileSystem.createFileExclusively(Native Method)
      	at java.io.File.createNewFile(File.java:1012)
      	at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.createFile(FileIoProvider.java:302)
      	at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createFileWithExistsCheck(DatanodeUtil.java:69)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:306)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:933)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbw(FsVolumeImpl.java:1202)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1356)
      	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:215)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1291)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
      

      It is not known what file was being created.
      What's interesting is that DatanodeUtil#createFileWithExistsCheck does carry file name in the exception message, but the exception handler at DataTransfer#run() and BlockReceiver#BlockReceiver ignores it:

      BlockReceiver#BlockReceiver
            // check if there is a disk error
            IOException cause = DatanodeUtil.getCauseIfDiskError(ioe);
            DataNode.LOG.warn("IOException in BlockReceiver constructor"
                + (cause == null ? "" : ". Cause is "), cause);
            if (cause != null) {
              ioe = cause;
              // Volume error check moved to FileIoProvider
            }
      

      The logger should print the file name in addition to the cause.

        Attachments

        1. HDFS-12293.01.patch
          0.9 kB
          Ajay Kumar
        2. HDFS-12293.02.patch
          2 kB
          Ajay Kumar

          Activity

            People

            • Assignee:
              ajayydv Ajay Kumar
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: