Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12293

DataNode should log file name on disk error

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: datanode
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Found the following error message in precommit build https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/488/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/

      2017-08-10 09:36:53,619 [DataXceiver for client DFSClient_NONMAPREDUCE_670847838_18 at /127.0.0.1:55851 [Receiving block BP-219227751-172.17.0.2-1502357801473:blk_1073741829_1005]] WARN  datanode.DataNode (BlockReceiver.java:<init>(287)) - IOException in BlockReceiver constructor. Cause is 
      java.io.IOException: Not a directory
      	at java.io.UnixFileSystem.createFileExclusively(Native Method)
      	at java.io.File.createNewFile(File.java:1012)
      	at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.createFile(FileIoProvider.java:302)
      	at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createFileWithExistsCheck(DatanodeUtil.java:69)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:306)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:933)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbw(FsVolumeImpl.java:1202)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1356)
      	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:215)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1291)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
      

      It is not known what file was being created.
      What's interesting is that DatanodeUtil#createFileWithExistsCheck does carry file name in the exception message, but the exception handler at DataTransfer#run() and BlockReceiver#BlockReceiver ignores it:

      BlockReceiver#BlockReceiver
            // check if there is a disk error
            IOException cause = DatanodeUtil.getCauseIfDiskError(ioe);
            DataNode.LOG.warn("IOException in BlockReceiver constructor"
                + (cause == null ? "" : ". Cause is "), cause);
            if (cause != null) {
              ioe = cause;
              // Volume error check moved to FileIoProvider
            }
      

      The logger should print the file name in addition to the cause.

      1. HDFS-12293.01.patch
        0.9 kB
        Ajay Kumar
      2. HDFS-12293.02.patch
        2 kB
        Ajay Kumar

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 15m 36s trunk passed
        +1 compile 0m 57s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 1m 4s trunk passed
        -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings.
        +1 javadoc 0m 45s trunk passed
              Patch Compile Tests
        +1 mvninstall 1m 3s the patch passed
        +1 compile 0m 55s the patch passed
        +1 javac 0m 55s the patch passed
        +1 checkstyle 0m 38s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 4s the patch passed
        +1 javadoc 0m 42s the patch passed
              Other Tests
        -1 unit 73m 30s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        102m 43s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12293
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881816/HDFS-12293.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3bede22a8253 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5558792
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20691/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20691/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20691/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 15m 36s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 1m 4s trunk passed -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. +1 javadoc 0m 45s trunk passed       Patch Compile Tests +1 mvninstall 1m 3s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 0m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 42s the patch passed       Other Tests -1 unit 73m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 102m 43s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12293 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881816/HDFS-12293.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3bede22a8253 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5558792 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20691/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20691/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20691/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ajayydv Ajay Kumar added a comment - - edited

        Wei-Chiu Chuang, Could you plz review the patch, test failures in jenkins are unrelated.

        Show
        ajayydv Ajay Kumar added a comment - - edited Wei-Chiu Chuang , Could you plz review the patch, test failures in jenkins are unrelated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for the patch, Ajay Kumar.

        Looks good. Would you also update DataTransfer#run() as well?

        Thanks

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for the patch, Ajay Kumar. Looks good. Would you also update DataTransfer#run() as well? Thanks
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 40s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 14m 33s trunk passed
        +1 compile 0m 50s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 0m 40s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 7s the patch passed
        +1 javadoc 0m 45s the patch passed
              Other Tests
        -1 unit 92m 10s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        120m 12s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12293
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882434/HDFS-12293.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8feff599c05f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ab1a8ae
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20747/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20747/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20747/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 40s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 14m 33s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 0m 56s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 40s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 0m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 45s the patch passed       Other Tests -1 unit 92m 10s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 120m 12s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12293 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882434/HDFS-12293.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8feff599c05f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ab1a8ae Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20747/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20747/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20747/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ajayydv Ajay Kumar added a comment -

        Wei-Chiu Chuang could you please check the updated patch?

        Show
        ajayydv Ajay Kumar added a comment - Wei-Chiu Chuang could you please check the updated patch?
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 for the v2 patch.

        Wei-Chiu Chuang, does the patch look fine to you?

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 for the v2 patch. Wei-Chiu Chuang , does the patch look fine to you?
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Arpit Agarwal and Ajay Kumar
        +1 LGTM!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Arpit Agarwal and Ajay Kumar +1 LGTM!
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks Wei-Chiu Chuang. I've committed this.

        Thanks for the contribution Ajay Kumar.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks Wei-Chiu Chuang . I've committed this. Thanks for the contribution Ajay Kumar .
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12256/)
        HDFS-12293. DataNode should log file name on disk error. Contributed by (arp: rev a1e3f84afe6c02cc642699634052d2fb60b30179)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12256/ ) HDFS-12293 . DataNode should log file name on disk error. Contributed by (arp: rev a1e3f84afe6c02cc642699634052d2fb60b30179) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java

          People

          • Assignee:
            ajayydv Ajay Kumar
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development