Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently in HDFS there are a lot of calls to IOUtils.closeStream that are from finally blocks. I'm worried that this can lead to data corruption in the file system. Take the first instance in DataNode.copyBlock: it writes the block and then calls closeStream on the output stream. If there is an error at the end of the file that is detected in the close, it will be completely ignored. Note that logging the error is not enough, the error should be thrown so that the client knows the failure happened.

         try {
           file1.write(...);
           file2.write(...);
         } finally {
            IOUtils.closeStream(file);
        }
      

      is bad. It must be rewritten as:

         try {
           file1.write(...);
           file2.write(...);
           file1.close(...);
           file2.close(...);
         } catch (IOException ie) {
           IOUtils.closeStream(file1);
           IOUtils.closeStream(file2);
           throw ie;
         }
      

      I also think that IOUtils.closeStream should be renamed IOUtils.cleanupFailedStream or something to make it clear it can only be used after the write operation has failed and is being cleaned up.

      1. closeStream.patch
        2 kB
        dhruba borthakur

        Activity

        Owen O'Malley created issue -
        dhruba borthakur made changes -
        Field Original Value New Value
        Attachment closeStream.patch [ 12377211 ]
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        dhruba borthakur made changes -
        Fix Version/s 0.16.1 [ 12312927 ]
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Owen O'Malley made changes -
        Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
        Key HADOOP-2926 HDFS-43
        Affects Version/s 0.16.0 [ 12312740 ]
        Component/s dfs [ 12310710 ]

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            Owen O'Malley
          • Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development