Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12612

DFSStripedOutputStream#close will throw if called a second time with a failed streamer

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      Found while testing with Hive. We have a cluster with 2 DNs and the XOR-2-1 policy. If you write a file and call close() twice, it throws this exception:

      17/10/04 16:02:14 WARN hdfs.DFSOutputStream: Cannot allocate parity block(index=2, policy=XOR-2-1-1024k). Not enough datanodes? Exclude nodes=[]
      ...
      Caused by: java.io.IOException: Failed to get parity block, index=2
      at org.apache.hadoop.hdfs.DFSStripedOutputStream.allocateNewBlock(DFSStripedOutputStream.java:500) ~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
      at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:524) ~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
      

      This is because in DFSStripedOutputStream#closeImpl, if the stream is closed, we throw an exception if any of the striped streamers had an exception:

        protected synchronized void closeImpl() throws IOException {
          if (isClosed()) {
            final MultipleIOException.Builder b = new MultipleIOException.Builder();
            for(int i = 0; i < streamers.size(); i++) {
              final StripedDataStreamer si = getStripedDataStreamer(i);
              try {
                si.getLastException().check(true);
              } catch (IOException e) {
                b.add(e);
              }
            }
            final IOException ioe = b.build();
            if (ioe != null) {
              throw ioe;
            }
            return;
          }
      

      I think this is incorrect, since we only need to throw in this situation if we have too many failed streamers. close should also be idempotent, so it should throw the first time we call close if it's going to throw at all.

        Attachments

        1. HDFS-12612.00.patch
          3 kB
          Lei (Eddy) Xu
        2. HDFS-12612.01.patch
          9 kB
          Lei (Eddy) Xu
        3. HDFS-12612.02.patch
          11 kB
          Lei (Eddy) Xu
        4. HDFS-12612.03.patch
          11 kB
          Lei (Eddy) Xu

        Issue Links

          Activity

            People

            • Assignee:
              eddyxu Lei (Eddy) Xu
              Reporter:
              andrew.wang Andrew Wang

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment