Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14048

DFSOutputStream close() throws exception on subsequent call after DataNode restart

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      We recently discovered an issue in which, during a rolling upgrade, some jobs were failing with exceptions like (sadly this is the whole stack trace):

      java.io.IOException: A datanode is restarting: DatanodeInfoWithStorage[1.1.1.1:71,BP-XXXX,DISK]
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
      

      with an earlier statement in the log like:

      INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting: DatanodeInfoWithStorage[1.1.1.1:71,BP-XXXX,DISK]
      

      Strangely we did not see any other logs about the DFSOutputStream failing after waiting for the DataNode restart. We eventually realized that in some cases DFSOutputStream#close() may be called more than once, and that if so, the IOException above is thrown on the second call to close() (this is even with HDFS-5335; prior to this it would have been thrown on all calls to close() besides the first).

      The problem is that in DataStreamer#createBlockOutputStream(), after the new output stream is created, it resets the error states:

              errorState.resetInternalError();
              // remove all restarting nodes from failed nodes list
              failed.removeAll(restartingNodes);
              restartingNodes.clear(); 
      

      But it forgets to clear lastException. When DFSOutputStream#closeImpl() is called a second time, this block is triggered:

          if (isClosed()) {
            LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
                closed, getStreamer().streamerClosed());
            try {
              getStreamer().getLastException().check(true);
      

      The second time, isClosed() is true, so the exception checking occurs and the "Datanode is restarting" exception is thrown even though the stream has already been successfully closed.

      Attachments

        1. HDFS-14048-branch-2.000.patch
          2 kB
          Erik Krogen
        2. HDFS-14048.000.patch
          2 kB
          Erik Krogen

        Issue Links

          Activity

            People

              xkrogen Erik Krogen
              xkrogen Erik Krogen
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: