Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12142

Files may be closed before streamer is done

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.8.0
    • None
    • hdfs
    • None

    Description

      We're encountering multiple cases of clients calling updateBlockForPipeline on completed blocks. Initial analysis is the client closes a file, completeFile succeeds, then it immediately attempts recovery. The exception is swallowed on the client, only logged on the NN by checkUCBlock.

      The problem "appears" to be benign (no data loss) but it's unproven if the issue always occurs for successfully closed files. There appears to be very poor coordination between the dfs output stream's threads which leads to races that confuse the streamer thread – which probably should have been joined before returning from close.

      Attachments

        Activity

          People

            Unassigned Unassigned
            daryn Daryn Sharp
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated: