Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-970

FSImage writing should always fsync before close

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.20.1, 0.21.0, 0.22.0
    • 0.22.0
    • namenode
    • None
    • Reviewed

    Description

      Without an fsync, it's common that filesystems will delay the writing of metadata to the journal until all of the data blocks have been flushed. If the system crashes while the dirty pages haven't been flushed, the file is left in an indeterminate state. In some FSs (eg ext4) this will result in a 0-length file. In others (eg XFS) it will result in the correct length but any number of data blocks getting zeroed. Calling FileChannel.force before closing the FSImage prevents this issue.

      Attachments

        1. hdfs-970.txt
          2 kB
          Todd Lipcon

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tlipcon Todd Lipcon
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment