Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11915

Sync rbw dir on the first hsync() to avoid file lost on power failure

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 3.1.0, 2.10.0, 2.9.1, 3.0.1
    • None
    • None

    Description

      As discussed in HDFS-5042, there is a chance to lose blocks on power failure if rbw file creation entry is not yet sync to device. Then the block created is nowhere exists on disk. Neither in rbw nor in finalized.

      As suggested by Kihwal Lee, will discuss and track it in this JIRA.

      As suggested by [~vinayrpet], May be first hsync() request on block file can call fsync on its parent directory (rbw) directory.

      Attachments

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HDFS-13586 Fsync fails on directories on Windows

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-5042 Completed files lost after power failure

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        relates to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-13487 Backport HDFS-11915 to branch-2.8, branch-2.7

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vinayakumarb Vinayakumar B Assign to me
            kanaka Kanaka Kumar Avvaru
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment