Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2260

Log block manager should handle null bytes in metadata on crash

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.8.0
    • fs
    • None

    Description

      The log block manager currently may leave null bytes at the end of the metadata log file if there is a system crash in the middle of a write. The log block manager should detect null bytes at the end of a metadata entry on startup and potentially truncate the entry or close the container.

      Currently, it prints an error along the following lines:

      F0111 09:30:27.327011 28843 tablet_server_main.cc:64] Check failed: _s.ok() Bad status: Corruption: Failed to load FS layout: Could not read records from container /data/3/kudu/data/f70391c7c6084e08bbae7448518e0b5e: Data length checksum does not match: Incorrect checksum in file /data/3/kudu/data/f70391c7c6084e08bbae7448518e0b5e.metadata at offset 372533: Checksum does not match. Expected: 0. Actual: 1323915147
      

      At the time of writing, the workaround for this issue is to truncate the affected file at the start of the incomplete entry in the file. While this may leave orphaned blocks, this should be safe because if the metadata entry was never successfully written then it should not have been considered durable, either.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wdberkeley William Berkeley
            mpercy Mike Percy
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment