Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1623 High Availability Framework for HDFS NN
  3. HDFS-2602

NN should log newly-allocated blocks without losing BlockInfo

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • HA branch (HDFS-1623)
    • HA branch (HDFS-1623)
    • ha
    • None

    Description

      Without the patch in HDFS-1108, new block allocations aren't logged to the edits log. For HA, we'll need that functionality and we'll need to make sure that block locations aren't blown away in the Standby NN when tailing the edits log.

      As described in HDFS-1975:

      When we close a file, or add another block to a file, we write OP_CLOSE or OP_ADD in the txn log. FSEditLogLoader, when it sees these types of transactions, creates new BlockInfo objects for all of the blocks listed in the transaction. These new BlockInfos have no block locations associated. So, when we close a file, the SBNN loses its block locations info for that file and is no longer "hot".

      I have an ugly hack which copies over the old BlockInfos from the existing INode, but I'm not convinced it's the right way. It might be cleaner to add new opcode types like OP_ADD_ADDITIONAL_BLOCK, and actually treat OP_CLOSE as just a finalization of INodeFileUnderConstruction to INodeFile, rather than replacing block info at all.

      Attachments

        1. HDFS-2602.patch
          46 kB
          Aaron Myers
        2. HDFS-2602.patch
          46 kB
          Aaron Myers
        3. HDFS-2602.patch
          38 kB
          Aaron Myers
        4. HDFS-2602.patch
          38 kB
          Aaron Myers
        5. HDFS-2602.patch
          47 kB
          Aaron Myers

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            atm Aaron Myers
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment