Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1245

Plugable block id generation

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 2.0.3-alpha
    • namenode
    • None

    Description

      The idea is to have a way to easily create block id generation engines that may fit a certain purpose. One of them could be HDFS-898 started by Konstantin, but potentially others.

      We chatted with Dhruba about this for a while and came up with the following approach:
      There should be a BlockIDGenerator interface that has following methods:

      void blockAdded(Block)
      void blockRemoved(Block)
      Block nextBlock()

      First two methods are needed for block generation engines that hold a certain state. During the restart, when namenode reads the fsimage it will notify generator about all the blocks it reads from the image and during runtime namenode will notify the generator about block removals on file deletion.

      The instance of the generator will also have a reference to the block registry, the interface that BlockManager implements. The only method there is _blockExists(Block)_, so that the current random block id generation can be implemented, since it needs to check with the block manager if the id is already present.

      What does the community think about this proposal?

      Attachments

        1. blockIdGenerator-branch2.patch
          8 kB
          Konstantin Shvachko
        2. blockIdGenerator.patch
          8 kB
          Konstantin Shvachko

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            shv Konstantin Shvachko
            dms Dmytro Molkov
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment