Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-978

Record every new block allocation of a file into the transaction log.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      HDFS should record every new block allocation (of a file) into its transaction logs. In the current code, block allocations are persisted only when a file is closed or hflush-ed. This feature will enable HDFS writers to survive namenode restarts.

        Issue Links

          Activity

          Hide
          gnawux Wang Xu added a comment -

          Do you mean log it after a block is written by datanodes or just allocated?

          Show
          gnawux Wang Xu added a comment - Do you mean log it after a block is written by datanodes or just allocated?
          Hide
          dhruba dhruba borthakur added a comment -

          When a writer requests a new block for a file, the NN should record the new blockid in the edits log. This persists the mapping from a filename to the list of its blocks. Now, the writer can write to this block (directly to the DataNode(s)) while the NN can be restarted without interrupting/invalidating that write.

          Show
          dhruba dhruba borthakur added a comment - When a writer requests a new block for a file, the NN should record the new blockid in the edits log. This persists the mapping from a filename to the list of its blocks. Now, the writer can write to this block (directly to the DataNode(s)) while the NN can be restarted without interrupting/invalidating that write.
          Hide
          eli Eli Collins added a comment -

          FSNameSystem#getAdditionalBlock currently doesn't sync the log, it will need to once block allocations are logged.

          Show
          eli Eli Collins added a comment - FSNameSystem#getAdditionalBlock currently doesn't sync the log, it will need to once block allocations are logged.
          Hide
          shv Konstantin Shvachko added a comment -

          This is required for HA. If there is no addBlock transaction the StandbyNode will not know that the block exists, and therefore will treat blockReceived from DNs as not belonging to any file.
          I propose

          1. to measure write performance degradation if addBlock is synced, and if performance really bad,
          2. introduce a "lightweight" flush(), which does flush, but doesn't sync to disk.

          The latter is a performance/reliability tradeoff. For EditLogBackupOutputStream the lightweight flush should push transaction to BN.

          Show
          shv Konstantin Shvachko added a comment - This is required for HA. If there is no addBlock transaction the StandbyNode will not know that the block exists, and therefore will treat blockReceived from DNs as not belonging to any file. I propose to measure write performance degradation if addBlock is synced, and if performance really bad, introduce a "lightweight" flush(), which does flush, but doesn't sync to disk. The latter is a performance/reliability tradeoff. For EditLogBackupOutputStream the lightweight flush should push transaction to BN.
          Hide
          tlipcon Todd Lipcon added a comment -

          HDFS-1108 has some work which supports doing this on a file-by-file basis

          Show
          tlipcon Todd Lipcon added a comment - HDFS-1108 has some work which supports doing this on a file-by-file basis
          Hide
          tlipcon Todd Lipcon added a comment -

          Going to resolve this as duplicate of HDFS-1108, where there's an initial patch.

          Show
          tlipcon Todd Lipcon added a comment - Going to resolve this as duplicate of HDFS-1108 , where there's an initial patch.
          Hide
          mattf Matt Foley added a comment -

          Closing on release of 1.1.0.

          Show
          mattf Matt Foley added a comment - Closing on release of 1.1.0.
          Hide
          mattf Matt Foley added a comment -

          This was correctly resolved as a duplicate, but the link was not created at that time.

          Show
          mattf Matt Foley added a comment - This was correctly resolved as a duplicate, but the link was not created at that time.

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              dhruba dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development