Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7421

Move processing of postponed over-replicated blocks to a background task

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: ha, namenode
    • Labels:
      None
    • Target Version/s:

      Description

      In an HA environment, we postpone sending block invalidates to DNs until all DNs holding a given block have done at least one block report to the NN after it became active. When that first block report after becoming active does occur, we attempt to reprocess all postponed misreplicated blocks inline with the block report RPC. In the case where there are many postponed misreplicated blocks, this can cause block report RPCs to take an inordinately long time to complete, sometimes on the order of minutes, which has the potential to tie up RPC handlers, block incoming RPCs, etc. There's no need to hurriedly process all postponed misreplicated blocks so that we can quickly send invalidate commands back to DNs, so let's move this processing outside of the RPC handler context and into a background thread.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                atm Aaron Myers
                Reporter:
                atm Aaron Myers
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: