Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7836 BlockManager Scalability Improvements
  3. HDFS-7923

The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages

VotersStop watchingWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.8.0, 3.0.0-alpha1
    • None
    • None

    Description

      The DataNodes should rate-limit their full block reports. They can do this by first sending a heartbeat message to the NN with an optional boolean set which requests permission to send a full block report. If the NN responds with another optional boolean set, the DN will send an FBR... if not, it will wait until later. This can be done compatibly with optional fields.

      Attachments

        1. HDFS-7923.000.patch
          41 kB
          Charles Lamb
        2. HDFS-7923.001.patch
          41 kB
          Charles Lamb
        3. HDFS-7923.002.patch
          46 kB
          Charles Lamb
        4. HDFS-7923.003.patch
          74 kB
          Colin McCabe
        5. HDFS-7923.004.patch
          80 kB
          Colin McCabe
        6. HDFS-7923.006.patch
          77 kB
          Colin McCabe
        7. HDFS-7923.007.patch
          77 kB
          Colin McCabe

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HDFS-12914 Block report leases cause missing blocks until next report

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        is related to

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-8589 Fix unused imports in BPServiceActor and BlockReportLeaseManager

        • Trivial - Cosmetic problem like misspelt words or misaligned text.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cmccabe Colin McCabe
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            22 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment