Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7836 BlockManager Scalability Improvements
  3. HDFS-7923

The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.8.0, 3.0.0-alpha1
    • None
    • None

    Description

      The DataNodes should rate-limit their full block reports. They can do this by first sending a heartbeat message to the NN with an optional boolean set which requests permission to send a full block report. If the NN responds with another optional boolean set, the DN will send an FBR... if not, it will wait until later. This can be done compatibly with optional fields.

      Attachments

        1. HDFS-7923.000.patch
          41 kB
          Charles Lamb
        2. HDFS-7923.001.patch
          41 kB
          Charles Lamb
        3. HDFS-7923.002.patch
          46 kB
          Charles Lamb
        4. HDFS-7923.003.patch
          74 kB
          Colin McCabe
        5. HDFS-7923.004.patch
          80 kB
          Colin McCabe
        6. HDFS-7923.006.patch
          77 kB
          Colin McCabe
        7. HDFS-7923.007.patch
          77 kB
          Colin McCabe

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HDFS-12914 Block report leases cause missing blocks until next report

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-8589 Fix unused imports in BPServiceActor and BlockReportLeaseManager

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          Activity

            People

              cmccabe Colin McCabe
              cmccabe Colin McCabe
              Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: