Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7836 BlockManager Scalability Improvements
  3. HDFS-7923

The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      The DataNodes should rate-limit their full block reports. They can do this by first sending a heartbeat message to the NN with an optional boolean set which requests permission to send a full block report. If the NN responds with another optional boolean set, the DN will send an FBR... if not, it will wait until later. This can be done compatibly with optional fields.

        Attachments

        1. HDFS-7923.000.patch
          41 kB
          Charles Lamb
        2. HDFS-7923.001.patch
          41 kB
          Charles Lamb
        3. HDFS-7923.002.patch
          46 kB
          Charles Lamb
        4. HDFS-7923.003.patch
          74 kB
          Colin P. McCabe
        5. HDFS-7923.004.patch
          80 kB
          Colin P. McCabe
        6. HDFS-7923.006.patch
          77 kB
          Colin P. McCabe
        7. HDFS-7923.007.patch
          77 kB
          Colin P. McCabe

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin P. McCabe
                Reporter:
                cmccabe Colin P. McCabe
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: