Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14728

RBF: GetDatanodeReport causes a large GC pressure on the NameNodes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • rbf
    • None

    Description

      When a cluster contains millions of DNs, GetDatanodeReport is pretty expensive, and it will cause a large GC pressure on NameNode.
      When multiple NSs share the millions DNs by federation and the router listens to the NSs, the problem will be more serious.
      All the NSs will be GC at the same time.

      RBF should cache the datanode report informations and have an option to disable the cache.

      Attachments

        1. HDFS-14728-trunk-001.patch
          14 kB
          ZanderXu
        2. HDFS-14728-trunk-002.patch
          13 kB
          ZanderXu
        3. HDFS-14728-trunk-003.patch
          15 kB
          ZanderXu
        4. HDFS-14728-trunk-004.patch
          20 kB
          ZanderXu
        5. HDFS-14728-trunk-005.patch
          20 kB
          ZanderXu
        6. HDFS-14728-trunk-006.patch
          21 kB
          ZanderXu
        7. HDFS-14728-trunk-007.patch
          21 kB
          ZanderXu
        8. HDFS-14728-trunk-008.patch
          24 kB
          ZanderXu
        9. HDFS-14728-trunk-009.patch
          24 kB
          ZanderXu

        Activity

          People

            xuzq_zander ZanderXu
            xuzq_zander ZanderXu
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated: