Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10312

Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Our RPC server caps the maximum size of incoming messages at 64 MB by default. For exceptional circumstances, this can be uptuned using ipc.maximum.data.length. However, for block reports, there is still an internal maximum length restriction of 64 MB enforced by protobuf. (Sample stack trace to follow in comments.) This issue proposes to apply the same override to our block list decoding, so that large block reports can proceed.

        Attachments

        1. HDFS-10312.001.patch
          21 kB
          Chris Nauroth
        2. HDFS-10312.002.patch
          21 kB
          Chris Nauroth
        3. HDFS-10312.003.patch
          23 kB
          Chris Nauroth
        4. HDFS-10312.004.patch
          23 kB
          Chris Nauroth

          Issue Links

            Activity

              People

              • Assignee:
                cnauroth Chris Nauroth
                Reporter:
                cnauroth Chris Nauroth
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: