Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10312

Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • namenode
    • None
    • Reviewed

    Description

      Our RPC server caps the maximum size of incoming messages at 64 MB by default. For exceptional circumstances, this can be uptuned using ipc.maximum.data.length. However, for block reports, there is still an internal maximum length restriction of 64 MB enforced by protobuf. (Sample stack trace to follow in comments.) This issue proposes to apply the same override to our block list decoding, so that large block reports can proceed.

      Attachments

        1. HDFS-10312.004.patch
          23 kB
          Chris Nauroth
        2. HDFS-10312.003.patch
          23 kB
          Chris Nauroth
        3. HDFS-10312.002.patch
          21 kB
          Chris Nauroth
        4. HDFS-10312.001.patch
          21 kB
          Chris Nauroth

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HDFS-10349 Ozone: StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13039 Add documentation for configuration property ipc.maximum.data.length for controlling maximum RPC message size.

          • Major - Major loss of function.
          • Closed

          Activity

            People

              cnauroth Chris Nauroth
              cnauroth Chris Nauroth
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: