Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2296

Kudu RPC cannot deserialize messages larger than 64MB

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: rpc
    • Labels:
      None
    • Target Version/s:

      Description

      Impala has been testing Kudu RPC with a larger value for rpc_max_message_size. I noticed that when the message size exceeds 64MB, rpc::serialization::ParseMessage() hits this condition:

      if (PREDICT_FALSE(!in.Skip(main_msg_len))) {
        return Status::Corruption(
            StringPrintf("Invalid packet: data too short, expected %d byte main_msg", main_msg_len),
            KUDU_REDACT(buf.ToDebugString()));
      }
      

      The actual buffer is the appropriate size. What is happening is that protobuf imposes a 64MB total byte limit by default. Once a message exceeds that, the Skip() call will return false when trying to go past the 64MB limit. The deserialization code can get around this by setting the total byte limit with CodedInputSTream::SetTotalByteLimit().

      This should not impact existing systems at the moment, because the default value for rpc_max_message_size is 50MB.

        Attachments

          Activity

            People

            • Assignee:
              joemcdonnell Joe McDonnell
              Reporter:
              joemcdonnell Joe McDonnell
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: