Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-2151

Ozone client prints the entire request payload in DEBUG level.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.2
    • Component/s: None

      Description

      In XceiverClientRatis.java:221, we have the following snippet where we have a DEBUG line that prints out the entire Container Request proto.

            ContainerCommandRequestProto finalPayload =
                ContainerCommandRequestProto.newBuilder(request)
                    .setTraceID(TracingUtil.exportCurrentSpan())
                    .build();
            boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload);
            ByteString byteString = finalPayload.toByteString();
            LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload);
            return isReadOnlyRequest ?
                getClient().sendReadOnlyAsync(() -> byteString) :
                getClient().sendAsync(() -> byteString);
      

      This causes OOM while writing large (~300MB) keys.

      SLF4J: Failed toString() invocation on an object of type [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto]
      Reported exception:
      java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOf(Arrays.java:3332)
      	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
      	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
      	at java.lang.StringBuilder.append(StringBuilder.java:202)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68)
      	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
      	at org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
      	at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
      	at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
      	at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
      	at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
      	at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
      	at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252)
      	at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221)
      	at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302)
      	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310)
      	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601)
      	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459)
      	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240)
      	at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:129)
      SLF4J: Failed toString() invocation on an object of type [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto]
      Reported exception:
      java.lang.OutOfMemoryError: Java heap space
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                adoroszlai Attila Doroszlai
                Reporter:
                avijayan Aravindan Vijayan
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m