[HDDS-2151] Ozone client prints the entire request payload in DEBUG level.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Labels: pull-request-available
• Assignee: Attila Doroszlai
• Reporter: Aravindan Vijayan
• Created: 19/Sep/19 05:07
• Updated: 24/Feb/20 20:52
• Resolved: 19/Sep/19 17:59

Description

In XceiverClientRatis.java:221, we have the following snippet where we have a DEBUG line that prints out the entire Container Request proto.

      ContainerCommandRequestProto finalPayload =
          ContainerCommandRequestProto.newBuilder(request)
              .setTraceID(TracingUtil.exportCurrentSpan())
              .build();
      boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload);
      ByteString byteString = finalPayload.toByteString();
      LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload);
      return isReadOnlyRequest ?
          getClient().sendReadOnlyAsync(() -> byteString) :
          getClient().sendAsync(() -> byteString);

This causes OOM while writing large (~300MB) keys.

SLF4J: Failed toString() invocation on an object of type [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto]
Reported exception:
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
	at java.lang.StringBuilder.append(StringBuilder.java:202)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68)
	at org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
	at org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
	at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
	at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
	at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
	at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
	at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
	at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252)
	at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221)
	at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240)
	at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:129)
SLF4J: Failed toString() invocation on an object of type [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto]
Reported exception:
java.lang.OutOfMemoryError: Java heap space

Issue Links

• links to: GitHub Pull Request #1477