[HDDS-8085] Reduce network traffic in XceiverClientGrpc::sendCommandWithRetry

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Component/s: Ozone Client
• Labels: performance
• Assignee: Ritesh Shukla
• Reporter: Rajesh Balamohan
• Created: 06/Mar/23 07:21
• Updated: 30/Mar/23 18:35
• Resolved: 30/Mar/23 18:35

Description

Context: simple "select count from store_sales" causes lot of network traffic in Ozone than in HDFS.

Probing revealed good amount of time being spent in splits, which needs to be fixed separately.

Another potential fix is that, XceiverClientGrpc gets a list of DataNodes which it can communicate with in the task.

If DN matches that of localhost, it should prefer localhost rather than communicating with another node.

Request is get "InetAddress.getLocalHost().getHostName().trim()", and compare with the set of hosts available in "DatanodeDetails". If found, prefer that DN rather than remote node.

https://github.com/apache/ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java#L397-L440

Stacktrace below many not match with apache ozone, but pasting it to provide codepath details.

	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:398)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:330)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc$$Lambda$71/1551468933.get(Unknown Source)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:174)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:148)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:324)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:305)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:283)
	at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:432)
	at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkDataIntoBuffers(ChunkInputStream.java:404)
	at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:389)
	- eliminated <0x00000007823965b0> (a org.apache.hadoop.hdds.scm.storage.ChunkInputStream)
	at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:321)
	- locked <0x00000007823965b0> (a org.apache.hadoop.hdds.scm.storage.ChunkInputStream)
	at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:207)
	- locked <0x00000007823965b0> (a org.apache.hadoop.hdds.scm.storage.ChunkInputStream)
	at org.apache.hadoop.hdds.scm.storage.ByteBufferReader.readFromBlock(ByteBufferReader.java:56)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:303)
	- locked <0x0000000783646d30> (a org.apache.hadoop.hdds.scm.storage.BlockInputStream)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:64)
	- locked <0x0000000783646d30> (a org.apache.hadoop.hdds.scm.storage.BlockInputStream)
	at org.apache.hadoop.hdds.scm.storage.ByteBufferReader.readFromBlock(ByteBufferReader.java:56)
	at org.apache.hadoop.ozone.client.io.KeyInputStream.readWithStrategy(KeyInputStream.java:273)
	- locked <0x0000000783646ca0> (a org.apache.hadoop.ozone.client.io.KeyInputStream)
	at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:250)
	- locked <0x0000000783646ca0> (a org.apache.hadoop.ozone.client.io.KeyInputStream)
	at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:110)
	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:152)
	at org.apache.parquet.hadoop.util.H2SeekableInputStream$H2Reader.read(H2SeekableInputStream.java:81)
	at org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:90)
	at org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:75)
	at org.apache.parquet.hadoop.ParquetFileReader$ConsecutivePartList.readAll(ParquetFileReader.java:1427)
	at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:846)
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:447)
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:425)
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:377)
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:99)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
	at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:118)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecor

Issue Links

• is fixed by: HDDS-8300 Enable topology aware replication and reads by default