[KUDU-3489] Subprocess communication with large messages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17.0
• Assignee: Unassigned
• Reporter: Abhishek Chennaka
• Created: 10/Jul/23 18:55
• Updated: 26/Jul/23 05:40
• Resolved: 13/Jul/23 17:15

Description

With KUDU-3450 we started supporting messages of size up to 8MB by default to be transmitted between Kudu master and subprocess server. Given the the maximum pipe buffer size is 1MB in linux machines [1] we see flaky failures if the size of the message is more than 1MB between Kudu master and the subprocess server.

[1] https://man7.org/linux/man-pages/man7/pipe.7.html (pipe-max-size)

Example failure:

Exception in thread "main" java.util.concurrent.CompletionException: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf message
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf message
	at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:77)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
	... 3 more
Caused by: java.io.IOException: unable to receive message, expected (2814368) bytes but read (1777660) bytes
	at org.apache.kudu.subprocess.MessageIO.doRead(MessageIO.java:101)
	at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:81)
	at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
	... 4 more