[SPARK-39702] Reduce memory overhead of TransportCipher$EncryptedMessage's byteRawChannel buffer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core, YARN
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 07/Jul/22 01:05
• Updated: 10/Jul/22 20:46
• Resolved: 10/Jul/22 20:46

Description

With Spark's encryption enabled (spark.network.crypto.enabled set to true and spark.network.crypto.saslFallback set to false), I ran into memory usage problems in the external shuffle service.

This was caused by a problem that is very similar to SPARK-24801: each TransportCipher$EncryptedMessage eagerly initializes a buffer that is used during the encryption process. This buffer is only used once transferTo is called, but it is eagerly initialized in the EncryptedMessage constructor. This leads to high memory usage when there are many messages queued in an outgoing channel.

One possible fix would be to mimic SPARK-24801 and make the initialization lazy. However, we can actually go one step further and share a single re-used buffer across multiple messages. This is safe because those messages are already sharing a different buffer which is accessed in the same write paths.

Issue Links

• relates to: SPARK-24801 Empty byte[] arrays in spark.network.sasl.SaslEncryption$EncryptedMessage can waste a lot of memory
• links to: [Github] Pull Request #37110 (JoshRosen)