[SPARK-43583] When encryption is enabled on the External Shuffle Service, then processing of push meta requests throws NPE

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.4, 3.3.2, 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Shuffle
• Assignee: Chandni Singh
• Reporter: Chandni Singh
• Created: 19/May/23 02:48
• Updated: 22/May/23 20:48
• Resolved: 22/May/23 20:48

Description

After enabling support for over-the-wire encryption for spark shuffle services, the meta requests for push-merged blocks fail with this error:

java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.spark.network.server.AbstractAuthRpcHandler.getMergedBlockMetaReqHandler(AbstractAuthRpcHandler.java:110)
	at org.apache.spark.network.crypto.AuthRpcHandler.getMergedBlockMetaReqHandler(AuthRpcHandler.java:144)
	at org.apache.spark.network.server.TransportRequestHandler.processMergedBlockMetaRequest(TransportRequestHandler.java:275)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:117)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:140)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)
	at org.sparkproject.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at org.sparkproject.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) 

Issue Links

• relates to: SPARK-36744 Support IO encryption for push-based shuffle
• links to: [Github] Pull Request #41225 (otterc)