[CELEBORN-1668] Fix NullPointerException for PartitionDataWriter#getDiskFileInfo in PushDataHandler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0, 0.5.2
• Fix Version/s: 0.6.0, 0.5.2
• Labels: pull-request-available
• Assignee: Nicholas Jiang
• Reporter: Nicholas Jiang
• Created: 23/Oct/24 13:15
• Updated: 28/Oct/24 12:48
• Resolved: 24/Oct/24 09:52

Description

After https://issues.apache.org/jira/browse/CELEBORN-1133, PushDataHandler has NullPointerException for PartitionDataWriter#getDiskFileInfo, which exception is as follows:

24/10/22 18:09:18,214 ERROR [push-server-6-5] PushDataHandler: Error while handlePUSH_MERGED_DATA PushMergedData[requestId=6087,mode=0,shuffleKey=spark-57392a5e407d400ba80689647a19b20c-2,partitionIds=[482-0, 486-0, 490-0, 494-0, 498-0, 502-0],batchOffsets=[0, 12015, 24656, 37699, 51625, 64868],body size=78330]
java.lang.NullPointerException
	at org.apache.celeborn.service.deploy.worker.PushDataHandler.$anonfun$handlePushMergedData$10(PushDataHandler.scala:552)
	at org.apache.celeborn.common.internal.Logging.logWarning(Logging.scala:55)
	at org.apache.celeborn.common.internal.Logging.logWarning$(Logging.scala:54)
	at org.apache.celeborn.service.deploy.worker.PushDataHandler.logWarning(PushDataHandler.scala:50)
	at org.apache.celeborn.service.deploy.worker.PushDataHandler.handlePushMergedData(PushDataHandler.scala:552)
	at org.apache.celeborn.service.deploy.worker.PushDataHandler.$anonfun$receive$2(PushDataHandler.scala:151)
	at org.apache.celeborn.service.deploy.worker.PushDataHandler.handleCore(PushDataHandler.scala:771)
	at org.apache.celeborn.service.deploy.worker.PushDataHandler.receive(PushDataHandler.scala:152)
	at org.apache.celeborn.common.network.server.TransportRequestHandler.processOtherMessages(TransportRequestHandler.java:132)
	at org.apache.celeborn.common.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:88)
	at org.apache.celeborn.common.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:156)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at org.apache.celeborn.common.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:74)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)

Issue Links

• links to: GitHub Pull Request #2846