[HDDS-10780] NullPointerException in watchForCommit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0, 1.4.1
• Component/s: Ozone Client
• Labels: pull-request-available
• Target Version/s: 2.0.0, 1.4.1
• Assignee: Duong
• Reporter: Duong
• Created: 30/Apr/24 17:11
• Updated: 25/Oct/24 14:00
• Resolved: 03/May/24 16:10

Description

NPE happens during watchForCommit. In updateCommitInfosMap, when there's a new Datanode ID appears in the commitInfoProtos, we'll see a NPE.

java.lang.NullPointerException
        at java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:229)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:479)
        at java.base/java.util.stream.LongPipeline.min(LongPipeline.java:437)
        at org.apache.hadoop.hdds.scm.XceiverClientRatis.updateCommitInfosMap(XceiverClientRatis.java:154)
        at java.base/java.util.Optional.map(Optional.java:265)
        at org.apache.hadoop.hdds.scm.XceiverClientRatis.updateCommitInfosMap(XceiverClientRatis.java:133)
        at org.apache.hadoop.hdds.scm.XceiverClientRatis.watchForCommit(XceiverClientRatis.java:280)
        at org.apache.hadoop.hdds.scm.storage.AbstractCommitWatcher.watchForCommit(AbstractCommitWatcher.java:142)
        at org.apache.hadoop.hdds.scm.storage.AbstractCommitWatcher.watchOnFirstIndex(AbstractCommitWatcher.java:104)
        at org.apache.hadoop.hdds.scm.storage.RatisBlockOutputStream.sendWatchForCommit(RatisBlockOutputStream.java:106)
        at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.watchForCommit(BlockOutputStream.java:418)
        at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.waitForFlushAndCommit(BlockOutputStream.java:389)
        at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFullBuffer(BlockOutputStream.java:376)
        at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.doFlushOrWatchIfNeeded(BlockOutputStream.java:310)
        at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:294)
        at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:137)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.writeToOutputStream(KeyOutputStream.java:250)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:228)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:208)
        at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:94)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1484)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1107)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1456)
        at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:328)
        at jdk.internal.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:359) 

Issue Links

• relates to: HDDS-10700 [Hbase Ozone] Both HMaster down with "BlockOutputStream: Failed to commit BlockId"
• links to: GitHub Pull Request #6627