Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-15196

Null seems to be an unexpected result for ScaleCubeMessagingService#invoke() on node stopping

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha2
    • 3.0.0-alpha3
    • None

    Description

      In case of node stop it's possible to get null as a result of an invoke operation.

      return cluster
          .requestResponse(fromNetworkAddress(addr), message)
          .timeout(Duration.ofMillis(timeout))
          .toFuture()
          .thenApply(m -> m == null ? null : m.data()); // The result can be null on node stopping.
      

      It's a rather confusing behavior that produces NPE. See IGNITE-15195 for more details. In order not not fail with NPE I've updated null handling in a following way

      return cluster
          .requestResponse(fromNetworkAddress(addr), message)
          .timeout(Duration.ofMillis(timeout))
          .toFuture()
          .thenApply(m -> {
              if (m == null)
                  throw new CompletionException(new NodeStoppingCheckedException());
              else
                  return m.data();
          }); // The result can be null on node stopping.

      within IGNITE-15148

      In order to reproduce given issue restore original

      .thenApply(m -> m == null ? null : m.data());
      

      and run

       org.apache.ignite.internal.runner.app.IgnitionTest#testNodeStartWithoutBootstrapConfiguration

      several times.

      As a result NPE will be logged on node stop:

      SEVERE: Unexpected exception
      class org.apache.ignite.lang.IgniteInternalException: java.lang.NullPointerException
      	at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:109)
      	at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:446)
      Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
      	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
      	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
      	at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:101)
      	... 1 more
      Caused by: java.lang.NullPointerException
      	at org.apache.ignite.raft.client.service.impl.RaftGroupServiceImpl.lambda$run$15(RaftGroupServiceImpl.java:323)
      	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
      	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
      	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
      	at org.apache.ignite.raft.client.service.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:406)
      	at org.apache.ignite.raft.client.service.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:360)
      	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
      	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
      	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
      	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
      	at reactor.core.publisher.MonoToCompletableFuture.onComplete(MonoToCompletableFuture.java:83)
      	at reactor.core.publisher.SerializedSubscriber.onComplete(SerializedSubscriber.java:146)
      	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onComplete(FluxTimeout.java:228)
      	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:129)
      	at reactor.core.publisher.LambdaSubscriber.onComplete(LambdaSubscriber.java:132)
      	at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onComplete(FluxTake.java:423)
      	at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onComplete(FluxFilterFuseable.java:165)
      	at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onComplete(FluxFilterFuseable.java:384)
      	at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.checkTerminated(FluxOnBackpressureBuffer.java:416)
      	at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:253)
      	at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225)
      	at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onComplete(FluxOnBackpressureBuffer.java:204)
      	at reactor.core.publisher.DirectProcessor$DirectInner.onComplete(DirectProcessor.java:349)
      	at reactor.core.publisher.DirectProcessor.onComplete(DirectProcessor.java:166)
      	at reactor.core.publisher.FluxCreate$BaseSink.complete(FluxCreate.java:438)
      	at reactor.core.publisher.FluxCreate$SerializedSink.drainLoop(FluxCreate.java:239)
      	at reactor.core.publisher.FluxCreate$SerializedSink.drain(FluxCreate.java:205)
      	at reactor.core.publisher.FluxCreate$SerializedSink.complete(FluxCreate.java:196)
      	at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.lambda$doStop$2(ScaleCubeDirectMarshallerTransport.java:134)
      	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
      	at reactor.core.publisher.Mono.subscribe(Mono.java:4219)
      	at reactor.core.publisher.FluxConcatArray$ConcatArrayDelayErrorSubscriber.onComplete(FluxConcatArray.java:303)
      	at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:81)
      	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
      	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
      	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onComplete(FluxPeekFuseable.java:931)
      	at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.onComplete(MonoSubscribeOn.java:160)
      	at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:81)
      	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
      	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1783)
      	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
      	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
      	at io.scalecube.cluster.gossip.GossipProtocolImpl.doSpreadGossip(GossipProtocolImpl.java:177)
      	at reactor.core.scheduler.PeriodicSchedulerTask.call(PeriodicSchedulerTask.java:49)
      	at reactor.core.scheduler.PeriodicSchedulerTask.run(PeriodicSchedulerTask.java:63)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
      	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      
      

      Attachments

        Issue Links

          Activity

            People

              apolovtcev Aleksandr Polovtsev
              alapin Alexander Lapin
              Semyon Danilov Semyon Danilov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m