Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3065

Very frequent test failures in storm-server

    XMLWordPrintableJSON

    Details

      Description

      I'm seeing the following intermittent test failures in storm-server when I run locally

      2018-05-09 16:32:23.377 [main] ERROR org.apache.storm.blobstore.KeySequenceNumber - Exception {}
      java.lang.NullPointerException: null
      	at java.lang.String.contains(String.java:2133) ~[?:1.8.0_152]
      	at org.apache.storm.blobstore.KeySequenceNumber.checkIfStateContainsCurrentNimbusHost(KeySequenceNumber.java:206) ~[classes/:?]
      	at org.apache.storm.blobstore.KeySequenceNumber.getKeySequenceNumber(KeySequenceNumber.java:159) [classes/:?]
      	at org.apache.storm.daemon.nimbus.Nimbus.getVersionForKey(Nimbus.java:655) [classes/:?]
      	at org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:223) [classes/:?]
      	at org.apache.storm.blobstore.LocalFsBlobStore$MockitoMock$1067706995.createBlob$accessor$Ub7aO1Cr(Unknown Source) [classes/:?]
      	at org.apache.storm.blobstore.LocalFsBlobStore$MockitoMock$1067706995$auxiliary$x0GVZISq.call(Unknown Source) [classes/:?]
      	at org.mockito.internal.invocation.RealMethod$FromCallable.invoke(RealMethod.java:48) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.creation.bytebuddy.InterceptedInvocation.callRealMethod(InterceptedInvocation.java:129) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:43) [mockito-core-2.10.0.jar:?]
      	at org.mockito.Answers.answer(Answers.java:100) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:97) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:35) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:65) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:51) [mockito-core-2.10.0.jar:?]
      	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptSuperCallable(MockMethodInterceptor.java:135) [mockito-core-2.10.0.jar:?]
      	at org.apache.storm.blobstore.LocalFsBlobStore$MockitoMock$1067706995.createBlob(Unknown Source) [classes/:?]
      	at org.apache.storm.blobstore.LocalFsBlobStoreTest.testBasic(LocalFsBlobStoreTest.java:325) [test-classes/:?]
      	at org.apache.storm.blobstore.LocalFsBlobStoreTest.testBasicLocalFs(LocalFsBlobStoreTest.java:114) [test-classes/:?]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_152]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_152]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_152]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_152]
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) [junit-4.11.jar:?]
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.11.jar:?]
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) [junit-4.11.jar:?]
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.11.jar:?]
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.11.jar:?]
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) [junit-4.11.jar:?]
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) [junit-4.11.jar:?]
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309) [junit-4.11.jar:?]
      	at org.junit.runners.Suite.runChild(Suite.java:127) [junit-4.11.jar:?]
      	at org.junit.runners.Suite.runChild(Suite.java:26) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) [junit-4.11.jar:?]
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309) [junit-4.11.jar:?]
      	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) [surefire-junit47-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) [surefire-junit47-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119) [surefire-junit47-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87) [surefire-junit47-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) [surefire-junit47-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161) [surefire-junit47-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1]
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1]
      

      and

      java.lang.NullPointerException
      	at java.lang.String.startsWith(String.java:1405)
      	at java.lang.String.startsWith(String.java:1434)
      	at org.apache.storm.zookeeper.ClientZookeeper.deleteNodeBlobstore(ClientZookeeper.java:86)
      	at org.apache.storm.cluster.ZKStateStorage.delete_node_blobstore(ZKStateStorage.java:93)
      	at org.apache.storm.cluster.StormClusterStateImpl.setupBlob(StormClusterStateImpl.java:704)
      	at org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:223)
      	at org.apache.storm.blobstore.LocalFsBlobStoreTest.testBasic(LocalFsBlobStoreTest.java:325)
      	at org.apache.storm.blobstore.LocalFsBlobStoreTest.testBasicLocalFs(LocalFsBlobStoreTest.java:114)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.junit.runners.Suite.runChild(Suite.java:127)
      	at org.junit.runners.Suite.runChild(Suite.java:26)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
      	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161)
      	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
      

      This seems to be because the LocalFsBlobStoreTest is mocking NimbusInfo, and the properties on that object aren't stubbed. Occasionally the test will try to e.g. do a string contains with the null value from the stub, which throws an NPE. Since NimbusInfo is just a POJO, I don't see why we would need to mock it, we could just supply dummy values.

      I'm also seeing test failures on nearly every run locally, seemingly due to interference between the tests. Example:

      testLocalTransport(org.apache.storm.MessagingTest)  Time elapsed: 8.197 sec  <<< ERROR!
      java.lang.RuntimeException: java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?
              at org.apache.storm.daemon.nimbus.Nimbus.submitTopologyWithOpts(Nimbus.java:2952)
              at org.apache.storm.daemon.nimbus.Nimbus.submitTopology(Nimbus.java:2761)
              at org.apache.storm.LocalCluster.submitTopology(LocalCluster.java:378)
              at org.apache.storm.LocalCluster.submitTopology(LocalCluster.java:121)
              at org.apache.storm.Testing.completeTopology(Testing.java:424)
              at org.apache.storm.MessagingTest.testLocalTransport(MessagingTest.java:57)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
              at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
              at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
              at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
              at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
              at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
              at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
              at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
              at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
              at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
              at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
              at org.junit.runners.Suite.runChild(Suite.java:127)
              at org.junit.runners.Suite.runChild(Suite.java:26)
              at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
              at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
              at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
              at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
              at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
              at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
              at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
              at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
              at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119)
              at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87)
              at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
              at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161)
              at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
              at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
              at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
      Caused by: java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?
              at org.apache.storm.zookeeper.Zookeeper.toNimbusInfo(Zookeeper.java:109)
              at org.apache.storm.zookeeper.LeaderElectorImp.getLeader(LeaderElectorImp.java:108)
              at org.apache.storm.daemon.nimbus.Nimbus.assertIsLeader(Nimbus.java:1301)
              at org.apache.storm.daemon.nimbus.Nimbus.submitTopologyWithOpts(Nimbus.java:2798)
              ... 39 more
      

      as well as occasional connection losses to Zookeeper. Reducing storm-server to run one test at a time, rather than with a forkCount of 1 per core eliminates these issues.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srdo Stig Rohde Døssing
                Reporter:
                srdo Stig Rohde Døssing
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h