[METRON-1177] Stale running topologies seen post-kerberization and cause exceptions

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Affects Version/s: 0.4.1
• Fix Version/s: 0.4.2
• Assignee: Nick Allen
• Reporter: Anand Subramanian
• Created: 11/Sep/17 17:42
• Updated: 15/Dec/17 17:18
• Resolved: 31/Oct/17 11:04

Description

Steps to Reproduce

1. Install Ambari on 12 node cluster
2. Install Metron using MPack
3. Metron topologies are now running as user 'storm'
4. Setup standalone KDC on Ambari node by following these steps
5. Initiate "kerberize" cluster using Ambari
6. Ambari triggers "Stop all services"
7. In the event of a race condition, the REST service shuts down (which prevents from being able to shutdown metron topologies)
8. Metron topology shut down occurs, but Metron Mpack cannot stop topologies because REST already stopped. Metron ignores this condition and continues on assuming the topologies are down
9. Storm service shuts down
10. Cluster kerberization occurs
11. Ambari triggers "Start all services"
12. Storm service starts.
13. Since the Metron topologies were running when Storm was shutdown, these same topologies are attempted to be started again by Storm, but are declared as "already running"
14. All other services start successfully
15. Now, if we attempt to run a test with the stale topologies, following exceptions are seen:

AuthorizationException(msg:getTopologyInfo on topology snort is not authorized)
        at org.apache.storm.generated.Nimbus$getTopologyInfo_result$getTopologyInfo_resultStandardScheme.read(Nimbus.java:36971)
        at org.apache.storm.generated.Nimbus$getTopologyInfo_result$getTopologyInfo_resultStandardScheme.read(Nimbus.java:36939)
        at org.apache.storm.generated.Nimbus$getTopologyInfo_result.read(Nimbus.java:36870)
        at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
        at org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
        at org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
        at org.hw.metron.lib.storm.StormClient.isTopologyRunning(StormClient.java:150)
        at org.hw.metron.lib.storm.StormClient.startTopology(StormClient.java:68)
        at functional.SnortTopologyTest.snortElasticsearchIndexTest(SnortTopologyTest.java:36)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
        at org.testng.internal.Invoker.invokeMethod(Invoker.java:643)
        at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
        at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
        at org.testng.TestRunner.privateRun(TestRunner.java:782)
        at org.testng.TestRunner.run(TestRunner.java:632)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
        at org.testng.SuiteRunner.run(SuiteRunner.java:268)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
        at org.testng.TestNG.run(TestNG.java:1064)
        at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:295)
        at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:84)
        at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:90)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

Issue Links

• is duplicated by: METRON-1093 Start/stop/restart topologies use Storm CLI
• links to: GitHub Pull Request #748