Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-1045

[Metron enrichment] Cannot restart service - enrichment table already exists

    Details

    • Type: Bug
    • Status: To Do
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Environment:
      CentOS 7
      Metron 0.4.0
      Ambari 2.5.1.10

      Description

      Hi,

      This is a new install on Ambari, downloaded from the hortonworks repo if I remember correctly. Although I have been able to start Metron once, for any subsequent launches I get an error complaining the table 'enrichment' already exists in the hdfs store. If I delete the table from the command line and try again, then the startup process runs properly. Why is this script assuming that the database is clean before it starts?

      Although this is potentially unrelated, the Metron REST API module doesn't stay running for very long after I start it, which means I am unable to log into the management interface.

      Any assistance would be much appreciated.

      Execution of '/usr/metron/0.4.0/bin/start_enrichment_topology.sh -s enrichment -z metron:2181,hadoop-slave-1:2181,hadoop-master:2181' returned 1. Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer /usr/metron/0.4.0/lib/metron-enrichment-0.4.0-uber.jar /tmp/b70910e6688d11e78398000c2953f5b1.jar
      Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/tmp/b70910e6688d11e78398000c2953f5b1.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.1.0-129/storm/bin -Dstorm.jar=/tmp/b70910e6688d11e78398000c2953f5b1.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.flux.Flux --remote /usr/metron/0.4.0/flux/enrichment/remote.yaml --filter /usr/metron/0.4.0/config/enrichment.properties

      <FLUX startup stuff>

      Exception in thread "main" java.lang.RuntimeException: Topology with name `enrichment` already exists on cluster
      at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:240)
      at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)
      at org.apache.storm.flux.Flux.runCli(Flux.java:171)
      at org.apache.storm.flux.Flux.main(Flux.java:98)

        Activity

        Hide
        nickwallen Nick Allen added a comment -

        > Although I have been able to start Metron once, for any subsequent launches I get an error complaining the table 'enrichment' already exists in the hdfs store

        How exactly are you launching? What command are you running? What button are you clicking?

        Show
        nickwallen Nick Allen added a comment - > Although I have been able to start Metron once, for any subsequent launches I get an error complaining the table 'enrichment' already exists in the hdfs store How exactly are you launching? What command are you running? What button are you clicking?
        Hide
        rphi Rob Phipps added a comment - - edited

        Hi Nick,

        I am using the option within the Ambari web UI to start the Metron role? Either the role on its own (the rest of the cluster starts successfully) or as a whole cluster restart. It seems Ambari tries to install everything on every start which is odd. If it would be better to log an issue with the Ambari team, I can do that, however I believe the Metron plugin for Ambari was from this repo.

        Many thanks.
        Rob

        Show
        rphi Rob Phipps added a comment - - edited Hi Nick, I am using the option within the Ambari web UI to start the Metron role? Either the role on its own (the rest of the cluster starts successfully) or as a whole cluster restart. It seems Ambari tries to install everything on every start which is odd. If it would be better to log an issue with the Ambari team, I can do that, however I believe the Metron plugin for Ambari was from this repo. Many thanks. Rob
        Hide
        rphi Rob Phipps added a comment - - edited

        I literally have no idea what has happened, but I left the cluster running over the weekend and now it appears happy to restart of its own accord without any nasty database hacks.

        Never mind

        closes ticket

        Show
        rphi Rob Phipps added a comment - - edited I literally have no idea what has happened, but I left the cluster running over the weekend and now it appears happy to restart of its own accord without any nasty database hacks. Never mind closes ticket
        Hide
        rphi Rob Phipps added a comment -

        I have just reopened this, as I have experienced the issue again in the Ambari web interface. It appears that although this specific task fails, the service does manage to start (eg adding the topic fails, but that doesn't stop the service from just using the one that is already there) - however Ambari still recognisises this as a failed job which then stops any dependent services from being started. So it looks like a problem with the Ambari plugin as opposed to the enrichment module itself.

        Show
        rphi Rob Phipps added a comment - I have just reopened this, as I have experienced the issue again in the Ambari web interface. It appears that although this specific task fails, the service does manage to start (eg adding the topic fails, but that doesn't stop the service from just using the one that is already there) - however Ambari still recognisises this as a failed job which then stops any dependent services from being started. So it looks like a problem with the Ambari plugin as opposed to the enrichment module itself.

          People

          • Assignee:
            Unassigned
            Reporter:
            rphi Rob Phipps
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development