[METRON-1325] Cannot submit data to the dockerized elasticsearch

• Type: Bug
• Status: Done
• Priority: Critical
• Resolution: Done
• Affects Version/s: 0.4.1
• Fix Version/s: 0.4.1
• Labels: patch
• Environment: Ubuntu 14.04 , Docker container, 16 GB Memory, 400GB Hard Drive
• Flags: Patch, Important
• Assignee: Unassigned
• Reporter: Barbara
• Created: 21/Nov/17 12:43
• Updated: 24/Apr/19 02:09
• Resolved: 24/Apr/19 02:09

Description

I am currently running the examples on the following url:

+https://github.com/apache/metron/tree/master/metron-contrib/metron-docker +

Everything seems to work until i get to the last entry used in indexing data into elasticsearch:

docker-compose exec storm ./bin/start_elasticsearch_topology.sh

which gives me the following error:

Error:

:/opt/apache-metron-0.4.1-rc4/metron-contrib/metron-docker/compose# docker-compose exec storm ./bin/start_elasticsearch_topology.sh
WARNING: The DOCKER_HOST variable is not set. Defaulting to a blank string.
Running: java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/share/apache-storm -Dstorm.log.dir=/usr/share/apache-storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/share/apache-storm/lib/minlog-1.3.0.jar:/usr/share/apache-storm/lib/disruptor-3.3.2.jar:/usr/share/apache-storm/lib/slf4j-api-1.7.7.jar:/usr/share/apache-storm/lib/servlet-api-2.5.jar:/usr/share/apache-storm/lib/kryo-3.0.3.jar:/usr/share/apache-storm/lib/log4j-core-2.1.jar:/usr/share/apache-storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/share/apache-storm/lib/asm-5.0.3.jar:/usr/share/apache-storm/lib/clojure-1.7.0.jar:/usr/share/apache-storm/lib/storm-rename-hack-1.0.1.jar:/usr/share/apache-storm/lib/objenesis-2.1.jar:/usr/share/apache-storm/lib/log4j-slf4j-impl-2.1.jar:/usr/share/apache-storm/lib/log4j-api-2.1.jar:/usr/share/apache-storm/lib/reflectasm-1.10.1.jar:/usr/share/apache-storm/lib/storm-core-1.0.1.jar:/usr/metron/0.4.1/lib/metron-elasticsearch-0.4.1-uber.jar:/usr/share/apache-storm/conf:/usr/share/apache-storm/bin -Dstorm.jar=/usr/metron/0.4.1/lib/metron-elasticsearch-0.4.1-uber.jar org.apache.storm.flux.Flux --remote /usr/metron/0.4.1/flux/indexing/remote.yaml --filter /usr/metron/0.4.1/config/elasticsearch.properties
??????????? ??? ?????? ???
??????????? ??? ???????????
?????? ??? ??? ??? ??????
?????? ??? ??? ??? ??????
??? ????????????????????? ???
??? ???????? ??????? ??? ???
- Apache Storm -
- data FLow User eXperience -
Version: 1.0.1
Parsing file: /usr/metron/0.4.1/flux/indexing/remote.yaml
404 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
407 [main] INFO o.a.s.f.p.FluxParser - Performing property substitution.
412 [main] INFO o.a.s.f.p.FluxParser - Not performing environment variable substitution.
516 [main] INFO o.a.c.f.i.CuratorFrameworkImpl - Starting
593 [main-EventThread] INFO o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
1771 [main] INFO o.a.s.f.FluxBuilder - Detected DSL topology...
1898 [main] INFO o.a.m.w.b.BulkMessageWriterBolt - Requesting topology.tick.tuple.freq.secs set to 14
1942 [main] INFO o.a.m.w.b.BulkMessageWriterBolt - Requesting topology.tick.tuple.freq.secs set to 14
1960 [main] INFO o.a.m.w.b.BulkMessageWriterBolt - Requesting topology.tick.tuple.freq.secs set to 14
---------- TOPOLOGY DETAILS ----------
Topology Name: indexing
--------------- SPOUTS ---------------
kafkaSpout [1] (org.apache.metron.storm.kafka.flux.StormKafkaSpout)
---------------- BOLTS ---------------
indexingBolt [1] (org.apache.metron.writer.bolt.BulkMessageWriterBolt)
hdfsIndexingBolt [1] (org.apache.metron.writer.bolt.BulkMessageWriterBolt)
indexingErrorBolt [1] (org.apache.metron.writer.bolt.BulkMessageWriterBolt)
--------------- STREAMS ---------------
kafkaSpout -SHUFFLE-> indexingBolt
kafkaSpout -SHUFFLE-> hdfsIndexingBolt
indexingBolt -SHUFFLE-> indexingErrorBolt
hdfsIndexingBolt -SHUFFLE-> indexingErrorBolt
--------------------------------------
2025 [main] INFO o.a.s.f.Flux - Running remotely...
2025 [main] INFO o.a.s.f.Flux - Deploying topology in an ACTIVE state...
2063 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -5563484848678206372:-8295574411886467738
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException:
at org.apache.storm.security.auth.AuthUtils.GetAutoCredentials(AuthUtils.java:211)
at org.apache.storm.StormSubmitter.populateCredentials(StormSubmitter.java:91)
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:210)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:288)
at org.apache.storm.flux.Flux.runCli(Flux.java:171)
at org.apache.storm.flux.Flux.main(Flux.java:98)
Caused by: java.lang.ClassNotFoundException:
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.storm.security.auth.AuthUtils.GetAutoCredentials(AuthUtils.java:203)
... 5 more

Issue Links

• links to: GitHub Pull Request #6