[SPOT-130] [ML] ml_ops.sh does not complete succesfully

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Labels: spot-ml
• Environment: RHEL7 ; CDH5.10.0
• Assignee: Ricardo Barona
• Reporter: Daniel Oakley
• Created: 20/Mar/17 23:47
• Updated: 07/Apr/17 17:17
• Resolved: 07/Apr/17 17:17

Description

After running ml_ops.sh, it hangs on "starting remoting" for a while, then this:

17/03/20 19:38:18 INFO Remoting: Starting remoting
17/03/20 19:38:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.1.0.252:43128]
17/03/20 19:38:18 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.1.0.252:43128]
17/03/20 19:42:42 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

I have ingested data:

1. hdfs dfs -ls -R /user/spot/flow/hive
   drwxr-xr-x - spot supergroup 0 2017-03-13 21:46 /user/spot/flow/hive/y=2017
   drwxr-xr-x - spot supergroup 0 2017-03-13 21:46 /user/spot/flow/hive/y=2017/m=03
   drwxr-xr-x - spot supergroup 0 2017-03-13 23:14 /user/spot/flow/hive/y=2017/m=03/d=14
   drwxr-xr-x - spot supergroup 0 2017-03-13 21:46 /user/spot/flow/hive/y=2017/m=03/d=14/h=01
   -rwxr-xr-x 3 spot supergroup 441 2017-03-13 21:46 /user/spot/flow/hive/y=2017/m=03/d=14/h=01/000000_0
   drwxr-xr-x - spot supergroup 0 2017-03-13 22:32 /user/spot/flow/hive/y=2017/m=03/d=14/h=02

These are the options spark saw when starting:

Main class:
org.apache.spot.SuspiciousConnects
Arguments:
--analysis
flow
--input
/user/spot/flow/hive/y=2017/m=03/d=14/
--dupfactor
1000
--feedback
/data/spot/ml/flow/20170314/flow_scores.csv
--ldatopiccount
20
--scored
/user/spot/flow/scored_results/20170314/scores
--threshold
1
--maxresults
2000
--ldamaxiterations
20