[SPOT-236] [ML] Spot-ml cannot locate top-1m.csv when executing proxy

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: 1.0
• Labels: proxy spot-ml
• Environment: cloudera 5.12.1
• Assignee: Ricardo Barona
• Reporter: hubert dulay
• Created: 10/Oct/17 18:27
• Updated: 12/Oct/17 15:43
• Resolved: 12/Oct/17 15:43

Description

when executing ml_op.sh

./ml_ops.sh 20171010 proxy -1 500

I get error below.

[Stage 2:> (0 + 0) / 4]
[Stage 2:> (0 + 3) / 4]
[Stage 2:============================================> (3 + 0) / 4]
[Stage 2:============================================> (3 + 1) / 4]
17/10/10 18:21:52 INFO SuspiciousConnectsAnalysis: Read source data 17/10/10 18:21:52 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.ExceptionInInitializerError java.lang.ExceptionInInitializerError at org.apache.spot.proxy.ProxySuspiciousConnectsModel$.ipWordCountFromDF(ProxySuspiciousConnectsModel.scala:196) at org.apache.spot.proxy.ProxySuspiciousConnectsModel$.getIPWordCounts(ProxySuspiciousConnectsModel.scala:186) at org.apache.spot.proxy.ProxySuspiciousConnectsModel$.trainModel(ProxySuspiciousConnectsModel.scala:151) at org.apache.spot.proxy.ProxySuspiciousConnectsAnalysis$.run(ProxySuspiciousConnectsAnalysis.scala:106) at org.apache.spot.SuspiciousConnects$.main(SuspiciousConnects.scala:78) at org.apache.spot.SuspiciousConnects.main(SuspiciousConnects.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:645)

Caused by: java.io.FileNotFoundException: top-1m/top-1m.csv (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at scala.io.Source$.fromFile(Source.scala:91) at scala.io.Source$.fromFile(Source.scala:76) at scala.io.Source$.fromFile(Source.scala:54) at org.apache.spot.utilities.TopDomains$.<init>(TopDomains.scala:29) at org.apache.spot.utilities.TopDomains$.<clinit>(TopDomains.scala) ... 11 more