[SPOT-128] [ML] Make every pipeline to return results to main program to invoke a single save to HDFS method

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 1.0
• Labels: ML spot-ml spot-release
• Epic Link: Apache Spot (Incubating) Release 1.0
• Assignee: Ricardo Barona
• Reporter: Ricardo Barona
• Due: 22/Mar/17
• Created: 15/Mar/17 23:28
• Updated: 12/Jul/17 22:46
• Resolved: 12/Jun/17 16:58

Description

Currently, every pipeline invokes dataFrame.write.mode.parquet to save results in HDFS. To all the *SuspiciousConnectsAnalysis.scala free of that method call, we can return a couple of data frames to SuspiciousConnects.scala and from there save results.
With that, we are going to be able to write unit tests for *SuspiciousConnectsAnalysis.run method.

Issue Links

• links to: GitHub Pull Request #46
• links to: GitHub Pull Request #47