[SPOT-43] [ML] Job count at InvalidDataHandler.scala:43 in SPOT-ML for Flow data is taking too long.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Labels: patch performance
• Environment: Cluster 9 nodes, 256 GB RAM each node, 48 virtual CPUs each node.; Running Spark with; 43 Executors, 30 GB memory each, 8 cores per executor.
• Flags: Patch
• Assignee: Unassigned
• Reporter: Ricardo Barona
• Created: 07/Feb/17 17:20
• Updated: 17/Feb/17 22:11
• Resolved: 17/Feb/17 22:11

Description

When running spot-ml, flow, for ~ 1TB data, there is a job (last job) taking 3.3 hours. After reviewing Spark UI, I noticed that this step count at InvalidDataHandler.scala:43 is actually performing the same join that was already executed in job map at FlowSuspiciousConnectsAnalysis.scala:42 (6.4 hours).
count at InvalidDataHandler.scala:43 is filtering corrupt records from scoredFlowRecords, and it seems to be being re-calculated.

Steps to reproduce:
Start spot-ml for flow data with a data set of 1TB or bigger.
Wait until it completes, look at the time last job took.

Issue Links

• links to: GitHub Pull Request #20