[SPOT-183] [ML] Schema validation for input data

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Implemented
• Labels: spot-ml
• Environment: All environments
• Assignee: Ricardo Barona
• Reporter: Ricardo Barona
• Created: 22/Jun/17 20:13
• Updated: 11/Sep/19 00:30
• Resolved: 27/Jul/17 18:15

Description

Currently, spot-ml assumes input data will follow spot-ingest schema however users can create their own data sets using different ingest methods. For that, spot-ml should be able to validate that the input schema is good for the ML processing so that we avoid further errors or bad results.

Issue Links

• is a parent of: SPOT-174 [ML] Error while running ML for Proxy data
• is a parent of: SPOT-149 [ML] obtaining parquet file for different modes
• is related to: SPOT-187 [Documentation] As an Apache Spot user I want to have centralized documentation about NetFlow schema so that I can integrate/implement specific Apache Spot modules
• is related to: SPOT-188 [Documentation] As an Apache Spot user I want to have centralized documentation about DNS schema so that I can integrate/implement specific Apache Spot modules
• relates to: SPOT-202 Spot ML Failed with error "next on empty iterator"
• links to: GitHub Pull Request #72