[METRON-682] Unify and Improve the Flat File Loader

• Type: Improvement
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.3.1
• Assignee: Casey Stella
• Reporter: Casey Stella
• Created: 01/Feb/17 16:33
• Updated: 28/Aug/19 20:23
• Resolved: 28/Aug/19 18:43

Description

Currently the flat file loader is deficient in a couple ways:

• It only supports importing local data despite there being a separate, poorly named, application which supports importing enrichment via MapReduce called threat_intel_loader.sh
• It does not support local imports from HDFS
• It does not support local imports from URLs
• It does not support importing zipped archives locally
• You cannot import more than one file at once

This JIRA will:

• Unify the MapReduce and local imports into one program and allow the user to specify the import mode with a CLI flag
• Support local imports from HDFS and URLs
• Support local imports from zipped files
• Support importing more than one file at once

Issue Links

• links to: GitHub Pull Request #432