[METRON-2274] Flatfile loader and summarizer mapreduce mode broken

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Assignee: Michael Miklavcic
• Reporter: Michael Miklavcic
• Created: 03/Oct/19 21:28
• Updated: 04/Oct/19 19:21
• Resolved: 04/Oct/19 19:21

Description

Fix mapreduce mode for the loaders. This was broken in METRON-2149 in 2 places:

https://github.com/apache/metron/pull/1436/files#diff-1a76680a25447c0c0b3a245ce767b3f8R39

and

https://github.com/apache/metron/pull/1436/files#diff-8c698ee5b00cdf64f09bcd48847eae33R39

HADOOP_CLASSPATH only sets the classpath for the local client MR application, not for the JVM's running the map and reduce tasks. When Stellar was extracted from our uber jars, the metron-data-management jar no longer contained requisite classes from Stellar common. The correct way to handle adding dependencies for mapreduce jobs is via the "-libjars" option.

Issue Links

• is related to: METRON-2149 Shaded jar classifier is not consistent
• links to: GitHub Pull Request #1525