Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-2274

Flatfile loader and summarizer mapreduce mode broken

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Fix mapreduce mode for the loaders. This was broken in METRON-2149 in 2 places:

      https://github.com/apache/metron/pull/1436/files#diff-1a76680a25447c0c0b3a245ce767b3f8R39

      and

      https://github.com/apache/metron/pull/1436/files#diff-8c698ee5b00cdf64f09bcd48847eae33R39

      HADOOP_CLASSPATH only sets the classpath for the local client MR application, not for the JVM's running the map and reduce tasks. When Stellar was extracted from our uber jars, the metron-data-management jar no longer contained requisite classes from Stellar common. The correct way to handle adding dependencies for mapreduce jobs is via the "-libjars" option.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mmiklavcic Michael Miklavcic
                Reporter:
                mmiklavcic Michael Miklavcic
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m