Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5478

TeraInputFormat unnecessarily defines its own FileSplit subclass

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: 2.1.1-beta
    • Component/s: examples
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TeraInputFormat defines its own TeraFileSplit subclass of FileSplit that adds a locations field, which is already included in FileSplit.

      This is causing MR2 TeraSort to fail on MR1, which, for a System.arraycopy, requires splits to be of the FileSplit class. While nobody is promising that everything that runs on MR2 should run on MR1, fixing this would be easy and make it possible to compare MR2 TeraSort performance between MR1 and MR2.

      We should just get rid of TeraFileSplit and use FileSplit.

        Attachments

        1. MAPREDUCE-5478.patch
          4 kB
          Sandy Ryza

          Activity

            People

            • Assignee:
              sandyr Sandy Ryza
              Reporter:
              sandyr Sandy Ryza
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: