Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4565

MultiFileInputSplit can use data locality information to create splits

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.20.0
    • None
    • None
    • Reviewed
    • Improved MultiFileInputFormat so that multiple blocks from the same node or same rack can be combined into a single split.

    Description

      The MultiFileInputFormat takes a set of paths and creates splits based on file sizes. Each splits contains a few files an each split are roughly equal in size. It would be efficient if we can extend this InputFormat to create splits such each all the blocks in one split and either node-local or rack-local.

      Attachments

        1. CombineMultiFile.patch
          19 kB
          Dhruba Borthakur
        2. CombineMultiFile2.patch
          37 kB
          Dhruba Borthakur
        3. CombineMultiFile3.patch
          35 kB
          Dhruba Borthakur
        4. CombineMultiFile4.patch
          36 kB
          Dhruba Borthakur
        5. CombineMultiFile5.patch
          39 kB
          Dhruba Borthakur
        6. CombineMultiFile7.patch
          42 kB
          Dhruba Borthakur
        7. CombineMultiFile8.patch
          47 kB
          Dhruba Borthakur
        8. CombineMultiFile9.patch
          49 kB
          Dhruba Borthakur
        9. TestCombine.txt
          21 kB
          Dhruba Borthakur

        Issue Links

        blocks

        Improvement - An improvement or enhancement to an existing feature or task. HIVE-74 Hive can use CombineFileInputFormat for when the input are many small files

        • Major - Major loss of function.
        • Closed
        duplicates

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-2560 Processing multiple input splits per mapper task

        • Major - Major loss of function.
        • Resolved
        is blocked by

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-3293 When an input split spans cross block boundary, the split location should be the host having most of bytes on it.

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4567 GetFileBlockLocations should return the NetworkTopology information of the machines that hosts those blocks

        • Major - Major loss of function.
        • Closed
        is depended upon by

        New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-214 Concrete implementation of MultiFileInputFormat

        • Major - Major loss of function.
        • Open

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dhruba Dhruba Borthakur Assign to me
            dhruba Dhruba Borthakur
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment