Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2046

A CombineFileInputSplit cannot be less than a dfs block

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None

      Description

      I ran into this while testing some hive features.

      Whether we use hiveinputformat or combinehiveinputformat, a split cannot be less than a dfs block size.
      This is a problem if we want to increase the block size for older data to reduce memory consumption for the
      name node.

      It would be useful if the input split was independent of the dfs block size.

      1. combineFileInputFormatMaxSize.txt
        7 kB
        dhruba borthakur
      2. combineFileInputFormatMaxSize2.txt
        7 kB
        dhruba borthakur
      3. patch-2046-ydist.txt
        6 kB
        Amareshwari Sriramadasu

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            Namit Jain
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development