Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2046

A CombineFileInputSplit cannot be less than a dfs block

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.22.0
    • None
    • None

    Description

      I ran into this while testing some hive features.

      Whether we use hiveinputformat or combinehiveinputformat, a split cannot be less than a dfs block size.
      This is a problem if we want to increase the block size for older data to reduce memory consumption for the
      name node.

      It would be useful if the input split was independent of the dfs block size.

      Attachments

        1. combineFileInputFormatMaxSize.txt
          7 kB
          Dhruba Borthakur
        2. combineFileInputFormatMaxSize2.txt
          7 kB
          Dhruba Borthakur
        3. patch-2046-ydist.txt
          6 kB
          Amareshwari Sriramadasu

        Activity

          People

            dhruba Dhruba Borthakur
            namit Namit Jain
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: