Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16483

HoS should populate split related configurations to HiveConf

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Spark
    • Labels:
      None

      Description

      There are several split related configurations, such as MAPREDMINSPLITSIZE, MAPREDMINSPLITSIZEPERNODE, MAPREDMINSPLITSIZEPERRACK, etc., that should be populated to HiveConf. Currently we only do this for MAPREDMINSPLITSIZE.
      All the others, if not set, will be using the default value, which is 1.

      Without these, Spark sometimes will not merge small files for file formats such as text.

        Attachments

          Activity

            People

            • Assignee:
              csun Chao Sun
              Reporter:
              csun Chao Sun
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: