Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16483

HoS should populate split related configurations to HiveConf

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Spark
    • Labels:
      None

      Description

      There are several split related configurations, such as MAPREDMINSPLITSIZE, MAPREDMINSPLITSIZEPERNODE, MAPREDMINSPLITSIZEPERRACK, etc., that should be populated to HiveConf. Currently we only do this for MAPREDMINSPLITSIZE.
      All the others, if not set, will be using the default value, which is 1.

      Without these, Spark sometimes will not merge small files for file formats such as text.

        Attachments

        1. HIVE-16483.1.patch
          2 kB
          Chao Sun

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment