Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17781

Map MR settings to Tez settings via DeprecatedKeys

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.2.1, 2.4.0, 3.0.0
    • Component/s: Configuration, Tez
    • Labels:
      None
    • External issue ID:
      YHIVE-883

      Description

      Here's one that Chris Drome and Thiruvel Thirumoolan worked on:

      We found that certain Hadoop Map/Reduce settings that are set in site config files do not take effect in Hive jobs, because the Tez site configs do not contain the same settings.

      In Yahoo's case, the problem was that, at the time, there was no mapping between MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART and TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION. There were situations where significant capacity on production clusters were being used up doing nothing, while waiting for slow tasks to complete. This would have been avoided, were the mappings in place.

      Tez provides a DeprecatedKeys utility class, to help map MR settings to Tez settings. Hive should use this to ensure that the mappings are in sync.

      (Note to self: YHIVE-883)

        Attachments

        1. HIVE-17781.2-branch-2.2.patch
          6 kB
          Mithun Radhakrishnan
        2. HIVE-17781.2-branch-2.patch
          6 kB
          Mithun Radhakrishnan
        3. HIVE-17781.2.patch
          6 kB
          Mithun Radhakrishnan
        4. HIVE-17781.1.patch
          6 kB
          Mithun Radhakrishnan

          Activity

            People

            • Assignee:
              cdrome Chris Drome
              Reporter:
              mithun Mithun Radhakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: