Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17754

HCat InputJobInfo in Pig UDFContext is heavyweight, and causes OOMs in Tez AMs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.2.0, 3.0.0
    • None
    • HCatalog
    • None

    Description

      HIVE-9845 dealt with reducing the size of HCat split-info, to improve job-launch times for Pig/HCat jobs.
      For large Pig queries that scan a large number of Hive partitions, it was found that the Pig UDFContext stored full-fat HCat InputJobInfo objects, thus blowing out the Pig Tez AM. Since this information is already stored in the HCatSplit, the serialization of InputJobInfo can be spared.

      Attachments

        1. HIVE-17754.1.patch
          66 kB
          Mithun Radhakrishnan
        2. HIVE-17754.2.patch
          67 kB
          Mithun Radhakrishnan
        3. HIVE-17754.2-branch-2.patch
          68 kB
          Mithun Radhakrishnan

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-17763 HCatLoader should fetch delegation tokens for partitions on remote HDFS

          • Major - Major loss of function.
          • Patch Available
          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. HIVE-11548 HCatLoader should support predicate pushdown.

          • Major - Major loss of function.
          • Patch Available

          Activity

            People

              mithun Mithun Radhakrishnan
              mithun Mithun Radhakrishnan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: