Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6262

Remove unnecessary copies of schema + table desc from serialized plan

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently for a partitioned table the following are true:

      • for each partitiondesc we send a copy of the corresponding tabledesc
      • for each partitiondesc we send two copies of the schema (in different formats).

      Obviously we need to send different schemas if they are required by schema evolution, but in our case we'll always end up with multiple copies.

      The effect can be dramatic. The reductions by removing those on partitioned tables easily be can be 8-10x in size. Plans themselves can be 10s to 100s of mb (even with kryo). The size difference also plays out in every task on the cluster we run.

        Attachments

        1. HIVE-6262.1.patch
          294 kB
          Gunther Hagleitner

          Activity

            People

            • Assignee:
              hagleitn Gunther Hagleitner
              Reporter:
              hagleitn Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: