Hive
  1. Hive
  2. HIVE-6262

Remove unnecessary copies of schema + table desc from serialized plan

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently for a partitioned table the following are true:

      • for each partitiondesc we send a copy of the corresponding tabledesc
      • for each partitiondesc we send two copies of the schema (in different formats).

      Obviously we need to send different schemas if they are required by schema evolution, but in our case we'll always end up with multiple copies.

      The effect can be dramatic. The reductions by removing those on partitioned tables easily be can be 8-10x in size. Plans themselves can be 10s to 100s of mb (even with kryo). The size difference also plays out in every task on the cluster we run.

      1. HIVE-6262.1.patch
        294 kB
        Gunther Hagleitner

        Activity

        Gunther Hagleitner created issue -
        Gunther Hagleitner made changes -
        Field Original Value New Value
        Attachment HIVE-6262.1.patch [ 12624296 ]
        Gunther Hagleitner made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Gunther Hagleitner made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.13.0 [ 12324986 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Gunther Hagleitner
            Reporter:
            Gunther Hagleitner
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development