Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14310

ORC schema evolution should not completely disable PPD

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.2.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Follow up for HIVE-14214 which completely shuts off PPD when there is any schema evolution. Some evolutions are safer for PPD like

      byte -> short -> int -> long
      float -> double (This is unsafe, see comments in SchemaEvolution.java in the patch)
      varchar <-> string (string to char, varchar to char and vice versa is also unsafe conversion as Orc stores internal index with padded spaces for char)

      For all other conversions we can disable PPD for that specific column that has evolved by returning TruthValue.YES_NO

        Attachments

        1. HIVE-14310.WIP.patch
          81 kB
          Prasanth Jayachandran
        2. HIVE-14310.1.patch
          85 kB
          Prasanth Jayachandran
        3. HIVE-14310.2.patch
          105 kB
          Prasanth Jayachandran
        4. HIVE-14310.4.patch
          104 kB
          Prasanth Jayachandran

          Issue Links

            Activity

              People

              • Assignee:
                prasanth_j Prasanth Jayachandran
                Reporter:
                prasanth_j Prasanth Jayachandran
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: