Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14310

ORC schema evolution should not completely disable PPD

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.1.0, 2.2.0
    • 2.1.1, 2.2.0
    • None
    • None

    Description

      Follow up for HIVE-14214 which completely shuts off PPD when there is any schema evolution. Some evolutions are safer for PPD like

      byte -> short -> int -> long
      float -> double (This is unsafe, see comments in SchemaEvolution.java in the patch)
      varchar <-> string (string to char, varchar to char and vice versa is also unsafe conversion as Orc stores internal index with padded spaces for char)

      For all other conversions we can disable PPD for that specific column that has evolved by returning TruthValue.YES_NO

      Attachments

        1. HIVE-14310.WIP.patch
          81 kB
          Prasanth Jayachandran
        2. HIVE-14310.1.patch
          85 kB
          Prasanth Jayachandran
        3. HIVE-14310.2.patch
          105 kB
          Prasanth Jayachandran
        4. HIVE-14310.4.patch
          104 kB
          Prasanth Jayachandran

        Issue Links

          Activity

            People

              prasanth_j Prasanth Jayachandran
              prasanth_j Prasanth Jayachandran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: