Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6783

Incompatible schema for maps between parquet-hive and parquet-pig

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.14.0, 0.13.1
    • Component/s: File Formats
    • Labels:
      None

      Description

      see also in following parquet issue:
      https://github.com/Parquet/parquet-mr/issues/290

      The schema written for maps isn't compatible between hive and pig. This means any files written in one cannot be properly read in the other.

      More specifically, for the same map column c1, parquet-pig generates schema:

      message pig_schema {
      optional group c1 (MAP) {
      repeated group map (MAP_KEY_VALUE)

      { required binary key (UTF8); optional binary value; }


      }
      }

      while parquet-hive generates schema:
      message hive_schema {
      optional group c1 (MAP_KEY_VALUE) {
      repeated group map

      { required binary key; optional binary value; }

      }
      }

        Attachments

        1. HIVE-6783.4.patch.txt
          3 kB
          Tongjie Chen
        2. HIVE-6783.3.patch.txt
          3 kB
          Tongjie Chen
        3. HIVE-6783.2.patch.txt
          3 kB
          Tongjie Chen
        4. HIVE-6783.1.patch.txt
          3 kB
          Tongjie Chen

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tongjie Tongjie Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: