Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23455

Improve error message for external orc table

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Since there is no schema validation for external tables, users may face various errors if their orc data and external table schema does not match. If orc schema has fewer columns than projection OrcEncodedDataConsumer may receive an incomplete TypeDescription array which will manifest itself as NullPointerException later.

      We can at least verify that OrcEncodedDataConsumer gets enough TypeDescriptions. If assertion fails, user sees there is something wrong with the schema and hopefully resolves the problem quickly. If there are enough columns in the file but the schema of the query does not match, user generally sees a ClassCastException. If there are enough columns and types accidentally match, there is nothing we can do as this is an external table.

      We have seen this when trying to use a managed table as external table location. Although user facing schemas are the same, managed table has acid related metadata. I am adding a q file demonstrating NullPointerException with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added this to precommit tests as it is hard to assert the exception message from mini driver framework and effectively it is just changing the error.

        Attachments

        1. acid_table_as_external.q
          1 kB
          Mustafa İman
        2. acid_table_as_external.q.out
          34 kB
          Mustafa İman
        3. HIVE-23455.patch
          1 kB
          Mustafa İman

          Activity

            People

            • Assignee:
              mustafaiman Mustafa İman
              Reporter:
              mustafaiman Mustafa İman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: