Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19205 Hive streaming ingest improvements (v2)
  3. HIVE-19494

Accept shade prefix during reflective instantiation of output format

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0, 3.0.0
    • Fix Version/s: 3.1.0, 3.0.0
    • Component/s: Streaming
    • Labels:
      None

      Description

      Hive Streaming API jars are sometimes shaded with a different prefix when used in environments where another version of hive already exists (spark for example). In most cases, shading is done with rename of classes with some prefix. If an uber/assembly jar is generated with renamed prefix, Hive Streaming API will not work as Hive Streaming API will reflectively instantiate outputformat class using FQCN string provided by metastore table storage descriptor object.
      For example:
      RecordWriter will create instance of OutputFormat using string "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with renamed class references are used, this class will not be found by the classloader.

      We can optionally accept a shade prefix from user via config which will be tried (as fallback) when ClassNotFoundException is thrown.

        Attachments

        1. HIVE-19494.1.patch
          6 kB
          Prasanth Jayachandran
        2. HIVE-19494-branch-3.patch
          6 kB
          Prasanth Jayachandran

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              prasanth_j Prasanth Jayachandran Assign to me
              Reporter:
              prasanth_j Prasanth Jayachandran

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment