Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15434

Add UDF to allow interrogation of uniontype values

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.1.1
    • 2.3.0, 3.0.0
    • UDF
    • Added UDF to allow interrogation of uniontype values.

    Description

      Overview

      As stated in the documention:

      UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP BY clauses will fail, and Hive does not define syntax to extract the tag or value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively look-at-only.

      It is essential to have a usable uniontype. Until full support is added to Hive users should at least have the ability to inspect and extract values for further comparison or transformation.

      Proposal

      I propose to add a GenericUDF that has 2 modes of operation. Consider the following schema and data that contains a union:

      Schema:

      struct<field1:uniontype<int,string>>
      

      Query:

      hive> select field1 from thing;
      {0:0}
      {1:"one"}
      

      Explode to Struct

      This method will recursively convert all unions within the type to structs with fields named tag_n, n being the tag number. Only the tag_* field that matches the tag of the union will be populated with the value. In the case above the schema of field1 will be converted to:

      struct<tag_0:int,tag_1:string>
      
      hive> select extract_union(field1) from thing;
      {"tag_0":0,"tag_1":null}
      {"tag_0":null,"tag_1":one}
      
      hive> select extract_union(field1).tag_0 from thing;
      0
      null
      

      Extract the specified tag

      This method will simply extract the value of the specified tag. If the tag number matches then the value is returned, if it does not, then null is returned.

      hive> select extract_union(field1, 0) from thing;
      0
      null
      

      Attachments

        1. HIVE-15434.02.patch
          29 kB
          Dave Maughan
        2. HIVE-15434.01.patch
          29 kB
          Dave Maughan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            nahguam Dave Maughan Assign to me
            nahguam Dave Maughan
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment