Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-537

Hive TypeInfo/ObjectInspector to support union (besides struct, array, and map)

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      There are already some cases inside the code that we use heterogeneous data: JoinOperator, and UnionOperator (in the sense that different parents can pass in records with different ObjectInspectors).
      We currently use Operator's parentID to distinguish that. However that approach does not extend to more complex plans that might be needed in the future.

      We will support the union type like this:

      TypeDefinition:
        type: primitivetype | structtype | arraytype | maptype | uniontype
        uniontype: "union" "<" tag ":" type ("," tag ":" type)* ">"
      Example:
        union<0:int,1:double,2:array<string>,3:struct<a:int,b:string>>
      
      Example of serialized data format:
        We will first store the tag byte before we serialize the object. On deserialization, we will first read out the tag byte, then we know what is the current type of the following object, so we can deserialize it successfully.
      
      Interface for ObjectInspector:
      interface UnionObjectInspector {
        /** Returns the array of OIs that are for each of the tags
         */
        ObjectInspector[] getObjectInspectors();
        /** Return the tag of the object.
         */
        byte getTag(Object o);
        /** Return the field based on the tag value associated with the Object.
         */
        Object getField(Object o);
      };
      
      An example serialization format (Using deliminated format, with ' ' as first-level delimitor and '=' as second-level delimitor)
      userid:int,log:union<0:struct<touserid:int,message:string>>,1:string>
      123 1=login
      123 0=243=helloworld
      123 1=logout
      
      

        Attachments

        1. HIVE-537.1.patch
          10 kB
          Min Zhou
        2. patch-537.txt
          53 kB
          Amareshwari Sriramadasu
        3. patch-537-1.txt
          53 kB
          Amareshwari Sriramadasu
        4. patch-537-2.txt
          68 kB
          Amareshwari Sriramadasu
        5. patch-537-3.txt
          69 kB
          Amareshwari Sriramadasu
        6. patch-537-4.txt
          69 kB
          Amareshwari Sriramadasu
        7. patch-537-5.txt
          72 kB
          Amareshwari Sriramadasu

          Issue Links

            Activity

              People

              • Assignee:
                amareshwari Amareshwari Sriramadasu
                Reporter:
                zshao Zheng Shao
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: