Hive
  1. Hive
  2. HIVE-699

Make lots of existing ObjectInspectors settable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We should make JavaObjectInspectors, StandardList/Struct/MapObjectInspector, ReflectionObjectInspectors settable.

      This will allow us to make it possible to write into Thrift tables, as well as ProtocolBuffer tables in the future.
      This will also allow us to pass complex objects into UDF and receive complex objects from UDF by using ObjectInspectorConverters.

      1. HIVE-699.8.patch
        1.71 MB
        Zheng Shao
      2. HIVE-699.7.patch
        1.71 MB
        Zheng Shao
      3. HIVE-699.6.patch
        1.70 MB
        Zheng Shao
      4. HIVE-699.4.patch
        1.57 MB
        Zheng Shao
      5. HIVE-699.3.patch
        1.63 MB
        Zheng Shao
      6. HIVE-699.2.patch
        85 kB
        Zheng Shao
      7. HIVE-699.1.patch
        67 kB
        Zheng Shao

        Issue Links

          Activity

          Hide
          Namit Jain added a comment -

          Committed. Thanks Zheng

          Show
          Namit Jain added a comment - Committed. Thanks Zheng
          Hide
          Zheng Shao added a comment -

          Updated some tests.

          Show
          Zheng Shao added a comment - Updated some tests.
          Hide
          Zheng Shao added a comment -

          @HIVE-699.7.patch:
          Incorporated code review comments from Namit:
          1. Add test for UDFs that takes Map<String,String>, List<Integer>, and Object. (udf_example_arraymapstruct.q)
          2. TypeInfoUtils.getParameterTypeInfos should return null if the number of parameters do not match.

          Show
          Zheng Shao added a comment - @ HIVE-699 .7.patch: Incorporated code review comments from Namit: 1. Add test for UDFs that takes Map<String,String>, List<Integer>, and Object. (udf_example_arraymapstruct.q) 2. TypeInfoUtils.getParameterTypeInfos should return null if the number of parameters do not match.
          Hide
          Zheng Shao added a comment -

          Resolved conflicts.

          Show
          Zheng Shao added a comment - Resolved conflicts.
          Hide
          Namit Jain added a comment -

          Let us do this after the branch assuming we have a branch pretty soon

          Show
          Namit Jain added a comment - Let us do this after the branch assuming we have a branch pretty soon
          Hide
          Zheng Shao added a comment -

          This one contains the changes of HIVE-719 and passed all tests.

          Show
          Zheng Shao added a comment - This one contains the changes of HIVE-719 and passed all tests.
          Hide
          Zheng Shao added a comment -

          This patch fixes all test failures. It's ready to commit, pending code review.

          Show
          Zheng Shao added a comment - This patch fixes all test failures. It's ready to commit, pending code review.
          Hide
          Zheng Shao added a comment -

          This patch fixes the test failure and added another example UDAF group_concat for HIVE-707.
          Note that we are able to use UDAF (instead of GenericUDAF) to implement arbitrary-length argument list. This may not be really efficient (since we are doing reflection), but will provide a much easier way to write UDAF with arbitrary-length argument list.

          Show
          Zheng Shao added a comment - This patch fixes the test failure and added another example UDAF group_concat for HIVE-707 . Note that we are able to use UDAF (instead of GenericUDAF) to implement arbitrary-length argument list. This may not be really efficient (since we are doing reflection), but will provide a much easier way to write UDAF with arbitrary-length argument list.
          Hide
          Namit Jain added a comment -

          Some tests are failing for this

          Show
          Namit Jain added a comment - Some tests are failing for this
          Hide
          Zheng Shao added a comment -

          This patch does the trick. As a side effect, UDAF can take complex arguments and return complex values - Hive will convert the object to the corresponding Java Class on the fly.

          This patch also includes an example UDAF (average) that uses a static inner class with 2 fields: mCount and mSum to store the internal state. The terminatePartial function just returns an object of this inner class, and Hive is capable of reading the value out using reflection ObjectInspector. Then in the reducer, Hive converts a struct with these 2 fields into an object of this inner class by using the ReflectionStructObjectInspector(which implements SettableStructObjectInspector).

          Show
          Zheng Shao added a comment - This patch does the trick. As a side effect, UDAF can take complex arguments and return complex values - Hive will convert the object to the corresponding Java Class on the fly. This patch also includes an example UDAF (average) that uses a static inner class with 2 fields: mCount and mSum to store the internal state. The terminatePartial function just returns an object of this inner class, and Hive is capable of reading the value out using reflection ObjectInspector. Then in the reducer, Hive converts a struct with these 2 fields into an object of this inner class by using the ReflectionStructObjectInspector(which implements SettableStructObjectInspector).

            People

            • Assignee:
              Zheng Shao
              Reporter:
              Zheng Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development