Hive
  1. Hive
  2. HIVE-1735

Extend Explode UDTF to handle Maps

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The explode() UDTF currently only accepts arrays as input. We should modify it
      so that it can also handle map inputs, in which case it will output two columns
      corresponding to the key and value fields.

      1. HIVE-1735.1.patch
        56 kB
        Jonathan Natkins
      2. HIVE-1735.2.patch.txt
        55 kB
        Carl Steinbach
      3. HIVE-1735.2.patch.txt
        56 kB
        Jonathan Natkins
      4. HIVE-1735.3.patch.txt
        62 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #876 (See https://builds.apache.org/job/Hive-trunk-h0.21/876/)
        HIVE-1735. Extend Explode UDTF to handle Maps (Jonathan Natkins and Amareshwari Sriramadasu via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1154061
        Files :

        • /hive/trunk/ql/src/test/results/clientpositive/explode_null.q.out
        • /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q
        • /hive/trunk/ql/src/test/queries/clientpositive/explode_null.q
        • /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out
        • /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q
        • /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_explode.q.out
        • /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_explode.q
        • /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out
        • /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java
        • /hive/trunk/ql/src/test/results/clientpositive/udtf_explode.q.out
        • /hive/trunk/ql/src/test/queries/clientpositive/udtf_explode.q
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #876 (See https://builds.apache.org/job/Hive-trunk-h0.21/876/ ) HIVE-1735 . Extend Explode UDTF to handle Maps (Jonathan Natkins and Amareshwari Sriramadasu via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1154061 Files : /hive/trunk/ql/src/test/results/clientpositive/explode_null.q.out /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q /hive/trunk/ql/src/test/queries/clientpositive/explode_null.q /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_explode.q.out /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q /hive/trunk/ql/src/test/queries/clientpositive/udf_explode.q /hive/trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out /hive/trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java /hive/trunk/ql/src/test/results/clientpositive/udtf_explode.q.out /hive/trunk/ql/src/test/queries/clientpositive/udtf_explode.q
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Natty and Amareshwari!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Natty and Amareshwari!
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1256/#review1288
        -----------------------------------------------------------

        Ship it!

        +1. Will commit if tests pass.

        • Carl

        On 2011-08-03 04:45:10, Amareshwari Sriramadasu wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1256/

        -----------------------------------------------------------

        (Updated 2011-08-03 04:45:10)

        Review request for hive and Carl Steinbach.

        Summary

        -------

        Changes from last patch:

        Re-based to trunk.

        Removed assert not null statements for list and map.

        Updated explode_null.q to test NULL values for map also.

        This addresses bug HIVE-1735.

        https://issues.apache.org/jira/browse/HIVE-1735

        Diffs

        -----

        trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java 1151047

        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION

        trunk/ql/src/test/queries/clientpositive/explode_null.q 1151047

        trunk/ql/src/test/queries/clientpositive/udf_explode.q 1151047

        trunk/ql/src/test/queries/clientpositive/udtf_explode.q 1151047

        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION

        trunk/ql/src/test/results/clientpositive/explode_null.q.out 1151047

        trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047

        trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047

        Diff: https://reviews.apache.org/r/1256/diff

        Testing

        -------

        All tests passed with the patch.

        Thanks,

        Amareshwari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1256/#review1288 ----------------------------------------------------------- Ship it! +1. Will commit if tests pass. Carl On 2011-08-03 04:45:10, Amareshwari Sriramadasu wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1256/ ----------------------------------------------------------- (Updated 2011-08-03 04:45:10) Review request for hive and Carl Steinbach. Summary ------- Changes from last patch: Re-based to trunk. Removed assert not null statements for list and map. Updated explode_null.q to test NULL values for map also. This addresses bug HIVE-1735 . https://issues.apache.org/jira/browse/HIVE-1735 Diffs ----- trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java 1151047 trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/explode_null.q 1151047 trunk/ql/src/test/queries/clientpositive/udf_explode.q 1151047 trunk/ql/src/test/queries/clientpositive/udtf_explode.q 1151047 trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/explode_null.q.out 1151047 trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047 trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047 Diff: https://reviews.apache.org/r/1256/diff Testing ------- All tests passed with the patch. Thanks, Amareshwari
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1256/
        -----------------------------------------------------------

        (Updated 2011-08-03 04:45:10.263008)

        Review request for hive and Carl Steinbach.

        Summary
        -------

        Changes from last patch:

        Re-based to trunk.
        Removed assert not null statements for list and map.
        Updated explode_null.q to test NULL values for map also.

        This addresses bug HIVE-1735.
        https://issues.apache.org/jira/browse/HIVE-1735

        Diffs


        trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java 1151047
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/explode_null.q 1151047
        trunk/ql/src/test/queries/clientpositive/udf_explode.q 1151047
        trunk/ql/src/test/queries/clientpositive/udtf_explode.q 1151047
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/explode_null.q.out 1151047
        trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047
        trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047

        Diff: https://reviews.apache.org/r/1256/diff

        Testing (updated)
        -------

        All tests passed with the patch.

        Thanks,

        Amareshwari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1256/ ----------------------------------------------------------- (Updated 2011-08-03 04:45:10.263008) Review request for hive and Carl Steinbach. Summary ------- Changes from last patch: Re-based to trunk. Removed assert not null statements for list and map. Updated explode_null.q to test NULL values for map also. This addresses bug HIVE-1735 . https://issues.apache.org/jira/browse/HIVE-1735 Diffs trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java 1151047 trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/explode_null.q 1151047 trunk/ql/src/test/queries/clientpositive/udf_explode.q 1151047 trunk/ql/src/test/queries/clientpositive/udtf_explode.q 1151047 trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/explode_null.q.out 1151047 trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047 trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047 Diff: https://reviews.apache.org/r/1256/diff Testing (updated) ------- All tests passed with the patch. Thanks, Amareshwari
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1256/
        -----------------------------------------------------------

        Review request for hive and Carl Steinbach.

        Summary
        -------

        Changes from last patch:

        Re-based to trunk.
        Removed assert not null statements for list and map.
        Updated explode_null.q to test NULL values for map also.

        This addresses bug HIVE-1735.
        https://issues.apache.org/jira/browse/HIVE-1735

        Diffs


        trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java 1151047
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/explode_null.q 1151047
        trunk/ql/src/test/queries/clientpositive/udf_explode.q 1151047
        trunk/ql/src/test/queries/clientpositive/udtf_explode.q 1151047
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/explode_null.q.out 1151047
        trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047
        trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047

        Diff: https://reviews.apache.org/r/1256/diff

        Testing
        -------

        Thanks,

        Amareshwari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1256/ ----------------------------------------------------------- Review request for hive and Carl Steinbach. Summary ------- Changes from last patch: Re-based to trunk. Removed assert not null statements for list and map. Updated explode_null.q to test NULL values for map also. This addresses bug HIVE-1735 . https://issues.apache.org/jira/browse/HIVE-1735 Diffs trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java 1151047 trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/explode_null.q 1151047 trunk/ql/src/test/queries/clientpositive/udf_explode.q 1151047 trunk/ql/src/test/queries/clientpositive/udtf_explode.q 1151047 trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/explode_null.q.out 1151047 trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047 trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047 Diff: https://reviews.apache.org/r/1256/diff Testing ------- Thanks, Amareshwari
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-01-25 13:19:29, Carl Steinbach wrote:

        > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java, line 106

        > <https://reviews.apache.org/r/354/diff/1/?file=10207#file10207line106>

        >

        > Under what conditions are map and array null? Is this code path exercised by any of the tests? Just trying to wrap my head around what this means. I'd prefer to see this changed to an assertion if map and array should never be null.

        explode_null.q is failing because of this assert change. So, this is needed to handle if the map or array object is null in some of the rows.

        • Amareshwari

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/354/#review154
        -----------------------------------------------------------

        On 2011-01-31 21:19:47, Jonathan Natkins wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/354/

        -----------------------------------------------------------

        (Updated 2011-01-31 21:19:47)

        Review request for hive.

        Summary

        -------

        Review request for HIVE-1735.

        This addresses bug HIVE-1735.

        https://issues.apache.org/jira/browse/HIVE-1735

        Diffs

        -----

        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java d1e7def

        ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION

        ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION

        ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION

        ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION

        ql/src/test/queries/clientpositive/udf_explode.q f5c0323

        ql/src/test/queries/clientpositive/udtf_explode.q f5c0323

        ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION

        ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION

        ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION

        ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION

        ql/src/test/results/clientpositive/udf_explode.q.out 511732f

        ql/src/test/results/clientpositive/udtf_explode.q.out 6702c3f

        Diff: https://reviews.apache.org/r/354/diff

        Testing

        -------

        Manual and automated testing

        Thanks,

        Jonathan

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-01-25 13:19:29, Carl Steinbach wrote: > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java, line 106 > < https://reviews.apache.org/r/354/diff/1/?file=10207#file10207line106 > > > Under what conditions are map and array null? Is this code path exercised by any of the tests? Just trying to wrap my head around what this means. I'd prefer to see this changed to an assertion if map and array should never be null. explode_null.q is failing because of this assert change. So, this is needed to handle if the map or array object is null in some of the rows. Amareshwari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/354/#review154 ----------------------------------------------------------- On 2011-01-31 21:19:47, Jonathan Natkins wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/354/ ----------------------------------------------------------- (Updated 2011-01-31 21:19:47) Review request for hive. Summary ------- Review request for HIVE-1735 . This addresses bug HIVE-1735 . https://issues.apache.org/jira/browse/HIVE-1735 Diffs ----- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java d1e7def ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION ql/src/test/queries/clientpositive/udf_explode.q f5c0323 ql/src/test/queries/clientpositive/udtf_explode.q f5c0323 ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION ql/src/test/results/clientpositive/udf_explode.q.out 511732f ql/src/test/results/clientpositive/udtf_explode.q.out 6702c3f Diff: https://reviews.apache.org/r/354/diff Testing ------- Manual and automated testing Thanks, Jonathan
        Hide
        Carl Steinbach added a comment -

        @Jonathan: I get a test failure in explode_null.q. Can you please look into this? Thanks.

        Show
        Carl Steinbach added a comment - @Jonathan: I get a test failure in explode_null.q. Can you please look into this? Thanks.
        Hide
        Carl Steinbach added a comment -

        Attaching updated patch from reviewboard.

        Show
        Carl Steinbach added a comment - Attaching updated patch from reviewboard.
        Hide
        Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        Show
        Carl Steinbach added a comment - +1. Will commit if tests pass.
        Hide
        Jonathan Natkins added a comment -

        Updated patch available
        https://reviews.apache.org/r/354

        Show
        Jonathan Natkins added a comment - Updated patch available https://reviews.apache.org/r/354
        Hide
        Carl Steinbach added a comment -

        Please see my comments on reviewboard.

        Show
        Carl Steinbach added a comment - Please see my comments on reviewboard.
        Show
        Jonathan Natkins added a comment - https://reviews.apache.org/r/354/
        Hide
        Carl Steinbach added a comment -

        Is this patch ready for review?

        Show
        Carl Steinbach added a comment - Is this patch ready for review?

          People

          • Assignee:
            Jonathan Natkins
            Reporter:
            Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development