Hive
  1. Hive
  2. HIVE-3581

get_json_object and json_tuple return null in the presence of new line characters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This was introduced when these functions were updated to use Jackson.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        30m 35s 1 Kevin Wilfong 15/Oct/12 23:03
        Patch Available Patch Available Resolved Resolved
        6h 33m 1 Namit Jain 16/Oct/12 05:37
        Resolved Resolved Closed Closed
        86d 14h 17m 1 Ashutosh Chauhan 10/Jan/13 19:54
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters
        (Kevin Wilfong via namit) (Revision 1398654)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q
        • /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters (Kevin Wilfong via namit) (Revision 1398654) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Ashutosh Chauhan made changes -
        Fix Version/s 0.10.0 [ 12320745 ]
        Hide
        Kevin Wilfong added a comment -

        Filed HIVE-3607 to set mapreduce.task.classpath.user.precedence to true.

        Show
        Kevin Wilfong added a comment - Filed HIVE-3607 to set mapreduce.task.classpath.user.precedence to true.
        Hide
        Kevin Wilfong added a comment -

        Sorry about that. It looks like we'll either need to set mapreduce.task.classpath.user.precedence to true in Hive, or provide two versions of JSON parsing, one using JSONObject and the other using Jackson, controlled by a config. I think the former sounds preferable provided it works.

        Show
        Kevin Wilfong added a comment - Sorry about that. It looks like we'll either need to set mapreduce.task.classpath.user.precedence to true in Hive, or provide two versions of JSON parsing, one using JSONObject and the other using Jackson, controlled by a config. I think the former sounds preferable provided it works.
        Hide
        Philip Tromans added a comment -

        This patch breaks hive trunk on Hadoop 0.20.205.0, I guess because of the difference in Jackson versions. Running hive gives:

        Exception in thread "main" java.lang.NoSuchFieldError: ALLOW_UNQUOTED_CONTROL_CHARS
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFJSONTuple.<clinit>(GenericUDTFJSONTuple.java:59)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:545)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:539)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:472)
        at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:202)
        at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:86)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:635)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

        Backing the patch out makes everything work again.

        Show
        Philip Tromans added a comment - This patch breaks hive trunk on Hadoop 0.20.205.0, I guess because of the difference in Jackson versions. Running hive gives: Exception in thread "main" java.lang.NoSuchFieldError: ALLOW_UNQUOTED_CONTROL_CHARS at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFJSONTuple.<clinit>(GenericUDTFJSONTuple.java:59) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:545) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:539) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:472) at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:202) at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:86) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:635) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Backing the patch out makes everything work again.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1740 (See https://builds.apache.org/job/Hive-trunk-h0.21/1740/)
        HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters
        (Kevin Wilfong via namit) (Revision 1398654)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q
        • /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1740 (See https://builds.apache.org/job/Hive-trunk-h0.21/1740/ ) HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters (Kevin Wilfong via namit) (Revision 1398654) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin

        Show
        Namit Jain added a comment - Committed. Thanks Kevin
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        Kevin Wilfong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Kevin Wilfong made changes -
        Field Original Value New Value
        Attachment HIVE-3581.1.patch.txt [ 12549230 ]
        Hide
        Kevin Wilfong added a comment -
        Show
        Kevin Wilfong added a comment - diff here https://reviews.facebook.net/D6039
        Kevin Wilfong created issue -

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development