Hive
  1. Hive
  2. HIVE-3581

get_json_object and json_tuple return null in the presence of new line characters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This was introduced when these functions were updated to use Jackson.

        Activity

        Hide
        Kevin Wilfong added a comment -
        Show
        Kevin Wilfong added a comment - diff here https://reviews.facebook.net/D6039
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin

        Show
        Namit Jain added a comment - Committed. Thanks Kevin
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1740 (See https://builds.apache.org/job/Hive-trunk-h0.21/1740/)
        HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters
        (Kevin Wilfong via namit) (Revision 1398654)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q
        • /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1740 (See https://builds.apache.org/job/Hive-trunk-h0.21/1740/ ) HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters (Kevin Wilfong via namit) (Revision 1398654) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Hide
        Philip Tromans added a comment -

        This patch breaks hive trunk on Hadoop 0.20.205.0, I guess because of the difference in Jackson versions. Running hive gives:

        Exception in thread "main" java.lang.NoSuchFieldError: ALLOW_UNQUOTED_CONTROL_CHARS
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFJSONTuple.<clinit>(GenericUDTFJSONTuple.java:59)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:545)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:539)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:472)
        at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:202)
        at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:86)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:635)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

        Backing the patch out makes everything work again.

        Show
        Philip Tromans added a comment - This patch breaks hive trunk on Hadoop 0.20.205.0, I guess because of the difference in Jackson versions. Running hive gives: Exception in thread "main" java.lang.NoSuchFieldError: ALLOW_UNQUOTED_CONTROL_CHARS at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFJSONTuple.<clinit>(GenericUDTFJSONTuple.java:59) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:545) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:539) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:472) at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:202) at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:86) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:635) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Backing the patch out makes everything work again.
        Hide
        Kevin Wilfong added a comment -

        Sorry about that. It looks like we'll either need to set mapreduce.task.classpath.user.precedence to true in Hive, or provide two versions of JSON parsing, one using JSONObject and the other using Jackson, controlled by a config. I think the former sounds preferable provided it works.

        Show
        Kevin Wilfong added a comment - Sorry about that. It looks like we'll either need to set mapreduce.task.classpath.user.precedence to true in Hive, or provide two versions of JSON parsing, one using JSONObject and the other using Jackson, controlled by a config. I think the former sounds preferable provided it works.
        Hide
        Kevin Wilfong added a comment -

        Filed HIVE-3607 to set mapreduce.task.classpath.user.precedence to true.

        Show
        Kevin Wilfong added a comment - Filed HIVE-3607 to set mapreduce.task.classpath.user.precedence to true.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters
        (Kevin Wilfong via namit) (Revision 1398654)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q
        • /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3581 get_json_object and json_tuple return null in the presence of new line characters (Kevin Wilfong via namit) (Revision 1398654) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398654 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java /hive/trunk/ql/src/test/queries/clientpositive/udf_get_json_object.q /hive/trunk/ql/src/test/queries/clientpositive/udtf_json_tuple.q /hive/trunk/ql/src/test/results/clientpositive/udf_get_json_object.q.out /hive/trunk/ql/src/test/results/clientpositive/udtf_json_tuple.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development