Hive
  1. Hive
  2. HIVE-4252

hiveserver2 string representation of complex types are inconsistent with cli

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.11.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      For example, it prints struct as "[null, null, null]" instead of "

      {\"r\":null,\"s\":null,\"t\":null}

      "
      And for maps it is printing it as "

      {k=v}

      " instead of

      {\"k\":\"v\"}
      1. HIVE-4252.2.patch
        4 kB
        Thejas M Nair
      2. HIVE-4252.1.patch
        4 kB
        Thejas M Nair

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/)
          HIVE-4252 : hiveserver2 string representation of complex types are inconsistent with cli (Thejas Nair via Ashutosh Chauhan) (Revision 1464049)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464049
          Files :

          • /hive/trunk/jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
          • /hive/trunk/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/ ) HIVE-4252 : hiveserver2 string representation of complex types are inconsistent with cli (Thejas Nair via Ashutosh Chauhan) (Revision 1464049) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464049 Files : /hive/trunk/jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java /hive/trunk/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #2043 (See https://builds.apache.org/job/Hive-trunk-h0.21/2043/)
          HIVE-4252 : hiveserver2 string representation of complex types are inconsistent with cli (Thejas Nair via Ashutosh Chauhan) (Revision 1464049)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464049
          Files :

          • /hive/trunk/jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
          • /hive/trunk/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #2043 (See https://builds.apache.org/job/Hive-trunk-h0.21/2043/ ) HIVE-4252 : hiveserver2 string representation of complex types are inconsistent with cli (Thejas Nair via Ashutosh Chauhan) (Revision 1464049) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464049 Files : /hive/trunk/jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java /hive/trunk/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Thejas!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Thejas!
          Hide
          Ashutosh Chauhan added a comment -

          I see. If spec doesn't mandate it one way or the other, than sticking to what cli already does is best, since than hive will be consistent. And json format which cli uses is a good format as it makes parsing back and forth easier.
          +1 will commit if tests pass.

          Show
          Ashutosh Chauhan added a comment - I see. If spec doesn't mandate it one way or the other, than sticking to what cli already does is best, since than hive will be consistent. And json format which cli uses is a good format as it makes parsing back and forth easier. +1 will commit if tests pass.
          Hide
          Thejas M Nair added a comment -

          Table B-6 "Use of ResultSet getter Methods to Retrieve JDBC Data Types" in jdbc 3(&4) spec shows what datatypes support getString(), and Array and Struct are not one of those. Ie, we have leeway here. There is also not much pattern to be followed here, as we don't have many databases supporting these types (eq mysql and sql server don't).
          I think having json strings is useful for being able to parse the strings back to types.

          btw, the current jdbc driver implementation returns strings for the complex types, if you do a ResultSetMetadata.getColumnType(int) on column that is actually of hive map/struct/array type, it returns the String type .
          Ideally, we should add support for returning real array and struct types. I am not sure how map type should be handled, maybe json string would remain the workaround.

          Show
          Thejas M Nair added a comment - Table B-6 "Use of ResultSet getter Methods to Retrieve JDBC Data Types" in jdbc 3(&4) spec shows what datatypes support getString(), and Array and Struct are not one of those. Ie, we have leeway here. There is also not much pattern to be followed here, as we don't have many databases supporting these types (eq mysql and sql server don't). I think having json strings is useful for being able to parse the strings back to types. btw, the current jdbc driver implementation returns strings for the complex types, if you do a ResultSetMetadata.getColumnType(int) on column that is actually of hive map/struct/array type, it returns the String type . Ideally, we should add support for returning real array and struct types. I am not sure how map type should be handled, maybe json string would remain the workaround.
          Hide
          Ashutosh Chauhan added a comment -

          It will be good to check if sql standard provides any guideline for string representation of complex type. Or implementations are free to choose their own. For example I ran a quick test against postgres for arrays and its different than what we do.

          CREATE TABLE sal_emp (
              name            text,
              pay_by_quarter  integer[]
          );
          
          INSERT INTO sal_emp
              VALUES ('Bill',
              '{10000, 10000, 10000, 10000}');
          
          SELECT * FROM sal_emp;
           name  |      pay_by_quarter       
          -------+---------------------------
           Bill  | {10000,10000,10000,10000}
          
          Show
          Ashutosh Chauhan added a comment - It will be good to check if sql standard provides any guideline for string representation of complex type. Or implementations are free to choose their own. For example I ran a quick test against postgres for arrays and its different than what we do. CREATE TABLE sal_emp ( name text, pay_by_quarter integer[] ); INSERT INTO sal_emp VALUES ('Bill', '{10000, 10000, 10000, 10000}'); SELECT * FROM sal_emp; name | pay_by_quarter -------+--------------------------- Bill | {10000,10000,10000,10000}
          Hide
          Thejas M Nair added a comment -
          Show
          Thejas M Nair added a comment - reviewboard link https://reviews.apache.org/r/10211/
          Hide
          Thejas M Nair added a comment -

          HIVE-4252.2.patch rebased on HIVE-4269, requires that to be applied first.

          Show
          Thejas M Nair added a comment - HIVE-4252 .2.patch rebased on HIVE-4269 , requires that to be applied first.
          Hide
          Thejas M Nair added a comment -

          With the patch the string representation are valid json strings. That makes it easy to parse them .
          I will create a patch to track converting them into proper jdbc complex types.

          Show
          Thejas M Nair added a comment - With the patch the string representation are valid json strings. That makes it easy to parse them . I will create a patch to track converting them into proper jdbc complex types.

            People

            • Assignee:
              Thejas M Nair
              Reporter:
              Thejas M Nair
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development