Hive
  1. Hive
  2. HIVE-2315

DatabaseMetadata.getColumns() does not return partition column names for a table

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.7.1
    • Fix Version/s: 0.8.0
    • Component/s: JDBC
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      getColumns() method of DatabaseMetadata for HIVE JDBC Driver does not return the partition column names. Where as from HIVE CLI, if you do a 'describe tablename' you get all columns including the partition columns. It would be nice if getColumns() method returns all columns.

      1. HIVE-2315.patch
        4 kB
        Patrick Hunt
      2. HIVE-2315.patch
        3 kB
        Patrick Hunt

        Issue Links

          Activity

          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Doug Sedlak added a comment -

          For an 'int' partition column, the 'getColumnTypeName' metadata call is returning type 'string'. However, a DESCRIBE correctly returns type 'int'. Seems like a bug if it replicates for others? I have the patched code in place on both client and server. Thanks!

          Sample table is:
          create EXTERNAL table partition_by_int (str_val string) PARTITIONED BY (int_col INT) ROW FORMAT delimited fields terminated by ',' STORED AS TEXTFILE;

          And (shortened) JDBC code is:
          res = stmt.executeQuery("select * from partition_by_int");
          ResultSetMetaData metaData = res.getMetaData();
          for (int i = 0; i < metaData.getColumnCount(); i++)

          { System.out.print(metaData.getColumnName(i + 1) + " \t"); System.out.print(metaData.getColumnDisplaySize(i + 1) + "\t"); System.out.println(metaData.getColumnTypeName(i + 1)); }

          With output:
          str_val 2147483647 string
          int_col 2147483647 string

          In contrast, DESCRIBE partition_by_int generates:
          str_val string
          int_col int

          Show
          Doug Sedlak added a comment - For an 'int' partition column, the 'getColumnTypeName' metadata call is returning type 'string'. However, a DESCRIBE correctly returns type 'int'. Seems like a bug if it replicates for others? I have the patched code in place on both client and server. Thanks! Sample table is: create EXTERNAL table partition_by_int (str_val string) PARTITIONED BY (int_col INT) ROW FORMAT delimited fields terminated by ',' STORED AS TEXTFILE; And (shortened) JDBC code is: res = stmt.executeQuery("select * from partition_by_int"); ResultSetMetaData metaData = res.getMetaData(); for (int i = 0; i < metaData.getColumnCount(); i++) { System.out.print(metaData.getColumnName(i + 1) + " \t"); System.out.print(metaData.getColumnDisplaySize(i + 1) + "\t"); System.out.println(metaData.getColumnTypeName(i + 1)); } With output: str_val 2147483647 string int_col 2147483647 string In contrast, DESCRIBE partition_by_int generates: str_val string int_col int
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #903 (See https://builds.apache.org/job/Hive-trunk-h0.21/903/)
          HIVE-2315. DatabaseMetadata.getColumns() does not return partition column names for a table (Patrick Hunt via cws)

          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1159423
          Files :

          • /hive/trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java
          • /hive/trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #903 (See https://builds.apache.org/job/Hive-trunk-h0.21/903/ ) HIVE-2315 . DatabaseMetadata.getColumns() does not return partition column names for a table (Patrick Hunt via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1159423 Files : /hive/trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java /hive/trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java
          Carl Steinbach made changes -
          Link This issue is duplicated by HIVE-1573 [ HIVE-1573 ]
          Carl Steinbach made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Carl Steinbach added a comment -

          Committed to trunk. Thanks Patrick!

          Show
          Carl Steinbach added a comment - Committed to trunk. Thanks Patrick!
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1468/
          -----------------------------------------------------------

          Review request for hive and Carl Steinbach.

          Summary
          -------

          This patch fixes the problem and adds a couple of tests.

          This addresses bug HIVE-2315.
          https://issues.apache.org/jira/browse/HIVE-2315

          Diffs


          jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java d570fca
          jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java d72cf43

          Diff: https://reviews.apache.org/r/1468/diff

          Testing
          -------

          units pass, a user also verified it fixed the issue they were seeing.

          Thanks,

          Patrick

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1468/ ----------------------------------------------------------- Review request for hive and Carl Steinbach. Summary ------- This patch fixes the problem and adds a couple of tests. This addresses bug HIVE-2315 . https://issues.apache.org/jira/browse/HIVE-2315 Diffs jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java d570fca jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java d72cf43 Diff: https://reviews.apache.org/r/1468/diff Testing ------- units pass, a user also verified it fixed the issue they were seeing. Thanks, Patrick
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Patrick Hunt made changes -
          Attachment HIVE-2315.patch [ 12490134 ]
          Hide
          Patrick Hunt added a comment -

          Updated to a single patch with fix and both sets of tests.

          Show
          Patrick Hunt added a comment - Updated to a single patch with fix and both sets of tests.
          Patrick Hunt made changes -
          Attachment HIVE-2315_part2.patch [ 12489724 ]
          Hide
          Mythili Gopalakrishnan added a comment -

          Patrick, it is my fault, I forgot to update the jars on the server side. Once I did that, I see the partition columns when I do a select * from foo2. This issue is fixed.

          Show
          Mythili Gopalakrishnan added a comment - Patrick, it is my fault, I forgot to update the jars on the server side. Once I did that, I see the partition columns when I do a select * from foo2. This issue is fixed.
          Hide
          Patrick Hunt added a comment -

          Mythili, please take a look at the HIVE-2315_part2.patch file I just attached to this jira. The existing tests were not strongly verifying that "select * from table" was returning the partition column as part of the result set. I updated the test to do so. However subsequently the test is still passing (I verified in the debugger that this code is being executed).

          Please take a look at this test:
          org.apache.hadoop.hive.jdbc.TestJdbcDriver.doTestSelectAll(String, int, int)
          called from here:
          org.apache.hadoop.hive.jdbc.TestJdbcDriver.testSelectAllPartioned()

          So, afaict, we are returning the partition column as part of the result set. (I'm also seeing this in squirrel as I mentioned).

          Can you take a look at your setup again? Perhaps there is another issue somewhere in the Hive JDBC driver that's causing DBVizualizer to display this problem while Squirrel is fine?

          Show
          Patrick Hunt added a comment - Mythili, please take a look at the HIVE-2315 _part2.patch file I just attached to this jira. The existing tests were not strongly verifying that "select * from table" was returning the partition column as part of the result set. I updated the test to do so. However subsequently the test is still passing (I verified in the debugger that this code is being executed). Please take a look at this test: org.apache.hadoop.hive.jdbc.TestJdbcDriver.doTestSelectAll(String, int, int) called from here: org.apache.hadoop.hive.jdbc.TestJdbcDriver.testSelectAllPartioned() So, afaict, we are returning the partition column as part of the result set. (I'm also seeing this in squirrel as I mentioned). Can you take a look at your setup again? Perhaps there is another issue somewhere in the Hive JDBC driver that's causing DBVizualizer to display this problem while Squirrel is fine?
          Patrick Hunt made changes -
          Attachment HIVE-2315_part2.patch [ 12489724 ]
          Hide
          Patrick Hunt added a comment -

          Updated test to explicitly verify that JDBC result set contains the partition column on a "select * from t". Otw the code is unchanged.

          Show
          Patrick Hunt added a comment - Updated test to explicitly verify that JDBC result set contains the partition column on a "select * from t". Otw the code is unchanged.
          Hide
          Mythili Gopalakrishnan added a comment -

          Patrick

          This is exactly what I have done as well. I use a tool called DBVizualizer, very similar to Squirrel. When I click on foo2, I see the partition columns, but when I do a select *, I don't see the partition columns.

          One thing I noticed (please pardon my ignorance as I have not had a chance to look deep into the HIVE JDBC driver source code) is that the patch listed only 2 files to patch, HiveDatabaseMetaData.java and TestJdbcDriver.java. Is it missing some other file that would have included the changes for the query part?

          Thanks,
          Mythili

          Show
          Mythili Gopalakrishnan added a comment - Patrick This is exactly what I have done as well. I use a tool called DBVizualizer, very similar to Squirrel. When I click on foo2, I see the partition columns, but when I do a select *, I don't see the partition columns. One thing I noticed (please pardon my ignorance as I have not had a chance to look deep into the HIVE JDBC driver source code) is that the patch listed only 2 files to patch, HiveDatabaseMetaData.java and TestJdbcDriver.java. Is it missing some other file that would have included the changes for the query part? Thanks, Mythili
          Patrick Hunt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Patrick Hunt added a comment -

          waiting for feedback from Mythili.

          Show
          Patrick Hunt added a comment - waiting for feedback from Mythili.
          Carl Steinbach made changes -
          Link This issue duplicates HIVE-1573 [ HIVE-1573 ]
          Hide
          Patrick Hunt added a comment -

          Hi Mythili can you give an example of the query you are running? i.e. steps how to reproduce.

          When I run the following from hive:

          create table foo2 (a string, b string, c string) partitioned by (dt string) row format delimited fields terminated by '\t';
          load data local inpath './data.txt' overwrite into table foo2 partition (dt='8');
          load data local inpath './data.txt' overwrite into table foo2 partition (dt='9');
          select * from foo2;
          

          I see dt in the output (hive cli).

          When I run "select * from foo2" from Squirrel I also see dt in the output after applying this patch (I also see dt in the metadata reported by Squirrel for the table).

          Show
          Patrick Hunt added a comment - Hi Mythili can you give an example of the query you are running? i.e. steps how to reproduce. When I run the following from hive: create table foo2 (a string, b string, c string) partitioned by (dt string) row format delimited fields terminated by '\t'; load data local inpath './data.txt' overwrite into table foo2 partition (dt='8'); load data local inpath './data.txt' overwrite into table foo2 partition (dt='9'); select * from foo2; I see dt in the output (hive cli). When I run "select * from foo2" from Squirrel I also see dt in the output after applying this patch (I also see dt in the metadata reported by Squirrel for the table).
          Hide
          Mythili Gopalakrishnan added a comment -
          • I applied this patch and confirm that it fixes this issue HIVE-2315.
          • However it does NOT fix issue HIVE-1573. When I do a select * from a partitioned column, I don't see the partition columns as part of the resultset.
          Show
          Mythili Gopalakrishnan added a comment - I applied this patch and confirm that it fixes this issue HIVE-2315 . However it does NOT fix issue HIVE-1573 . When I do a select * from a partitioned column, I don't see the partition columns as part of the resultset.
          Patrick Hunt made changes -
          Link This issue relates to HIVE-1573 [ HIVE-1573 ]
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Patrick Hunt made changes -
          Attachment HIVE-2315.patch [ 12489509 ]
          Hide
          Patrick Hunt added a comment -

          use get_schema rather than get_fields. Verified via unit test as well as via squirrel.

          Show
          Patrick Hunt added a comment - use get_schema rather than get_fields. Verified via unit test as well as via squirrel.
          Patrick Hunt made changes -
          Field Original Value New Value
          Assignee Patrick Hunt [ phunt ]
          Mythili Gopalakrishnan created issue -

            People

            • Assignee:
              Patrick Hunt
              Reporter:
              Mythili Gopalakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development