Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1010

Implement INFORMATION_SCHEMA in Hive

    Details

      Description

      INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to implement using our metastore.

      1. HIVE-1010.1.patch
        206 kB
        Gunther Hagleitner
      2. HIVE-1010.2.patch
        195 kB
        Gunther Hagleitner
      3. HIVE-1010.3.patch
        180 kB
        Gunther Hagleitner
      4. HIVE-1010.4.patch
        240 kB
        Gunther Hagleitner
      5. HIVE-1010.5.patch
        246 kB
        Gunther Hagleitner

        Issue Links

          Activity

          Hide
          cwsteinbach Carl Steinbach added a comment -

          MySQL INFORMATION_SCHEMA views: http://dev.mysql.com/doc/refman/5.1/en/information-schema.html

          There's been some discussion offline about supporting this using the JDO query language, i.e. expressing HiveQL queries against the INFORMATION_SCHEMA as JDOQL queries against the metastore.

          Additionally, supporting an INFORMATION_SCHEMA is probably the biggest missing piece that blocks grafting the PostgreSQL ODBC driver onto Hive (as has been done for LucidDB, H2, etc).

          Show
          cwsteinbach Carl Steinbach added a comment - MySQL INFORMATION_SCHEMA views: http://dev.mysql.com/doc/refman/5.1/en/information-schema.html There's been some discussion offline about supporting this using the JDO query language, i.e. expressing HiveQL queries against the INFORMATION_SCHEMA as JDOQL queries against the metastore. Additionally, supporting an INFORMATION_SCHEMA is probably the biggest missing piece that blocks grafting the PostgreSQL ODBC driver onto Hive (as has been done for LucidDB, H2, etc).
          Hide
          jvs John Sichi added a comment -

          Can we get the native SQL translation from JDO and then wrap this to get the data out of the metastore?

          http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html

          Then use HIve views to map metastore representation to INFORMATION_SCHEMA.

          Show
          jvs John Sichi added a comment - Can we get the native SQL translation from JDO and then wrap this to get the data out of the metastore? http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html Then use HIve views to map metastore representation to INFORMATION_SCHEMA.
          Hide
          jvs John Sichi added a comment -

          See HIVE-1555 for a JDBC storage handler.

          Show
          jvs John Sichi added a comment - See HIVE-1555 for a JDBC storage handler.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          WIP. Based on patch in HIVE-1555. SCHEMATA, TABLES, and COLUMNS are there.

          Show
          hagleitn Gunther Hagleitner added a comment - WIP. Based on patch in HIVE-1555 . SCHEMATA, TABLES, and COLUMNS are there.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12853414/HIVE-1010.5.patch

          ERROR: -1 due to build exiting with an error

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3648/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3648/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3648/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
          2017-02-18 11:12:01.027
          + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
          + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'MAVEN_OPTS=-Xmx1g '
          + MAVEN_OPTS='-Xmx1g '
          + cd /data/hiveptest/working/
          + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3648/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ git = \s\v\n ]]
          + [[ git = \g\i\t ]]
          + [[ -z master ]]
          + [[ -d apache-github-source-source ]]
          + [[ ! -d apache-github-source-source/.git ]]
          + [[ ! -d apache-github-source-source ]]
          + date '+%Y-%m-%d %T.%3N'
          2017-02-18 11:12:01.030
          + cd apache-github-source-source
          + git fetch origin
          + git reset --hard HEAD
          HEAD is now at 3f986d7 HIVE-15919: Row count mismatch for count * query (Jason Dere, reviewed by Matt McCline)
          + git clean -f -d
          Removing ql/src/test/queries/clientpositive/llap_reader.q
          Removing ql/src/test/results/clientpositive/llap/llap_reader.q.out
          Removing ql/src/test/results/clientpositive/llap_reader.q.out
          + git checkout master
          Already on 'master'
          Your branch is up-to-date with 'origin/master'.
          + git reset --hard origin/master
          HEAD is now at 3f986d7 HIVE-15919: Row count mismatch for count * query (Jason Dere, reviewed by Matt McCline)
          + git merge --ff-only origin/master
          Already up-to-date.
          + date '+%Y-%m-%d %T.%3N'
          2017-02-18 11:12:02.061
          + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hiveptest/working/scratch/build.patch
          + [[ -f /data/hiveptest/working/scratch/build.patch ]]
          + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
          + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
          error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java: No such file or directory
          error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java: No such file or directory
          error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/DatabaseType.java: No such file or directory
          error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java: No such file or directory
          error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java: No such file or directory
          error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java: No such file or directory
          error: ql/src/test/queries/clientpositive/jdbc_handler.q: No such file or directory
          error: ql/src/test/results/clientpositive/jdbc_handler.q.out: No such file or directory
          error: ql/src/test/results/clientpositive/llap/jdbc_handler.q.out: No such file or directory
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12853414 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12853414/HIVE-1010.5.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3648/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3648/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3648/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-02-18 11:12:01.027 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3648/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-02-18 11:12:01.030 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 3f986d7 HIVE-15919: Row count mismatch for count * query (Jason Dere, reviewed by Matt McCline) + git clean -f -d Removing ql/src/test/queries/clientpositive/llap_reader.q Removing ql/src/test/results/clientpositive/llap/llap_reader.q.out Removing ql/src/test/results/clientpositive/llap_reader.q.out + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 3f986d7 HIVE-15919: Row count mismatch for count * query (Jason Dere, reviewed by Matt McCline) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-02-18 11:12:02.061 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java: No such file or directory error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java: No such file or directory error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/DatabaseType.java: No such file or directory error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java: No such file or directory error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java: No such file or directory error: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java: No such file or directory error: ql/src/test/queries/clientpositive/jdbc_handler.q: No such file or directory error: ql/src/test/results/clientpositive/jdbc_handler.q.out: No such file or directory error: ql/src/test/results/clientpositive/llap/jdbc_handler.q.out: No such file or directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12853414 - PreCommit-HIVE-Build
          Hide
          hagleitn Gunther Hagleitner added a comment -

          (patch requires HIVE-1555) cc Thejas M Nair

          Show
          hagleitn Gunther Hagleitner added a comment - (patch requires HIVE-1555 ) cc Thejas M Nair

            People

            • Assignee:
              hagleitn Gunther Hagleitner
              Reporter:
              hammer Jeff Hammerbacher
            • Votes:
              5 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development