Hive
  1. Hive
  2. HIVE-1536

Add support for JDBC PreparedStatements

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: JDBC
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Many methods in HivePreparedStatement have been implemented. This cases "change log" attachment summarizes all changes.

      Description

      As a result of a Sprint which had us using Pentaho Data Integration with the Hive database we have updated the driver. Many PreparedStatement methods have been implemented. A patch will be attached tomorrow with a summary of changes.

      Note: A checkout of Hive/trunk was performed and the TestJdbcDriver test cased was run. This was done before any modifications were made to the checked out project. The testResultSetMetaData failed:

      java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
      at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:189)
      at org.apache.hadoop.hive.jdbc.TestJdbcDriver.testResultSetMetaData(TestJdbcDriver.java:530)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at junit.framework.TestCase.runTest(TestCase.java:154)
      at junit.framework.TestCase.runBare(TestCase.java:127)
      at junit.framework.TestResult$1.protect(TestResult.java:106)
      at junit.framework.TestResult.runProtected(TestResult.java:124)
      at junit.framework.TestResult.run(TestResult.java:109)
      at junit.framework.TestCase.run(TestCase.java:118)
      at junit.framework.TestSuite.runTest(TestSuite.java:208)
      at junit.framework.TestSuite.run(TestSuite.java:203)
      at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
      at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
      at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)

      A co-worker did the same and the tests passed. Both environments were Ubuntu and Hadoop version 0.20.2.

      Tests added to the TestJdbcDriver by us were successful.

      1. TestJdbcDriver-ant-3.log
        193 kB
        Sean Flatley
      2. JdbcDriverTest-ant-2.log
        191 kB
        Sean Flatley
      3. JdbcDriverTest-ant.log
        108 kB
        Sean Flatley
      4. HIVE-1536-changes-3.txt
        0.6 kB
        Sean Flatley
      5. HIVE-1536-changes-2.txt
        3 kB
        Sean Flatley
      6. HIVE-1536-changes.txt
        2 kB
        Sean Flatley
      7. HIVE-1536-3.patch
        5 kB
        Sean Flatley
      8. HIVE-1536-2.patch
        18 kB
        Sean Flatley
      9. HIVE-1536.patch
        44 kB
        Sean Flatley
      10. all-tests-ant.log
        53 kB
        Sean Flatley

        Issue Links

          Activity

          Sean Flatley created issue -
          Carl Steinbach made changes -
          Field Original Value New Value
          Summary Adding implementation to JDBC driver Add support for JDBC PreparedStatements
          Component/s Drivers [ 12313584 ]
          Hide
          John Sichi added a comment -

          Sounds like an environmental problem, since I don't think I've heard reports of failures on that test elsewhere. Can you check hive-trunk/build/ql/tmp/hive.log after the test run to find the exception details? (Search for the text of the select statement executed by that test.)

          Show
          John Sichi added a comment - Sounds like an environmental problem, since I don't think I've heard reports of failures on that test elsewhere. Can you check hive-trunk/build/ql/tmp/hive.log after the test run to find the exception details? (Search for the text of the select statement executed by that test.)
          Jeff Hammerbacher made changes -
          Link This issue is related to HIVE-48 [ HIVE-48 ]
          Hide
          Sean Flatley added a comment -

          The log entries can be found at the end of this comment. I did find in the Eclipse console an error entry reporting that JAVA_HOME is not set, so it does appear to be configuration:

          [junit] PREHOOK: query: select a,b,c,d,f as e,f*2 from testHiveJdbcDriverTable limit 1
          [junit] PREHOOK: type: QUERY
          [junit] PREHOOK: Input: default@testhivejdbcdrivertable
          [junit] PREHOOK: Output: file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-mr-10000
          [junit] Total MapReduce jobs = 1
          [junit] Launching Job 1 out of 1
          [junit] Number of reduce tasks is set to 0 since there's no reduce operator
          [junit] Error: JAVA_HOME is not set.
          [junit] FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

          I do have JAVA_HOME set and exported in my .bashrc file.

          Here is the hive.log file entries.

          2010-08-14 17:32:24,929 INFO ql.Driver (Driver.java:execute(425)) - Starting command: select a,b,c,d,f as e,f*2 from testHiveJdbcDriverTable limit 1
          2010-08-14 17:32:24,935 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: query: select a,b,c,d,f as e,f*2 from testHiveJdbcDriverTable limit 1
          2010-08-14 17:32:24,937 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: type: QUERY
          2010-08-14 17:32:24,938 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: Input: default@testhivejdbcdrivertable
          2010-08-14 17:32:24,939 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: Output: file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-mr-10000
          2010-08-14 17:32:24,940 INFO ql.Driver (SessionState.java:printInfo(268)) - Total MapReduce jobs = 1
          2010-08-14 17:32:24,942 INFO ql.Driver (SessionState.java:printInfo(268)) - Launching Job 1 out of 1
          2010-08-14 17:32:24,947 INFO exec.MapRedTask (SessionState.java:printInfo(268)) - Number of reduce tasks is set to 0 since there's no reduce operator
          2010-08-14 17:32:24,967 INFO exec.MapRedTask (MapRedTask.java:execute(152)) - Generating plan file file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-local-10002/plan.xml

          2010-08-14 17:32:25,588 INFO exec.MapRedTask (MapRedTask.java:execute(173)) - Executing: /home/sean/projects/hive/build/hadoopcore/hadoop-0.20.0/bin/hadoop jar /home/sean/projects/hive/build/ql/hive-exec-0.7.0.jar org.apache.hadoop.hive.ql.exec.ExecDriver libjars file:///home/sean/projects/hive/build/jdbc/test/test-udfs.jar -plan file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-local-10002/plan.xml -nolog -jobconf datanucleus.connectionPoolingType=DBCP -jobconf hive.exec.script.allow.partial.consumption=false -jobconf hive.query.id=sean_20100814173232_f49d776c-7273-4995-8cc3-f7aceabdbde3 -jobconf hive.hwi.listen.port=9999 -jobconf hive.map.aggr=true -jobconf hive.map.aggr.hash.min.reduction=0.5 -jobconf datanucleus.plugin.pluginRegistryBundleCheck=LOG -jobconf hive.exec.reducers.bytes.per.reducer=1000000000 -jobconf hive.optimize.cp=true -jobconf hive.exec.dynamic.partition.mode=strict -jobconf hive.merge.size.smallfiles.avgsize=16000000 -jobconf datanucleus.cache.level2.type=SOFT -jobconf hive.exec.max.created.files=100000 -jobconf hive.script.serde=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe -jobconf hive.fileformat.check=true -jobconf hive.exec.max.dynamic.partitions.pernode=100 -jobconf hive.enforce.sorting=false -jobconf hive.optimize.ppd=true -jobconf hive.optimize.groupby=true -jobconf hive.enforce.bucketing=false -jobconf javax.jdo.option.ConnectionUserName=APP -jobconf hive.mapred.reduce.tasks.speculative.execution=true -jobconf mapred.job.name=select+a%2Cb%2Cc%2Cd%2Cf+as+e%2Cf*2+from+testHiveJ...1%28Stage-1%29 -jobconf javax.jdo.option.DetachAllOnCommit=true -jobconf hive.mapred.local.mem=0 -jobconf datanucleus.cache.level2=false -jobconf hive.session.id=sean_201008141732 -jobconf fs.pfile.impl=org.apache.hadoop.fs.ProxyLocalFileSystem -jobconf hive.script.operator.id.env.var=HIVE_SCRIPT_OPERATOR_ID -jobconf hive.archive.har.parentdir.settable=false -jobconf hadoop.job.ugi=sean%2Csean%2Cadm%2Cdialout%2Ccdrom%2Cplugdev%2Clpadmin%2Cadmin%2Csambashare -jobconf test.src.dir=file%3A%2F%2F%24%7Bbuild.dir%7D%2Fsrc%2Ftest -jobconf hive.metastore.server.max.threads=100000 -jobconf hive.udtf.auto.progress=false -jobconf hive.hwi.war.file=lib%2Fhive-hwi%40VERSION%40.war jobconf datanucleus.validateTables=false -jobconf hive.exec.compress.output=false -jobconf hive.test.mode.prefix=test_ -jobconf hive.mapjoin.bucket.cache.size=100 -jobconf test.log.dir=%24%7Bbuild.dir%7D%2Ftest%2Flogs -jobconf test.data.files=%24%7Buser.dir%7D%2F..%2Fdata%2Ffiles -jobconf datanucleus.validateConstraints=false -jobconf hive.metastore.server.tcp.keepalive=true -jobconf mapred.reduce.tasks=-1 -jobconf hive.query.string=select+a%2Cb%2Cc%2Cd%2Cf+as+e%2Cf*2+from+testHiveJdbcDriverTable+limit+1 -jobconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat -jobconf hive.task.progress=false -jobconf hive.jar.path=%24%7Bbuild.dir.hive%7D%2Fql%2Fhive-exec%24%7Bversion%7D.jar jobconf hive.metastore.ds.retry.interval=1000 -jobconf javax.jdo.option.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver -jobconf hive.skewjoin.mapjoin.map.tasks=10000 -jobconf hive.mapjoin.maxsize=100000 -jobconf hive.archive.enabled=false -jobconf hive.aux.jars.path=file%3A%2F%2F%2Fhome%2Fsean%2Fprojects%2Fhive%2Fbuild%2Fjdbc%2Ftest%2Ftest-udfs.jar -jobconf hive.exec.dynamic.partition=false -jobconf hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.PreExecutePrinter%2C+org.apache.hadoop.hive.ql.hooks.EnforceReadOnlyTables -jobconf hive.optimize.skewjoin=false -jobconf hive.groupby.mapaggr.checkinterval=100000 -jobconf hive.test.mode=false -jobconf hive.exec.parallel=false -jobconf hive.exec.counters.pull.interval=1000 -jobconf hive.default.fileformat=TextFile -jobconf hive.exec.max.dynamic.partitions=1000 -jobconf fs.har.impl=org.apache.hadoop.hive.shims.HiveHarFileSystem -jobconf hive.test.mode.samplefreq=32 -jobconf hive.metastore.ds.retry.attempts=1 -jobconf javax.jdo.option.NonTransactionalRead=true -jobconf hive.script.auto.progress=false -jobconf hive.merge.mapredfiles=false -jobconf javax.jdo.option.ConnectionURL=jdbc%3Aderby%3A%3BdatabaseName%3D..%2Fbuild%2Ftest%2Fjunit_metastore_db%3Bcreate%3Dtrue -jobconf hive.exec.compress.intermediate=false -jobconf hive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore -jobconf hive.map.aggr.hash.percentmemory=0.5 -jobconf hive.hwi.listen.host=0.0.0.0 -jobconf datanucleus.transactionIsolation=read-committed -jobconf hive.merge.size.per.task=256000000 -jobconf datanucleus.autoCreateSchema=true -jobconf hive.exec.post.hooks=org.apache.hadoop.hive.ql.hooks.PostExecutePrinter -jobconf hive.groupby.skewindata=false -jobconf hive.metastore.local=true -jobconf hive.skewjoin.mapjoin.min.split=33554432 -jobconf hadoop.tmp.dir=%24%7Bbuild.dir.hive%7D%2Ftest%2Fhadoop%24%7Buser.name%7D -jobconf hive.mapred.mode=nonstrict -jobconf hive.optimize.pruner=true -jobconf hive.skewjoin.key=100000 -jobconf hive.default.partition.name=_HIVE_DEFAULT_PARTITION_ -jobconf hive.hbase.wal.enabled=true -jobconf datanucleus.validateColumns=false -jobconf datanucleus.identifierFactory=datanucleus -jobconf hive.querylog.location=%24%7Bbuild.dir%7D%2Ftmp -jobconf hive.optimize.reducededuplication=true -jobconf hive.exec.reducers.max=999 -jobconf javax.jdo.PersistenceManagerFactoryClass=org.datanucleus.jdo.JDOPersistenceManagerFactory -jobconf hive.heartbeat.interval=1000 -jobconf hive.join.cache.size=25000 -jobconf hive.metastore.warehouse.dir=%24%7Btest.warehouse.dir%7D -jobconf datanucleus.autoStartMechanismMode=checked -jobconf javax.jdo.option.ConnectionPassword=mine -jobconf hive.metastore.connect.retries=5 -jobconf hive.exec.mode.local.auto=false -jobconf hive.mapjoin.cache.numrows=25000 -jobconf hive.exec.parallel.thread.number=8 -jobconf datanucleus.storeManagerType=rdbms -jobconf hive.script.recordreader=org.apache.hadoop.hive.ql.exec.TextRecordReader -jobconf hive.exec.scratchdir=%24%7Bbuild.dir%7D%2Fscratchdir -jobconf hive.metastore.metadb.dir=file%3A%2F%2F%24%7Bbuild.dir%7D%2Ftest%2Fdata%2Fmetadb%2F -jobconf hive.metastore.server.min.threads=200 -jobconf hive.script.recordwriter=org.apache.hadoop.hive.ql.exec.TextRecordWriter -jobconf hive.merge.mapfiles=true -jobconf hive.exec.script.maxerrsize=100000 -jobconf test.query.file1=file%3A%2F%2F%24%7Buser.dir%7D%2F..%2Fql%2Fsrc%2Ftest%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Finput2.q -jobconf hive.join.emit.interval=1000 -jobconf hive.added.jars.path= -jobconf mapred.system.dir=%2Fhome%2Fsean%2Fprojects%2Fhive%2Fbuild%2Ftest%2Fhadoop-sean%2Fmapred%2Fsystem%2F1753935500 -jobconf mapred.local.dir=%2Fhome%2Fsean%2Fprojects%2Fhive%2Fbuild%2Ftest%2Fhadoop-sean%2Fmapred%2Flocal%2F-2115807459

          2010-08-14 17:32:25,633 ERROR exec.MapRedTask (MapRedTask.java:execute(224)) - Execution failed with exit status: 1
          2010-08-14 17:32:25,635 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

          Show
          Sean Flatley added a comment - The log entries can be found at the end of this comment. I did find in the Eclipse console an error entry reporting that JAVA_HOME is not set, so it does appear to be configuration: [junit] PREHOOK: query: select a,b,c,d,f as e,f*2 from testHiveJdbcDriverTable limit 1 [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivejdbcdrivertable [junit] PREHOOK: Output: file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-mr-10000 [junit] Total MapReduce jobs = 1 [junit] Launching Job 1 out of 1 [junit] Number of reduce tasks is set to 0 since there's no reduce operator [junit] Error: JAVA_HOME is not set. [junit] FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask I do have JAVA_HOME set and exported in my .bashrc file. Here is the hive.log file entries. 2010-08-14 17:32:24,929 INFO ql.Driver (Driver.java:execute(425)) - Starting command: select a,b,c,d,f as e,f*2 from testHiveJdbcDriverTable limit 1 2010-08-14 17:32:24,935 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: query: select a,b,c,d,f as e,f*2 from testHiveJdbcDriverTable limit 1 2010-08-14 17:32:24,937 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: type: QUERY 2010-08-14 17:32:24,938 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: Input: default@testhivejdbcdrivertable 2010-08-14 17:32:24,939 ERROR SessionState (SessionState.java:printError(277)) - PREHOOK: Output: file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-mr-10000 2010-08-14 17:32:24,940 INFO ql.Driver (SessionState.java:printInfo(268)) - Total MapReduce jobs = 1 2010-08-14 17:32:24,942 INFO ql.Driver (SessionState.java:printInfo(268)) - Launching Job 1 out of 1 2010-08-14 17:32:24,947 INFO exec.MapRedTask (SessionState.java:printInfo(268)) - Number of reduce tasks is set to 0 since there's no reduce operator 2010-08-14 17:32:24,967 INFO exec.MapRedTask (MapRedTask.java:execute(152)) - Generating plan file file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-local-10002/plan.xml 2010-08-14 17:32:25,588 INFO exec.MapRedTask (MapRedTask.java:execute(173)) - Executing: /home/sean/projects/hive/build/hadoopcore/hadoop-0.20.0/bin/hadoop jar /home/sean/projects/hive/build/ql/hive-exec-0.7.0.jar org.apache.hadoop.hive.ql.exec.ExecDriver libjars file:///home/sean/projects/hive/build/jdbc/test/test-udfs.jar -plan file:/tmp/sean/hive_2010-08-14_17-32-24_783_1792704952460259887/-local-10002/plan.xml -nolog -jobconf datanucleus.connectionPoolingType=DBCP -jobconf hive.exec.script.allow.partial.consumption=false -jobconf hive.query.id=sean_20100814173232_f49d776c-7273-4995-8cc3-f7aceabdbde3 -jobconf hive.hwi.listen.port=9999 -jobconf hive.map.aggr=true -jobconf hive.map.aggr.hash.min.reduction=0.5 -jobconf datanucleus.plugin.pluginRegistryBundleCheck=LOG -jobconf hive.exec.reducers.bytes.per.reducer=1000000000 -jobconf hive.optimize.cp=true -jobconf hive.exec.dynamic.partition.mode=strict -jobconf hive.merge.size.smallfiles.avgsize=16000000 -jobconf datanucleus.cache.level2.type=SOFT -jobconf hive.exec.max.created.files=100000 -jobconf hive.script.serde=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe -jobconf hive.fileformat.check=true -jobconf hive.exec.max.dynamic.partitions.pernode=100 -jobconf hive.enforce.sorting=false -jobconf hive.optimize.ppd=true -jobconf hive.optimize.groupby=true -jobconf hive.enforce.bucketing=false -jobconf javax.jdo.option.ConnectionUserName=APP -jobconf hive.mapred.reduce.tasks.speculative.execution=true -jobconf mapred.job.name=select+a%2Cb%2Cc%2Cd%2Cf+as+e%2Cf*2+from+testHiveJ...1%28Stage-1%29 -jobconf javax.jdo.option.DetachAllOnCommit=true -jobconf hive.mapred.local.mem=0 -jobconf datanucleus.cache.level2=false -jobconf hive.session.id=sean_201008141732 -jobconf fs.pfile.impl=org.apache.hadoop.fs.ProxyLocalFileSystem -jobconf hive.script.operator.id.env.var=HIVE_SCRIPT_OPERATOR_ID -jobconf hive.archive.har.parentdir.settable=false -jobconf hadoop.job.ugi=sean%2Csean%2Cadm%2Cdialout%2Ccdrom%2Cplugdev%2Clpadmin%2Cadmin%2Csambashare -jobconf test.src.dir=file%3A%2F%2F%24%7Bbuild.dir%7D%2Fsrc%2Ftest -jobconf hive.metastore.server.max.threads=100000 -jobconf hive.udtf.auto.progress=false -jobconf hive.hwi.war.file=lib%2Fhive-hwi %40VERSION%40.war jobconf datanucleus.validateTables=false -jobconf hive.exec.compress.output=false -jobconf hive.test.mode.prefix=test_ -jobconf hive.mapjoin.bucket.cache.size=100 -jobconf test.log.dir=%24%7Bbuild.dir%7D%2Ftest%2Flogs -jobconf test.data.files=%24%7Buser.dir%7D%2F..%2Fdata%2Ffiles -jobconf datanucleus.validateConstraints=false -jobconf hive.metastore.server.tcp.keepalive=true -jobconf mapred.reduce.tasks=-1 -jobconf hive.query.string=select+a%2Cb%2Cc%2Cd%2Cf+as+e%2Cf*2+from+testHiveJdbcDriverTable+limit+1 -jobconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat -jobconf hive.task.progress=false -jobconf hive.jar.path=%24%7Bbuild.dir.hive%7D%2Fql%2Fhive-exec %24%7Bversion%7D.jar jobconf hive.metastore.ds.retry.interval=1000 -jobconf javax.jdo.option.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver -jobconf hive.skewjoin.mapjoin.map.tasks=10000 -jobconf hive.mapjoin.maxsize=100000 -jobconf hive.archive.enabled=false -jobconf hive.aux.jars.path=file%3A%2F%2F%2Fhome%2Fsean%2Fprojects%2Fhive%2Fbuild%2Fjdbc%2Ftest%2Ftest-udfs.jar -jobconf hive.exec.dynamic.partition=false -jobconf hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.PreExecutePrinter%2C+org.apache.hadoop.hive.ql.hooks.EnforceReadOnlyTables -jobconf hive.optimize.skewjoin=false -jobconf hive.groupby.mapaggr.checkinterval=100000 -jobconf hive.test.mode=false -jobconf hive.exec.parallel=false -jobconf hive.exec.counters.pull.interval=1000 -jobconf hive.default.fileformat=TextFile -jobconf hive.exec.max.dynamic.partitions=1000 -jobconf fs.har.impl=org.apache.hadoop.hive.shims.HiveHarFileSystem -jobconf hive.test.mode.samplefreq=32 -jobconf hive.metastore.ds.retry.attempts=1 -jobconf javax.jdo.option.NonTransactionalRead=true -jobconf hive.script.auto.progress=false -jobconf hive.merge.mapredfiles=false -jobconf javax.jdo.option.ConnectionURL=jdbc%3Aderby%3A%3BdatabaseName%3D..%2Fbuild%2Ftest%2Fjunit_metastore_db%3Bcreate%3Dtrue -jobconf hive.exec.compress.intermediate=false -jobconf hive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore -jobconf hive.map.aggr.hash.percentmemory=0.5 -jobconf hive.hwi.listen.host=0.0.0.0 -jobconf datanucleus.transactionIsolation=read-committed -jobconf hive.merge.size.per.task=256000000 -jobconf datanucleus.autoCreateSchema=true -jobconf hive.exec.post.hooks=org.apache.hadoop.hive.ql.hooks.PostExecutePrinter -jobconf hive.groupby.skewindata=false -jobconf hive.metastore.local=true -jobconf hive.skewjoin.mapjoin.min.split=33554432 -jobconf hadoop.tmp.dir=%24%7Bbuild.dir.hive%7D%2Ftest%2Fhadoop %24%7Buser.name%7D -jobconf hive.mapred.mode=nonstrict -jobconf hive.optimize.pruner=true -jobconf hive.skewjoin.key=100000 -jobconf hive.default.partition.name=_ HIVE_DEFAULT_PARTITION _ -jobconf hive.hbase.wal.enabled=true -jobconf datanucleus.validateColumns=false -jobconf datanucleus.identifierFactory=datanucleus -jobconf hive.querylog.location=%24%7Bbuild.dir%7D%2Ftmp -jobconf hive.optimize.reducededuplication=true -jobconf hive.exec.reducers.max=999 -jobconf javax.jdo.PersistenceManagerFactoryClass=org.datanucleus.jdo.JDOPersistenceManagerFactory -jobconf hive.heartbeat.interval=1000 -jobconf hive.join.cache.size=25000 -jobconf hive.metastore.warehouse.dir=%24%7Btest.warehouse.dir%7D -jobconf datanucleus.autoStartMechanismMode=checked -jobconf javax.jdo.option.ConnectionPassword=mine -jobconf hive.metastore.connect.retries=5 -jobconf hive.exec.mode.local.auto=false -jobconf hive.mapjoin.cache.numrows=25000 -jobconf hive.exec.parallel.thread.number=8 -jobconf datanucleus.storeManagerType=rdbms -jobconf hive.script.recordreader=org.apache.hadoop.hive.ql.exec.TextRecordReader -jobconf hive.exec.scratchdir=%24%7Bbuild.dir%7D%2Fscratchdir -jobconf hive.metastore.metadb.dir=file%3A%2F%2F%24%7Bbuild.dir%7D%2Ftest%2Fdata%2Fmetadb%2F -jobconf hive.metastore.server.min.threads=200 -jobconf hive.script.recordwriter=org.apache.hadoop.hive.ql.exec.TextRecordWriter -jobconf hive.merge.mapfiles=true -jobconf hive.exec.script.maxerrsize=100000 -jobconf test.query.file1=file%3A%2F%2F%24%7Buser.dir%7D%2F..%2Fql%2Fsrc%2Ftest%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Finput2.q -jobconf hive.join.emit.interval=1000 -jobconf hive.added.jars.path= -jobconf mapred.system.dir=%2Fhome%2Fsean%2Fprojects%2Fhive%2Fbuild%2Ftest%2Fhadoop-sean%2Fmapred%2Fsystem%2F1753935500 -jobconf mapred.local.dir=%2Fhome%2Fsean%2Fprojects%2Fhive%2Fbuild%2Ftest%2Fhadoop-sean%2Fmapred%2Flocal%2F-2115807459 2010-08-14 17:32:25,633 ERROR exec.MapRedTask (MapRedTask.java:execute(224)) - Execution failed with exit status: 1 2010-08-14 17:32:25,635 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
          Hide
          Sean Flatley added a comment -

          Change log.

          Show
          Sean Flatley added a comment - Change log.
          Sean Flatley made changes -
          Attachment HIVE-1536-changes.txt [ 12452121 ]
          Hide
          Sean Flatley added a comment -

          Log file of: ant clean test tar -logfile ant.log

          Show
          Sean Flatley added a comment - Log file of: ant clean test tar -logfile ant.log
          Sean Flatley made changes -
          Attachment all-tests-ant.log [ 12452135 ]
          Hide
          Sean Flatley added a comment -

          Log file of: ant clean test tar -Dtestcase=JdbcDriverTest -logfile JdbcDriverTest-ant.log

          Show
          Sean Flatley added a comment - Log file of: ant clean test tar -Dtestcase=JdbcDriverTest -logfile JdbcDriverTest-ant.log
          Sean Flatley made changes -
          Attachment JdbcDriverTest-ant.log [ 12452136 ]
          Hide
          Sean Flatley added a comment -

          Patch.

          Show
          Sean Flatley added a comment - Patch.
          Sean Flatley made changes -
          Attachment HIVE-1536.patch [ 12452137 ]
          Hide
          Sean Flatley added a comment -

          The modified JdbcDriverTest test case succeeds. I have attached two log files for the ant commands:

          • ant clean test tar -logfile all-tests-ant.log
          • ant clean test tar -Dtestcase=JdbcDriverTest -logfile JdbcDriverTest-ant.log

          The second command works. Running the equivalent command in Eclipse using the "ant" interface results in the error where "JAVA_HOME" is not set.

          This appears to be an environmental issue and I feel that it should not hold up the contribution process. As out Sprint ends tomorrow I can circle back and take a look at my Linux environments. and nail down what is causing the existing tests to error.

          Show
          Sean Flatley added a comment - The modified JdbcDriverTest test case succeeds. I have attached two log files for the ant commands: ant clean test tar -logfile all-tests-ant.log ant clean test tar -Dtestcase=JdbcDriverTest -logfile JdbcDriverTest-ant.log The second command works. Running the equivalent command in Eclipse using the "ant" interface results in the error where "JAVA_HOME" is not set. This appears to be an environmental issue and I feel that it should not hold up the contribution process. As out Sprint ends tomorrow I can circle back and take a look at my Linux environments. and nail down what is causing the existing tests to error.
          Sean Flatley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Many methods in HivePreparedStatement have been implemented. This cases "change log" attachment summarizes all changes.
          John Sichi made changes -
          Assignee Sean Flatley [ sflatley@pentaho.com ]
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/663/
          -----------------------------------------------------------

          Review request for Hive Developers.

          Summary
          -------

          Submitted on behalf of Sean Flatley.

          This addresses bug HIVE-1536.
          http://issues.apache.org/jira/browse/HIVE-1536

          Diffs


          trunk/data/files/pstmt.txt PRE-CREATION
          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java 985606
          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 985606
          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java 985606
          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 985606
          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveResultSetMetaData.java 985606
          trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 985606

          Diff: http://review.cloudera.org/r/663/diff

          Testing
          -------

          Thanks,

          Carl

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/663/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- Submitted on behalf of Sean Flatley. This addresses bug HIVE-1536 . http://issues.apache.org/jira/browse/HIVE-1536 Diffs trunk/data/files/pstmt.txt PRE-CREATION trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java 985606 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 985606 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java 985606 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 985606 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveResultSetMetaData.java 985606 trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 985606 Diff: http://review.cloudera.org/r/663/diff Testing ------- Thanks, Carl
          Hide
          John Sichi added a comment -

          First pass of review comments added on Review Board. I may have more in a second pass.

          Show
          John Sichi added a comment - First pass of review comments added on Review Board. I may have more in a second pass.
          Hide
          HBase Review Board added a comment -

          Message from: "John Sichi" <jsichi@facebook.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/663/#review928
          -----------------------------------------------------------

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java
          <http://review.cloudera.org/r/663/#comment3025>

          Looks like this got added back in by merge mistake.

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java
          <http://review.cloudera.org/r/663/#comment3026>

          I don't understand this if statement...we do the same thing anyway below?

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java
          <http://review.cloudera.org/r/663/#comment3027>

          Shouldn't we prevent setAutoCommit(false) rather than setAutoCommit(true)? Hive does not support user-level transactions.

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java
          <http://review.cloudera.org/r/663/#comment3028>

          Where did the number 100 come from?

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java
          <http://review.cloudera.org/r/663/#comment3029>

          This should be a backtick, not an apostrophe.

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java
          <http://review.cloudera.org/r/663/#comment3030>

          This is inaccurate. The database is read/write since you can issue INSERT statements etc.

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java
          <http://review.cloudera.org/r/663/#comment3032>

          This should also close the resultSet (if there is one) before nullifying it.

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java
          <http://review.cloudera.org/r/663/#comment3034>

          For PreparedStatement, public executeQuery(sql) and others inherited from vanilla Statement should throw, since they aren't supposed to take the SQL string as parameter (that's what prepare is for).

          Instead, use a protected helper method with a different name (and share with HiveStatement so that excn handling code is reused).

          trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java
          <http://review.cloudera.org/r/663/#comment3035>

          Where is the maxRows enforcement implemented? I may have missed it. Ideally we'd like to turn this into a LIMIT clause for efficiency, but we can do a followup for that.

          • John
          Show
          HBase Review Board added a comment - Message from: "John Sichi" <jsichi@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/663/#review928 ----------------------------------------------------------- trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java < http://review.cloudera.org/r/663/#comment3025 > Looks like this got added back in by merge mistake. trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java < http://review.cloudera.org/r/663/#comment3026 > I don't understand this if statement...we do the same thing anyway below? trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java < http://review.cloudera.org/r/663/#comment3027 > Shouldn't we prevent setAutoCommit(false) rather than setAutoCommit(true)? Hive does not support user-level transactions. trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java < http://review.cloudera.org/r/663/#comment3028 > Where did the number 100 come from? trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java < http://review.cloudera.org/r/663/#comment3029 > This should be a backtick, not an apostrophe. trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java < http://review.cloudera.org/r/663/#comment3030 > This is inaccurate. The database is read/write since you can issue INSERT statements etc. trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java < http://review.cloudera.org/r/663/#comment3032 > This should also close the resultSet (if there is one) before nullifying it. trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java < http://review.cloudera.org/r/663/#comment3034 > For PreparedStatement, public executeQuery(sql) and others inherited from vanilla Statement should throw, since they aren't supposed to take the SQL string as parameter (that's what prepare is for). Instead, use a protected helper method with a different name (and share with HiveStatement so that excn handling code is reused). trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java < http://review.cloudera.org/r/663/#comment3035 > Where is the maxRows enforcement implemented? I may have missed it. Ideally we'd like to turn this into a LIMIT clause for efficiency, but we can do a followup for that. John
          John Sichi made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Sean Flatley added a comment -

          JdbcDriverTest test case result for HIVE-1536-2.patch.

          Show
          Sean Flatley added a comment - JdbcDriverTest test case result for HIVE-1536 -2.patch.
          Sean Flatley made changes -
          Attachment JdbcDriverTest-ant-2.log [ 12452933 ]
          Hide
          Sean Flatley added a comment -

          Change log for HIVE-1536-2.patch

          Show
          Sean Flatley added a comment - Change log for HIVE-1536 -2.patch
          Sean Flatley made changes -
          Attachment HIVE-1536-changes-2.txt [ 12452934 ]
          Hide
          Sean Flatley added a comment -

          Patch file #2.

          Show
          Sean Flatley added a comment - Patch file #2.
          Sean Flatley made changes -
          Attachment HIVE-1536-2.patch [ 12452935 ]
          Hide
          Sean Flatley added a comment -

          In this second go around most of the code was reverted, Please see the change log for details.

          Change log: HIVE-1536-changes-2.txt
          Junit log file : JdbcDriverTest-ant-2.log
          Patch: HIVE-1536-2.patch

          Show
          Sean Flatley added a comment - In this second go around most of the code was reverted, Please see the change log for details. Change log: HIVE-1536 -changes-2.txt Junit log file : JdbcDriverTest-ant-2.log Patch: HIVE-1536 -2.patch
          Sean Flatley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          John Sichi made changes -
          Fix Version/s 0.7.0 [ 12315150 ]
          Affects Version/s 0.6.0 [ 12314524 ]
          Hide
          John Sichi added a comment -

          Thanks Sean. I have two more items and then I think this one is ready for commit.

          • for the column name length, we should use 128, since that's the length declared for MFieldSchema.FNAME in metastore/src/model/package.jdo
          • for PreparedStatement.execute, you can get rid of the TODO comment; the behavior is correct as is (the true/false return only discriminates between a query with a cursor vs DDL/DML; it's not related to whether any rows are actually returned by the query)
          Show
          John Sichi added a comment - Thanks Sean. I have two more items and then I think this one is ready for commit. for the column name length, we should use 128, since that's the length declared for MFieldSchema.FNAME in metastore/src/model/package.jdo for PreparedStatement.execute, you can get rid of the TODO comment; the behavior is correct as is (the true/false return only discriminates between a query with a cursor vs DDL/DML; it's not related to whether any rows are actually returned by the query)
          John Sichi made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Sean Flatley added a comment -

          Revision 3 change log.

          Show
          Sean Flatley added a comment - Revision 3 change log.
          Sean Flatley made changes -
          Attachment HIVE-1536-changes-3.txt [ 12453142 ]
          Hide
          Sean Flatley added a comment -

          Unit test results for revision 3.

          Show
          Sean Flatley added a comment - Unit test results for revision 3.
          Sean Flatley made changes -
          Attachment TestJdbcDriver-ant-3.log [ 12453143 ]
          Hide
          Sean Flatley added a comment -

          Revision 3 patch.

          Show
          Sean Flatley added a comment - Revision 3 patch.
          Sean Flatley made changes -
          Attachment HIVE-1536-3.patch [ 12453144 ]
          Hide
          Sean Flatley added a comment -

          Please see the change log for details.

          Change log: HIVE-1536-changes-3.txt
          Junit log file : TestJdbcDriver-ant-3.log
          Patch: HIVE-1536-3.patch

          Show
          Sean Flatley added a comment - Please see the change log for details. Change log: HIVE-1536 -changes-3.txt Junit log file : TestJdbcDriver-ant-3.log Patch: HIVE-1536 -3.patch
          Sean Flatley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          John Sichi added a comment -

          +1. Will commit when tests pass.

          Show
          John Sichi added a comment - +1. Will commit when tests pass.
          Hide
          John Sichi added a comment -

          Committed. Thanks Sean!

          Show
          John Sichi added a comment - Committed. Thanks Sean!
          John Sichi made changes -
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Carl Steinbach made changes -
          Component/s JDBC [ 12314178 ]
          Component/s Drivers [ 12313584 ]
          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Sean Flatley
              Reporter:
              Sean Flatley
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development