Hive
  1. Hive
  2. HIVE-7090

Support session-level temporary tables in Hive

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: SQL
    • Labels:
      None

      Description

      It's common to see sql scripts that create some temporary table as an intermediate result, run some additional queries against it and then clean up at the end.

      We should support temporary tables properly, meaning automatically manage the life cycle and make sure the visibility is restricted to the creating connection/session. Without these it's common to see left over tables in meta-store or weird errors with clashing tmp table names.

      Proposed syntax:

      CREATE TEMPORARY TABLE ....

      CTAS, CTL, INSERT INTO, should all be supported as usual.

      Knowing that a user wants a temp table can enable us to further optimize access to it. E.g.: temp tables should be kept in memory where possible, compactions and merging table files aren't required, ...

      1. HIVE-7090.1.patch
        85 kB
        Gunther Hagleitner
      2. HIVE-7090.2.patch
        96 kB
        Jason Dere
      3. HIVE-7090.3.patch
        96 kB
        Jason Dere
      4. HIVE-7090.4.patch
        208 kB
        Jason Dere
      5. HIVE-7090.5.patch
        213 kB
        Jason Dere
      6. HIVE-7090.6.patch
        212 kB
        Jason Dere
      7. HIVE-7090.7.patch
        214 kB
        Jason Dere
      8. HIVE-7090.8.patch
        218 kB
        Jason Dere
      9. HIVE-7090.9.patch
        218 kB
        Jason Dere

        Issue Links

          Activity

          Hide
          Gunther Hagleitner added a comment -

          Implementation:

          Idea is simple:

          • Keep metadata for tmp table in memory tied to the SessionState (will result in the right visibility of the table)
          • Set location of tables to use the session scratch dir (automatic cleanup when session is closed)
          Show
          Gunther Hagleitner added a comment - Implementation: Idea is simple: Keep metadata for tmp table in memory tied to the SessionState (will result in the right visibility of the table) Set location of tables to use the session scratch dir (automatic cleanup when session is closed)
          Hide
          Gunther Hagleitner added a comment -

          Initial patch. Mostly done by Harish Butani. I've just added session scratch dirs and a temp table space.

          Show
          Gunther Hagleitner added a comment - Initial patch. Mostly done by Harish Butani . I've just added session scratch dirs and a temp table space.
          Hide
          Navis added a comment -

          Interesting. I've tried something similar but rather similar with variables in pig than a temporary table. +1 for idea.

          Show
          Navis added a comment - Interesting. I've tried something similar but rather similar with variables in pig than a temporary table. +1 for idea.
          Hide
          Jason Dere added a comment -

          Patch v2 moves the management of the temp tables completely to the client side. So changes are to HiveMetaStoreClient, rather than at the ObjectStore. Still needs more testing.

          Show
          Jason Dere added a comment - Patch v2 moves the management of the temp tables completely to the client side. So changes are to HiveMetaStoreClient, rather than at the ObjectStore. Still needs more testing.
          Hide
          Eugene Koifman added a comment -

          If the client fails, how does the temp table get cleaned up?

          Show
          Eugene Koifman added a comment - If the client fails, how does the temp table get cleaned up?
          Hide
          Jason Dere added a comment -

          The temp table scratch directory is deleted during session close, and also marked for deletion upon process close, which should clean up the directory for normal usage.
          If the client dies, this cleanup does not occur and the directory is left in the user's scratch directory. For HiveServer2, we could try to add cleanup thread to remove old temp table directories from the scratch directory. For other users like HiveCLI, there would probably not be any automated cleanup, similar to other stuff that could get left around in the user's scratch directory.

          Show
          Jason Dere added a comment - The temp table scratch directory is deleted during session close, and also marked for deletion upon process close, which should clean up the directory for normal usage. If the client dies, this cleanup does not occur and the directory is left in the user's scratch directory. For HiveServer2, we could try to add cleanup thread to remove old temp table directories from the scratch directory. For other users like HiveCLI, there would probably not be any automated cleanup, similar to other stuff that could get left around in the user's scratch directory.
          Hide
          Eugene Koifman added a comment -

          In that case it may make sense to generate unique names for artifacts that may be left over. The initial description in this ticket mentions 3rd party tools that will use this feature - I imagine they will generate the same Temp table name each time which may cause weird failures after crash.

          Show
          Eugene Koifman added a comment - In that case it may make sense to generate unique names for artifacts that may be left over. The initial description in this ticket mentions 3rd party tools that will use this feature - I imagine they will generate the same Temp table name each time which may cause weird failures after crash.
          Hide
          Jason Dere added a comment -

          Yes good point. The patch actually does this - each session will have its own scratch directory for temp tables, using the session ID (a UUID). Within the session's temp table scratch directory, each created temp table will get its own directory, also generated using UUID.

          Show
          Jason Dere added a comment - Yes good point. The patch actually does this - each session will have its own scratch directory for temp tables, using the session ID (a UUID). Within the session's temp table scratch directory, each created temp table will get its own directory, also generated using UUID.
          Hide
          Jason Dere added a comment -

          rebase with trunk

          Show
          Jason Dere added a comment - rebase with trunk
          Hide
          Navis added a comment -

          Submitting for test, let's see the result.

          Show
          Navis added a comment - Submitting for test, let's see the result.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652086/HIVE-7090.3.patch

          ERROR: -1 due to 59 failed/errored test(s), 5670 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_cube1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19
          org.apache.hadoop.hive.cli.TestCliSessionState.testConnect
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testBuildDag
          org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testEmptyWork
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLNoLock
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLShared
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testJoin
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testReadWrite
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadMultiPartition
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadTable
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable
          org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate
          org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testDataDeletion
          org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
          org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testCannotUseReservedWordAsName
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacro
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroDoesNotExist
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroExistsDoNotIgnoreErrors
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistent
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExists
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExistsDoNotIgnoreNonExistent
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testNoBody
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testOneInputParamters
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testOneUnusedParameterName
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testThreeDuplicateParameters
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testThreeInputParamters
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoDuplicateParameterNames
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoInputParamters
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoUnusedParameterNames
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testUnknownInputParameter
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testZeroInputParamters
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testCreateRole
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testDropRole
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantGroupTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleGroup
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleRole
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleUser
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantUserTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeGroupTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleGroup
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleRole
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleUser
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeUserTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowGrantGroupOnTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowGrantRoleOnTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowGrantUserOnTable
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowRoleGrantGroup
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowRoleGrantRole
          org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowRoleGrantUser
          org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.testOutputFormat
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/576/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/576/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-576/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 59 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12652086

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652086/HIVE-7090.3.patch ERROR: -1 due to 59 failed/errored test(s), 5670 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_cube1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19 org.apache.hadoop.hive.cli.TestCliSessionState.testConnect org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testBuildDag org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testEmptyWork org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLNoLock org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLShared org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testJoin org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testReadWrite org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadMultiPartition org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadTable org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testDataDeletion org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testCannotUseReservedWordAsName org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacro org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroDoesNotExist org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroExistsDoNotIgnoreErrors org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistent org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExists org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExistsDoNotIgnoreNonExistent org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testNoBody org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testOneInputParamters org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testOneUnusedParameterName org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testThreeDuplicateParameters org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testThreeInputParamters org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoDuplicateParameterNames org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoInputParamters org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoUnusedParameterNames org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testUnknownInputParameter org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testZeroInputParamters org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testCreateRole org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testDropRole org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantGroupTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleGroup org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleRole org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleUser org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantUserTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeGroupTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleGroup org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleRole org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleUser org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeUserTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowGrantGroupOnTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowGrantRoleOnTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowGrantUserOnTable org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowRoleGrantGroup org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowRoleGrantRole org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testShowRoleGrantUser org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.testOutputFormat Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/576/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/576/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-576/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 59 tests failed This message is automatically generated. ATTACHMENT ID: 12652086
          Hide
          Jason Dere added a comment -

          Yeah, still needs more testing. Looks like the failures are due to the scratch directory changes in the patch.

          Show
          Jason Dere added a comment - Yeah, still needs more testing. Looks like the failures are due to the scratch directory changes in the patch.
          Hide
          Jason Dere added a comment -

          Attaching patch v4.
          Most of the precommit test failures were due to the scratch directory changes. As a result ql.Context must now be be created after SessionState.startSession() is called. The failing tests have been updated.
          Also added more tests and some fixes based on the tests.

          Show
          Jason Dere added a comment - Attaching patch v4. Most of the precommit test failures were due to the scratch directory changes. As a result ql.Context must now be be created after SessionState.startSession() is called. The failing tests have been updated. Also added more tests and some fixes based on the tests.
          Hide
          Jason Dere added a comment -

          RB (minus generated files) at https://reviews.apache.org/r/22996/

          Show
          Jason Dere added a comment - RB (minus generated files) at https://reviews.apache.org/r/22996/
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652537/HIVE-7090.4.patch

          ERROR: -1 due to 8 failed/errored test(s), 5685 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformat
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_db_table
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_delimited
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/592/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/592/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-592/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12652537

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652537/HIVE-7090.4.patch ERROR: -1 due to 8 failed/errored test(s), 5685 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformat org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_db_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_delimited org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/592/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/592/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-592/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12652537
          Hide
          Jason Dere added a comment -

          Patch v5. The show create table tests were failing due to formatting (1 space difference) change in show create table, updated the diffs.
          stats19.q was failing because the scratchdir changes included adding the session ID to the path, which cause some pathnames to exceed the max prefix length used in the tests. Test seems a bit fragile since it's dependent on the path length of the build directory, but increasing the max prefix length before hashing should get this to pass.

          Show
          Jason Dere added a comment - Patch v5. The show create table tests were failing due to formatting (1 space difference) change in show create table, updated the diffs. stats19.q was failing because the scratchdir changes included adding the session ID to the path, which cause some pathnames to exceed the max prefix length used in the tests. Test seems a bit fragile since it's dependent on the path length of the build directory, but increasing the max prefix length before hashing should get this to pass.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652594/HIVE-7090.5.patch

          ERROR: -1 due to 1 failed/errored test(s), 5670 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/604/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/604/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-604/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12652594

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652594/HIVE-7090.5.patch ERROR: -1 due to 1 failed/errored test(s), 5670 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/604/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/604/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-604/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12652594
          Hide
          Jason Dere added a comment -

          Patch v6, changes based on Brock's comments.

          Show
          Jason Dere added a comment - Patch v6, changes based on Brock's comments.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652926/HIVE-7090.6.patch

          ERROR: -1 due to 2 failed/errored test(s), 5686 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/623/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/623/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-623/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12652926

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652926/HIVE-7090.6.patch ERROR: -1 due to 2 failed/errored test(s), 5686 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/623/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/623/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-623/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12652926
          Hide
          Alan Gates added a comment -

          A few comments left on review board.

          Show
          Alan Gates added a comment - A few comments left on review board.
          Hide
          Jason Dere added a comment -

          Brock Noland, does the patch look okay?

          Show
          Jason Dere added a comment - Brock Noland , does the patch look okay?
          Hide
          Jason Dere added a comment -

          I don't think the failure in TestHCatLoader is related, this passes locally for me

          Show
          Jason Dere added a comment - I don't think the failure in TestHCatLoader is related, this passes locally for me
          Hide
          Alan Gates added a comment -

          I believe we need to solve the views issue, as being able to create a view on a table when others can see the view and not the table is bogus.

          Other than that I'm +1 on the patch.

          Show
          Alan Gates added a comment - I believe we need to solve the views issue, as being able to create a view on a table when others can see the view and not the table is bogus. Other than that I'm +1 on the patch.
          Hide
          Jason Dere added a comment -

          Will add the view check as a followup item, I think this can be done during semantic analysis of the view creation.

          Show
          Jason Dere added a comment - Will add the view check as a followup item, I think this can be done during semantic analysis of the view creation.
          Hide
          Jason Dere added a comment -

          Patch v7, with more feedback from brock. Also rebased with trunk.

          Show
          Jason Dere added a comment - Patch v7, with more feedback from brock. Also rebased with trunk.
          Hide
          Selina Zhang added a comment -

          Just wonder how transaction works with temp table if we plan to add transaction management on top of it, since the meta data stores in the client side.

          Show
          Selina Zhang added a comment - Just wonder how transaction works with temp table if we plan to add transaction management on top of it, since the meta data stores in the client side.
          Hide
          Jason Dere added a comment -

          Are you referring to the ACID work that was done with HIVE-5317? My impression was that it would not be applicable for temp tables, because the temp table is only visible to the current session. 2 different sessions writing to a temp table with the same name will be updating their own separate version of the table, and will be unaffected by any updates from the other session.

          Show
          Jason Dere added a comment - Are you referring to the ACID work that was done with HIVE-5317 ? My impression was that it would not be applicable for temp tables, because the temp table is only visible to the current session. 2 different sessions writing to a temp table with the same name will be updating their own separate version of the table, and will be unaffected by any updates from the other session.
          Hide
          Selina Zhang added a comment -

          I mean if we want to support ROLLBACK/COMMIT in next release, how do we rollback the changes for temp table. It seems server does not have a clue where the data location is.

          Show
          Selina Zhang added a comment - I mean if we want to support ROLLBACK/COMMIT in next release, how do we rollback the changes for temp table. It seems server does not have a clue where the data location is.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12653646/HIVE-7090.7.patch

          ERROR: -1 due to 2 failed/errored test(s), 5673 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/659/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/659/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-659/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12653646

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12653646/HIVE-7090.7.patch ERROR: -1 due to 2 failed/errored test(s), 5673 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/659/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/659/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-659/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12653646
          Hide
          Alan Gates added a comment -

          Rolling back doesn't actually change any data. What is does is mark a transaction id as aborted. Then readers know to ignore records from that transaction id. So consider the following scenario:

          begin session
          begin transaction 1
          write to temp table
          commit
          begin transaction 2
          write more to temp table
          rollback
          read temp table

          The read will know to disregard all records marked with transaction id 2 (this holds whether the table is temporary or not) and thus will only return records from the first write.

          Show
          Alan Gates added a comment - Rolling back doesn't actually change any data. What is does is mark a transaction id as aborted. Then readers know to ignore records from that transaction id. So consider the following scenario: begin session begin transaction 1 write to temp table commit begin transaction 2 write more to temp table rollback read temp table The read will know to disregard all records marked with transaction id 2 (this holds whether the table is temporary or not) and thus will only return records from the first write.
          Hide
          Selina Zhang added a comment -

          Thanks for the explanation! It will work. Somehow I got a wrong impression ACID needs store addition information along with data location.

          Show
          Selina Zhang added a comment - Thanks for the explanation! It will work. Somehow I got a wrong impression ACID needs store addition information along with data location.
          Hide
          Jason Dere added a comment -

          Patch v8 fixes authorization for temporary table, and adds positive authorization test. Previously just had negative auth test.

          Show
          Jason Dere added a comment - Patch v8 fixes authorization for temporary table, and adds positive authorization test. Previously just had negative auth test.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12654353/HIVE-7090.8.patch

          ERROR: -1 due to 2 failed/errored test(s), 5703 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/699/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/699/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-699/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12654353

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12654353/HIVE-7090.8.patch ERROR: -1 due to 2 failed/errored test(s), 5703 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/699/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/699/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-699/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12654353
          Hide
          Jason Dere added a comment -

          Failures do not look related to patch.
          Brock Noland does this one look good to go now?

          Show
          Jason Dere added a comment - Failures do not look related to patch. Brock Noland does this one look good to go now?
          Hide
          Brock Noland added a comment -

          Thank you Jason!! +1

          Show
          Brock Noland added a comment - Thank you Jason!! +1
          Hide
          Jason Dere added a comment -

          patch v9 - rebase with trunk (testconfiguration.properties)

          Show
          Jason Dere added a comment - patch v9 - rebase with trunk (testconfiguration.properties)
          Hide
          Jason Dere added a comment -

          Committed to trunk. Thanks Brock for reviewing.

          Show
          Jason Dere added a comment - Committed to trunk. Thanks Brock for reviewing.
          Hide
          Lefty Leverenz added a comment -

          Could we please have a release note with all the new syntax that needs to be documented?

          Show
          Lefty Leverenz added a comment - Could we please have a release note with all the new syntax that needs to be documented?
          Hide
          Lefty Leverenz added a comment -

          Oh, I see Jason Dere already documented this. Thanks Jason, well done.

          Show
          Lefty Leverenz added a comment - Oh, I see Jason Dere already documented this. Thanks Jason, well done. DDL – Temporary Tables DDL – CREATE TABLE and CREATE TABLE LIKE syntax
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

            People

            • Assignee:
              Jason Dere
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development