Description
long getNextSequence(byte[] sequence) throws IOException {
Is not safe in presence of any concurrency. It should use HBase increment API.
Public signup for this instance is disabled. Go to our Self serve sign up page to request an account. Report potential security issues privately
long getNextSequence(byte[] sequence) throws IOException {
Is not safe in presence of any concurrency. It should use HBase increment API.
My original plan for this was to make use the transaction server that we'll need to add for HBase metastore anyway (most likely Omid). But given that we see a lot of contention on the sequences in the RDBMS world in ACID it might make sense to rely on HBase simpler check and put rather than the transaction service here.
This patch does a couple of things:
Hmm. Can increment API be used in the txn managers? that is separate from checkAndPut. See the patch at HIVE-12892
See comments on HIVE-12892 on why increment won't work in that case. In this case where I'm explicitly circumventing the transaction management it would be viable. Is it better than checkAndPut? I don't care which we use here.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786166/HIVE-12927.patch
SUCCESS: +1 due to 1 test(s) being added or modified.
ERROR: -1 due to 9 failed/errored test(s), 10052 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.jdbc.TestSSL.testSSLVersion
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6876/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6876/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6876/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed
This message is automatically generated.
ATTACHMENT ID: 12786166 - PreCommit-HIVE-TRUNK-Build
Based on the discussion over on HIVE-12892 I'm going to drop all the goofiness here about raw tables that circumvent the transaction manager. I think that will be more trouble than it's worth.
But I still need to fix it so that we have one sequence per row not one per column. I'll change the JIRA title and description and post a new patch.
New patch that corrects issues with sequences to be one per row instead of one per column. Also switched new CURRENT_VERSION table to be sequence based instead of its own table.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788163/HIVE-12927.2.patch
SUCCESS: +1 due to 1 test(s) being added or modified.
ERROR: -1 due to 3 failed/errored test(s), 9805 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hive.jdbc.TestSSL.testSSLVersion
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7012/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7012/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7012/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed
This message is automatically generated.
ATTACHMENT ID: 12788163 - PreCommit-HIVE-TRUNK-Build
gates daijy fyi