Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
-
None
-
ghx-label-9
Description
Hive complains 'Illegal location for managed table' although location '/test-warehouse/managed/insert_only_transactional_table' appears to be within database's managed location.
INFO : Compiling command(queryId=jenkins_20211116193803_e342124e-7a94-4024-b11a-58578cdf2ce4): CREATE TABLE IF NOT EXISTS functional.insert_only_transactional_table ( col1 int ) STORED AS TEXTFILE LOCATION '/test-warehouse/managed/insert_only_transactional_table' TBLPROPERTIES ( 'transactional_properties' = 'insert_only', 'transactional' = 'true' ) INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=jenkins_20211116193803_e342124e-7a94-4024-b11a-58578cdf2ce4); Time taken: 0.025 seconds INFO : Executing command(queryId=jenkins_20211116193803_e342124e-7a94-4024-b11a-58578cdf2ce4): CREATE TABLE IF NOT EXISTS functional.insert_only_transactional_table ( col1 int ) STORED AS TEXTFILE LOCATION '/test-warehouse/managed/insert_only_transactional_table' TBLPROPERTIES ( 'transactional_properties' = 'insert_only', 'transactional' = 'true' ) INFO : Starting task [Stage-0:DDL] in serial mode ERROR : Failed org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Illegal location for managed table, it has to be within database's managed location) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1170) ~[hive-exec-3.1.3000.7.1.8.0-393.jar:3.1.3000.7.1.8.0-393] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1175) ~[hive-exec-3.1.3000.7.1.8.0-393.jar:3.1.3000.7.1.8.0-393]
(For full hive stack see attachment)
HMS log:
2021-11-16T19:38:03,185 INFO [pool-9-thread-58] metastore.MetastoreDefaultTransformer: Starting translation for transformDatabase for processor HMSClient-@localhost with [EXTWRITE, EXTREAD, HIVEBUCKET2, HIVEFULLACIDREAD, HIVEFULLACIDWRITE, HIVECACHEINVALIDATE, HIVEMANAGESTATS, HIVEMANAGEDINSERTWRITE, HIVEMANAGEDINSERTREAD, HIVESQL, HIVEMQT, HIVEONLYMQTWRITE] on database functional locationUri=hdfs://localhost:20500/test-warehouse/functional.db managedLocationUri=hdfs://localhost:20500/test-warehouse/managed/functional.db 2021-11-16T19:38:03,185 INFO [pool-9-thread-58] metastore.MetastoreDefaultTransformer: Transformer returning database:Database(name:functional, description:null, locationUri:hdfs://localhost:20500/test-warehouse/functional.db, parameters:{}, ownerName:jenkins, ownerType:USER, catalogName:hive, createTime:1637119984, managedLocationUri:hdfs://localhost:20500/test-warehouse/managed/functional.db) 2021-11-16T19:38:03,323 INFO [pool-9-thread-58] metastore.HiveMetaStore: 63: source:127.0.0.1 create_table_req: Table(tableName:insert_only_transactional_table, dbName:functional, owner:jenkins, createTime:1637120283, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col1, type:int, comment:null)], location:hdfs://localhost:20500/test-warehouse/managed/insert_only_transactional_table, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{bucketing_version=2, transactional_properties=insert_only, transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{jenkins=[PrivilegeGrantInfo(privilege:INSERT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:SELECT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:UPDATE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:DELETE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true)]}, groupPrivileges:null, rolePrivileges:null), temporary:false, catName:hive, ownerType:USER, writeId:0) 2021-11-16T19:38:03,323 INFO [pool-9-thread-58] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 create_table_req: Table(tableName:insert_only_transactional_table, dbName:functional, owner:jenkins, createTime:1637120283, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col1, type:int, comment:null)], location:hdfs://localhost:20500/test-warehouse/managed/insert_only_transactional_table, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{bucketing_version=2, transactional_properties=insert_only, transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{jenkins=[PrivilegeGrantInfo(privilege:INSERT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:SELECT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:UPDATE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:DELETE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true)]}, groupPrivileges:null, rolePrivileges:null), temporary:false, catName:hive, ownerType:USER, writeId:0) 2021-11-16T19:38:03,323 INFO [pool-9-thread-58] metastore.MetastoreDefaultTransformer: Starting translation for CreateTable for processor HMSClient-@localhost with [EXTWRITE, EXTREAD, HIVEBUCKET2, HIVEFULLACIDREAD, HIVEFULLACIDWRITE, HIVECACHEINVALIDATE, HIVEMANAGESTATS, HIVEMANAGEDINSERTWRITE, HIVEMANAGEDINSERTREAD, HIVESQL, HIVEMQT, HIVEONLYMQTWRITE] on table insert_only_transactional_table 2021-11-16T19:38:03,327 ERROR [pool-9-thread-58] metastore.RetryingHMSHandler: MetaException(message:Illegal location for managed table, it has to be within database's managed location) at org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.validateTablePaths(MetastoreDefaultTransformer.java:886) at org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:666) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2234) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2544) at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at com.sun.proxy.$Proxy34.create_table_req(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16877) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16856) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)