[HDDS-8496] Hive with s3a connector for OBS bucket hits error

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Component/s: Ozone Manager
• Labels: proton pull-request-available
• Assignee: Duong
• Reporter: Saketa Chalamchala
• Created: 26/Apr/23 21:15
• Updated: 08/Aug/23 01:57
• Resolved: 08/Aug/23 01:57

Description

Error: seen

	... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: 
[s3a://obs-bucket-link/mytable1/.hive-staging_hive_2023-04-23_07-38-52_965_8384866535295320062-1/_task_tmp.-ext-10000/_tmp.000000_3]
 to: 
[s3a://obs-bucket-link/mytable1/.hive-staging_hive_2023-04-23_07-38-52_965_8384866535295320062-1/_tmp.-ext-10000/000000_3]
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commitOneOutPath(FileSinkOperator.java:296)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:254)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$400(FileSinkOperator.java:157)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1458)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:731)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:482)
	... 17 more

 

Steps to reproduce 

 

ozone volume create /vol1
ozone sh bucket create /vol1/obs-bucket --layout OBJECT_STORE
ozone sh bucket link /vol1/obs-bucket /s3v/obs-bucket-link

beeline -u "jdbc:
[hive2://<host>:10000/default;principal=hive/<host>@<REALM>;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;sslTrustStorePassword=<redacted>]
"

> create external table mytable1(key string, value int) location '
[s3a://obs-bucket-link/mytable1]
';
> insert into mytable1 values("cldr",1);

Issue Links

• is caused by: HDDS-7223 Caching container location information in OM
• links to: GitHub Pull Request #4647