[HIVE-28287] Attempt make the scratch directory writable before failing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.1.0, 4.0.1
• Component/s: Hive
• Labels: hive-4.0.1-merged pull-request-available
• Assignee: Ayush Saxena
• Reporter: Agnes Tevesz
• Created: 29/May/24 22:04
• Updated: 23/Sep/24 11:24
• Resolved: 30/May/24 16:30

Description

When hive is starting up it checks the tmp/hive directory privileges. Even if the Azure managed identity have write access on an Azure storage account, the rwx-wx-wx privileges still enforced, so in case rwxr-xr-x is set, the hive process startup will fail.

See the logs of a startup that resulted in failure.

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.hive.common.StringInternUtils (file:/usr/lib/hive/lib/hive-common-3.1.3000.2023.0.16.0-142.jar) to field java.net.URI.string
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.hive.common.StringInternUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" java.lang.RuntimeException: Error applying authorization policy on hive configuration: The dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x
    at org.apache.hive.service.cli.CLIService.init(CLIService.java:121)
    at org.apache.hive.service.cli.thrift.EmbeddedThriftBinaryCLIService.init(EmbeddedThriftBinaryCLIService.java:63)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:357)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:287)
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
    at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:88)
    at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:103)
    at org.apache.hadoop.hive.metastore.CDHMetaStoreSchemaInfo.getMetaStoreSchemaVersion(CDHMetaStoreSchemaInfo.java:323)
    at org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInitOrUpgrade.execute(SchemaToolTaskInitOrUpgrade.java:41)
    at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:482)
    at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:143)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.lang.RuntimeException: The dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x
    at org.apache.hadoop.hive.ql.exec.Utilities.ensurePathIsWritable(Utilities.java:5088)
    at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:896)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:837)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:749)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:708)
    at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:133)
    at org.apache.hive.service.cli.CLIService.init(CLIService.java:118)
    ... 18 more
Information schema initialization failed!
+ '[' 1 -eq 0 ']'
+ echo 'Information schema initialization failed!'
+ exit 1

To overcome this issue one approach could be to try to set the tmp/hive folder privileges to rwx-wx-wx before failing the startup.

Issue Links

• links to: GitHub Pull Request #5268