[HIVE-27447] Iceberg: Queries failing due to FileSystem close errors due to column statistics

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Labels: pull-request-available
• Assignee: Ayush Saxena
• Reporter: Ayush Saxena
• Created: 17/Jun/23 05:08
• Updated: 15/Aug/23 06:33
• Resolved: 20/Jun/23 04:19

Description

During fetching column statistics we are closing the filesystem in the HS2, FS is shared and cached, so closing it ain't a good idea.

HS2 Logs show

2023-06-16T16:57:09,515 DEBUG [45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22 HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() by method: org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511)); Key: (boroknagyz (auth:SIMPLE))@hdfs://localhost:20500; URI: hdfs://localhost:20500; Object Identity Hash: 14ec8672
2023-06-16T16:57:09,515 TRACE [45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22 HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() full stack trace:
java.lang.Throwable: null
        at org.apache.hadoop.fs.FileSystem.debugLogFileSystemClose(FileSystem.java:627) ~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
        at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2599) ~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
        at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511) ~[hadoop-hdfs-client-3.1.1.7.2.16.0-287.jar:?]
        at org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.canProvideColStatistics(HiveIcebergStorageHandler.java:444) ~[hive-iceberg-handler-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
        at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:1074) ~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:298) ~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]

Issue Links

• links to: GitHub Pull Request #4429