[IMPALA-11112] Impala can't resolve json tables created by Hive

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.1.0
• Fix Version/s: Impala 4.1.0
• Component/s: Catalog
• Epic Color: ghx-label-8
• Assignee: Pranav Yogi Lodha
• Reporter: Quanlong Huang
• Created: 09/Feb/22 07:11
• Updated: 11/Mar/22 06:53
• Resolved: 11/Mar/22 06:53

Description

JSON tables created by Hive can't be resolved in Impala. For instance, creating a JSON table in Hive using beeline:

$ beeline -u "jdbc:hive2://localhost:11050"
0: jdbc:hive2://localhost:11050> create table my_json_tbl (id int, name string) stored as jsonfile;

Then describe it in Impala using impala-shell:

$ bin/impala-shell.sh
[localhost:21050] default> invalidate metadata my_json_tbl;
[localhost:21050] default> desc my_json_tbl;
Query: describe my_json_tbl
ERROR: AnalysisException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
CAUSED BY: TableLoadingException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported. 

There are more details in logs

I0209 14:59:55.419886 10083 jni-util.cc:286] 914bfb645cab120b:443e5ec500000000] org.apache.impala.common.AnalysisException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
        at org.apache.impala.analysis.DescribeTableStmt.analyze(DescribeTableStmt.java:112)
        at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:497)
        at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:461)
        at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1677)
        at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1643)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1613)
        at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
Caused by: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
        at org.apache.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:123)
        at org.apache.impala.catalog.Table.fromThrift(Table.java:513)
        at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:469)
        at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:324)
        at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:253)
        at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
        at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:410)
        at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:185)
        at ========.<Remote stack trace on catalogd>: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
        at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1265)
        at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1147)
        at org.apache.impala.catalog.TableLoader.load(TableLoader.java:144)
        at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
        at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:242)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.HdfsStorageDescriptor$InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
        at org.apache.impala.catalog.HdfsStorageDescriptor.fromStorageDescriptor(HdfsStorageDescriptor.java:216)
        at org.apache.impala.catalog.HdfsTable.setPrototypePartition(HdfsTable.java:1139)
        at org.apache.impala.catalog.HdfsTable.initializePartitionMetadata(HdfsTable.java:656)
        at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:678)
        at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1253)
        ... 8 more 

We also need to fix the issue that Hive can't resolve JSON tables created by Impala.

Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.ClassNotFoundException Class org.apache.hadoop.hive.serde2.lazy.JsonSerDe not found

Probably change a SerDe that Hive recognizes.

Issue Links

• causes: IMPALA-11156 TestHmsIntegration.test_desc_json_table failed in exhaustive build