Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10725 Support JSON format tables
  3. IMPALA-11112

Impala can't resolve json tables created by Hive

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 4.1.0
    • Impala 4.1.0
    • Catalog
    • None
    • ghx-label-8

    Description

      JSON tables created by Hive can't be resolved in Impala. For instance, creating a JSON table in Hive using beeline:

      $ beeline -u "jdbc:hive2://localhost:11050"
      0: jdbc:hive2://localhost:11050> create table my_json_tbl (id int, name string) stored as jsonfile;

      Then describe it in Impala using impala-shell:

      $ bin/impala-shell.sh
      [localhost:21050] default> invalidate metadata my_json_tbl;
      [localhost:21050] default> desc my_json_tbl;
      Query: describe my_json_tbl
      ERROR: AnalysisException: Failed to load metadata for table: default.my_json_tbl
      CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
      CAUSED BY: TableLoadingException: Failed to load metadata for table: default.my_json_tbl
      CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported. 

      There are more details in logs

      I0209 14:59:55.419886 10083 jni-util.cc:286] 914bfb645cab120b:443e5ec500000000] org.apache.impala.common.AnalysisException: Failed to load metadata for table: default.my_json_tbl
      CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
              at org.apache.impala.analysis.DescribeTableStmt.analyze(DescribeTableStmt.java:112)
              at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:497)
              at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:461)
              at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1677)
              at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1643)
              at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1613)
              at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
      Caused by: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
      CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
              at org.apache.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:123)
              at org.apache.impala.catalog.Table.fromThrift(Table.java:513)
              at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:469)
              at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:324)
              at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:253)
              at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
              at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:410)
              at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:185)
              at ========.<Remote stack trace on catalogd>: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
              at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1265)
              at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1147)
              at org.apache.impala.catalog.TableLoader.load(TableLoader.java:144)
              at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
              at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:242)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.impala.catalog.HdfsStorageDescriptor$InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
              at org.apache.impala.catalog.HdfsStorageDescriptor.fromStorageDescriptor(HdfsStorageDescriptor.java:216)
              at org.apache.impala.catalog.HdfsTable.setPrototypePartition(HdfsTable.java:1139)
              at org.apache.impala.catalog.HdfsTable.initializePartitionMetadata(HdfsTable.java:656)
              at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:678)
              at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1253)
              ... 8 more 

      We also need to fix the issue that Hive can't resolve JSON tables created by Impala.

      Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.ClassNotFoundException Class org.apache.hadoop.hive.serde2.lazy.JsonSerDe not found
      

      Probably change a SerDe that Hive recognizes.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            pranav.lodha Pranav Yogi Lodha
            stigahuang Quanlong Huang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment