Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5491

Improve error message when loading metadata for table where HDFS files have missing blocks

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.11.0
    • Component/s: Catalog
    • Labels:
    • Epic Color:
      ghx-label-8

      Description

      Loading metadata for tables with corrupted files fails with the error message below which isn't descriptive of the actual problem.

      Query: describe catalog_returns
      ERROR: AnalysisException: Failed to load metadata for table: catalog_returns
      CAUSED BY: IllegalStateException: null
      CAUSED BY: TableLoadingException: Failed to load metadata for table: catalog_returns
      CAUSED BY: IllegalStateException: null
      

      Exception

      I0612 18:44:49.502610 19242 Frontend.java:892] Compiling query: describe catalog_returns
      E0612 18:44:49.503001 19242 Analyzer.java:2300] Failed to load metadata for table: catalog_returns
      Failed to load metadata for table: catalog_returns
      CAUSED BY: IllegalStateException: null
      I0612 18:44:49.503366 19242 Frontend.java:929] Compiled query.
      I0612 18:44:49.503698 19242 jni-util.cc:176] org.apache.impala.common.AnalysisException: Failed to load metadata for table: catalog_returns
      CAUSED BY: IllegalStateException: null
      	at org.apache.impala.analysis.DescribeTableStmt.analyze(DescribeTableStmt.java:122)
      	at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:377)
      	at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:358)
      	at org.apache.impala.service.Frontend.analyzeStmt(Frontend.java:901)
      	at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1050)
      	at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
      Caused by: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: catalog_returns
      CAUSED BY: IllegalStateException: null
      	at org.apache.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:113)
      	at org.apache.impala.catalog.Table.fromThrift(Table.java:252)
      	at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:369)
      	at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:253)
      	at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:140)
      	at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:225)
      	at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:191)
      	at ========.<Remote stack trace on catalogd>: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: catalog_returns
      	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1091)
      	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1019)
      	at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80)
      	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:237)
      	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:234)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException
      	at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
      	at org.apache.impala.catalog.HdfsPartition$FileBlock.createFbFileBlock(HdfsPartition.java:314)
      	at org.apache.impala.catalog.HdfsPartition$FileDescriptor.create(HdfsPartition.java:119)
      	at org.apache.impala.catalog.HdfsTable.loadBlockMetadata(HdfsTable.java:320)
      	at org.apache.impala.catalog.HdfsTable.loadMetadataAndDiskIds(HdfsTable.java:771)
      	at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:689)
      	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1082)
      	... 8 more
      ()
      I0612 18:44:49.509366 19242 status.cc:122] AnalysisException: Failed to load metadata for table: catalog_returns
      CAUSED BY: IllegalStateException: null
      CAUSED BY: TableLoadingException: Failed to load metadata for table: catalog_returns
      CAUSED BY: IllegalStateException: null
          @           0x83c329  impala::Status::Status()
          @           0xb96a50  impala::JniUtil::GetJniExceptionMsg()
          @           0xaa724b  impala::Frontend::GetExecRequest()
          @           0xac2005  impala::ImpalaServer::ExecuteInternal()
          @           0xac7fe8  impala::ImpalaServer::Execute()
          @           0xb04756  impala::ImpalaServer::query()
          @           0xd6b475  beeswax::BeeswaxServiceProcessor::process_query()
          @           0xd6e774  beeswax::BeeswaxServiceProcessor::dispatchCall()
          @           0x80c57c  apache::thrift::TDispatchProcessor::process()
          @          0x1b4a62b  apache::thrift::server::TThreadPoolServer::Task::run()
          @          0x1b321e9  apache::thrift::concurrency::ThreadManager::Worker::run()
          @           0x9fba09  impala::ThriftThread::RunRunnable()
          @           0x9fc462  boost::detail::function::void_function_obj_invoker0<>::invoke()
          @           0xbed859  impala::Thread::SuperviseThread()
          @           0xbee214  boost::detail::thread_data<>::run()
          @           0xe5539a  (unknown)
          @       0x3b1c407aa1  (unknown)
          @       0x3b1c0e8bcd  (unknown)
      

      From the Catalog logs

      I0612 18:44:27.362217 57831 TableLoader.java:58] Loading metadata for: tpcds_1000_parquet.catalog_returns
      I0612 18:44:27.397717 57831 HdfsTable.java:1077] Fetching partition metadata from the Metastore: tpcds_1000_parquet.catalog_returns
      I0612 18:44:28.176765 57831 HdfsTable.java:1081] Fetched partition metadata from the Metastore: tpcds_1000_parquet.catalog_returns
      I0612 18:44:30.614495 57831 HdfsTable.java:768] Loading file and block metadata for 2104 partitions from 1 paths: tpcds_1000_parquet.catalog_returns
      I0612 18:44:31.329075 57831 TableLoader.java:97] Loaded metadata for: tpcds_1000_parquet.catalog_returns
      I0612 18:44:32.327278 32915 catalog-server.cc:320] Publishing update: TABLE:tpcds_1000_parquet.catalog_returns@169
      I0612 18:44:32.327766 32915 catalog-server.cc:320] Publishing update: CATALOG:7d7086ae05a24b4e:90f40613772eeee7@169
      I0612 18:46:52.338260 32920 catalog-server.cc:236] Catalog Version: 169 Last Catalog Version: 169
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gaborkaszab Gabor Kaszab
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: