Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.1, Impala 2.2
-
None
Description
We see inconsistent error message related to compute stats in runtime profile vs explain vs analyze.
Impala throws compute stats non existing error even when there is stats present for a table.
Eg:
[hdpdev@eb2-hue1 logs]$ impala-shell -i x.xdn1.x.x -q "compute stats dev_ods.referrer;" Starting Impala Shell without Kerberos authentication Connected to x.xdn1.x.x:21000 Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78) Query: compute stats dev_ods.referrer +-----------------------------------------+ | summary | +-----------------------------------------+ | Updated 1 partition(s) and 3 column(s). | +-----------------------------------------+ Returned 1 row(s) in 63.34s [hdpdev@eb2-hue1 logs]$ impala-shell -i x.xdn1.x.x -q "show table stats dev_ods.referrer;" Starting Impala Shell without Kerberos authentication Connected to x.xdn1.x.x:21000 Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78) Query: show table stats dev_ods.referrer +-----------+--------+---------+--------------+---------+ | #Rows | #Files | Size | Bytes Cached | Format | +-----------+--------+---------+--------------+---------+ | 622700208 | 175 | 53.21GB | NOT CACHED | PARQUET | +-----------+--------+---------+--------------+---------+ Returned 1 row(s) in 0.01s [hdpdev@eb2-hue1 logs]$ impala-shell -i x.xdn1.x.x -q "describe formatted dev_ods.referrer;" Starting Impala Shell without Kerberos authentication Connected to x.xdn1.x.x:21000 Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78) Query: describe formatted dev_ods.referrer +------------------------------+------------------------------------------------------------------------------+----------------------+ | name | type | comment | +------------------------------+------------------------------------------------------------------------------+----------------------+ | # col_name | data_type | comment | | | NULL | NULL | | referrer_id | bigint | None | | query | string | None | | url | string | None | | | NULL | NULL | | # Detailed Table Information | NULL | NULL | | Database: | dev_ods | NULL | | Owner: | hdpdev | NULL | | CreateTime: | Mon Jan 12 03:11:36 PST 2015 | NULL | | LastAccessTime: | UNKNOWN | NULL | | Protect Mode: | None | NULL | | Retention: | 0 | NULL | | Location: | hdfs://x.xnn1.x.x:8020/user/hive/warehouse/dev_ods.db/referrer | NULL | | Table Type: | MANAGED_TABLE | NULL | | Table Parameters: | NULL | NULL | | | numRows | 622700208 | | | transient_lastDdlTime | 1421109619 | | | NULL | NULL | | # Storage Information | NULL | NULL | | SerDe Library: | parquet.hive.serde.ParquetHiveSerDe | NULL | | InputFormat: | parquet.hive.DeprecatedParquetInputFormat | NULL | | OutputFormat: | parquet.hive.DeprecatedParquetOutputFormat | NULL | | Compressed: | No | NULL | | Num Buckets: | 0 | NULL | | Bucket Columns: | [] | NULL | | Sort Columns: | [] | NULL | +------------------------------+------------------------------------------------------------------------------+----------------------+ Returned 27 row(s) in 0.02s Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78) Query: create TABLE dev_temp.load_prd_src_report_step0_1_20520150112151629584 STORED AS parquet AS select c.event_date,c.event_85_id,c.dw_websession_id,c.is_signup,c.request_url_query,c.request_url_stem,a.query,a.url from dev_temp.load_prd_src_report_step0_20520150112151629583 c join dev_ods.referrer a on c.referrer_id=a.referrer_id WARNINGS: Memory limit exceeded Backend 3:Memory Limit Exceeded Query Limit: Consumption=31.64 GB Fragment 144127d772cc0752:40eb3c94f22ea5b2: Consumption=28.49 GB UDFs: Consumption=0 HASH_JOIN_NODE (id=2): Consumption=28.48 GB EXCHANGE_NODE (id=3): Consumption=0 DataStreamRecvr: Consumption=10.48 MB EXCHANGE_NODE (id=4): Consumption=0 DataStreamRecvr: Consumption=2.42 MB HdfsTableSink: Consumption=0 Fragment 144127d772cc0752:40eb3c94f22ea5b6: Consumption=1.49 GB UDFs: Consumption=0 HDFS_SCAN_NODE (id=1): Consumption=1.49 GB DataStreamSender: Consumption=218.66 KB Fragment 144127d772cc0752:40eb3c94f22ea5ba: Consumption=1.66 GB UDFs: Consumption=0 HDFS_SCAN_NODE (id=0): Consumption=1.66 GB DataStreamSender: Consumption=183.09 KB WARNING: The following tables are missing relevant table and/or column statistics. dev_ods.referrer