Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1720

Inconsistent error messaging related to stats of table in query and analyze

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.1, Impala 2.2
    • Fix Version/s: Impala 2.3.0
    • Component/s: None
    • Labels:

      Description

      We see inconsistent error message related to compute stats in runtime profile vs explain vs analyze.

      Impala throws compute stats non existing error even when there is stats present for a table.

      Eg:

      [hdpdev@eb2-hue1 logs]$ impala-shell -i x.xdn1.x.x -q  "compute stats dev_ods.referrer;"
      Starting Impala Shell without Kerberos authentication
      Connected to x.xdn1.x.x:21000
      Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78)
      Query: compute stats dev_ods.referrer
      
      +-----------------------------------------+
      | summary                                 |
      +-----------------------------------------+
      | Updated 1 partition(s) and 3 column(s). |
      +-----------------------------------------+
      Returned 1 row(s) in 63.34s
      [hdpdev@eb2-hue1 logs]$ impala-shell -i x.xdn1.x.x -q  "show table stats dev_ods.referrer;"
      Starting Impala Shell without Kerberos authentication
      Connected to x.xdn1.x.x:21000
      Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78)
      Query: show table stats dev_ods.referrer
      +-----------+--------+---------+--------------+---------+
      | #Rows     | #Files | Size    | Bytes Cached | Format  |
      +-----------+--------+---------+--------------+---------+
      | 622700208 | 175    | 53.21GB | NOT CACHED   | PARQUET |
      +-----------+--------+---------+--------------+---------+
      Returned 1 row(s) in 0.01s
      [hdpdev@eb2-hue1 logs]$ impala-shell -i x.xdn1.x.x -q  "describe formatted dev_ods.referrer;"
      Starting Impala Shell without Kerberos authentication
      Connected to x.xdn1.x.x:21000
      Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78)
      Query: describe formatted dev_ods.referrer
      +------------------------------+------------------------------------------------------------------------------+----------------------+
      | name                         | type                                                                         | comment              |
      +------------------------------+------------------------------------------------------------------------------+----------------------+
      | # col_name                   | data_type                                                                    | comment              |
      |                              | NULL                                                                         | NULL                 |
      | referrer_id                  | bigint                                                                       | None                 |
      | query                        | string                                                                       | None                 |
      | url                          | string                                                                       | None                 |
      |                              | NULL                                                                         | NULL                 |
      | # Detailed Table Information | NULL                                                                         | NULL                 |
      | Database:                    | dev_ods                                                                      | NULL                 |
      | Owner:                       | hdpdev                                                                       | NULL                 |
      | CreateTime:                  | Mon Jan 12 03:11:36 PST 2015                                                 | NULL                 |
      | LastAccessTime:              | UNKNOWN                                                                      | NULL                 |
      | Protect Mode:                | None                                                                         | NULL                 |
      | Retention:                   | 0                                                                            | NULL                 |
      | Location:                    | hdfs://x.xnn1.x.x:8020/user/hive/warehouse/dev_ods.db/referrer | NULL                 |
      | Table Type:                  | MANAGED_TABLE                                                                | NULL                 |
      | Table Parameters:            | NULL                                                                         | NULL                 |
      |                              | numRows                                                                      | 622700208            |
      |                              | transient_lastDdlTime                                                        | 1421109619           |
      |                              | NULL                                                                         | NULL                 |
      | # Storage Information        | NULL                                                                         | NULL                 |
      | SerDe Library:               | parquet.hive.serde.ParquetHiveSerDe                                          | NULL                 |
      | InputFormat:                 | parquet.hive.DeprecatedParquetInputFormat                                    | NULL                 |
      | OutputFormat:                | parquet.hive.DeprecatedParquetOutputFormat                                   | NULL                 |
      | Compressed:                  | No                                                                           | NULL                 |
      | Num Buckets:                 | 0                                                                            | NULL                 |
      | Bucket Columns:              | []                                                                           | NULL                 |
      | Sort Columns:                | []                                                                           | NULL                 |
      +------------------------------+------------------------------------------------------------------------------+----------------------+
      Returned 27 row(s) in 0.02s
      
      Server version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78)
      Query: create TABLE dev_temp.load_prd_src_report_step0_1_20520150112151629584 STORED AS parquet  AS   select c.event_date,c.event_85_id,c.dw_websession_id,c.is_signup,c.request_url_query,c.request_url_stem,a.query,a.url from dev_temp.load_prd_src_report_step0_20520150112151629583 c  join dev_ods.referrer a on c.referrer_id=a.referrer_id
      WARNINGS: Memory limit exceeded
      
      
      Backend 3:Memory Limit Exceeded
      Query Limit: Consumption=31.64 GB
        Fragment 144127d772cc0752:40eb3c94f22ea5b2: Consumption=28.49 GB
          UDFs: Consumption=0
          HASH_JOIN_NODE (id=2): Consumption=28.48 GB
          EXCHANGE_NODE (id=3): Consumption=0
          DataStreamRecvr: Consumption=10.48 MB
          EXCHANGE_NODE (id=4): Consumption=0
          DataStreamRecvr: Consumption=2.42 MB
          HdfsTableSink: Consumption=0
        Fragment 144127d772cc0752:40eb3c94f22ea5b6: Consumption=1.49 GB
          UDFs: Consumption=0
          HDFS_SCAN_NODE (id=1): Consumption=1.49 GB
          DataStreamSender: Consumption=218.66 KB
        Fragment 144127d772cc0752:40eb3c94f22ea5ba: Consumption=1.66 GB
          UDFs: Consumption=0
          HDFS_SCAN_NODE (id=0): Consumption=1.66 GB
          DataStreamSender: Consumption=183.09 KB
      WARNING: The following tables are missing relevant table and/or column statistics.
      dev_ods.referrer
      
      

        Attachments

        1. profile-7.txt
          33 kB
          Mala Chikka Kempanna

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mala_ck Mala Chikka Kempanna
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: