[IMPALA-5399] Kudu tables created with wrong master address(es) are bricked

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.7.0
• Component/s: Catalog
• Labels: kudu
• Target Version: Product Backlog
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Matthew Jacobs
• Created: 31/May/17 18:16
• Updated: 19/Dec/17 00:35

Description

A kudu table with invalid master addresses cannot be loaded, and there isn't any way to fix the tblproperties through Impala. It's essentially 'bricked', and the only way to fix it is through the HMS db.

E.g.

[vd1302.halxg.cloudera.com:21000] > select count(*) from lineitem_kudu_ts;
Query: select count(*) from lineitem_kudu_ts
Query submitted at: 2017-05-31 09:10:16 (Coordinator: https://vd1302.halxg.cloudera.com:25000)
ERROR: AnalysisException: Failed to load metadata for table: 'lineitem_kudu_ts'
CAUSED BY: TableLoadingException: Error opening Kudu table 'impala::scan_primitives_tpch_3tb.lineitem_kudu_ts', Kudu error: The table does not exist: table_name: "impala::scan_primitives_tpch_3tb.lineitem_kudu_ts"

[vd1302.halxg.cloudera.com:21000] > describe lineitem_kudu_ts;
Query: describe lineitem_kudu_ts
ERROR: AnalysisException: Error opening Kudu table 'impala::scan_primitives_tpch_3tb.lineitem_kudu_ts', Kudu error: The table does not exist: table_name: "impala::scan_primitives_tpch_3tb.lineitem_kudu_ts"
CAUSED BY: TableLoadingException: Error opening Kudu table 'impala::scan_primitives_tpch_3tb.lineitem_kudu_ts', Kudu error: The table does not exist: table_name: "impala::scan_primitives_tpch_3tb.lineitem_kudu_ts"

Fetched 0 row(s) in 0.04s

Attempting to fix the table to set the correct master addr through Impala also doesn't work:

[vd1302.halxg.cloudera.com:21000] > ALTER TABLE lineitem_kudu_ts SET TBLPROPERTIES ('kudu.master_addresses'='vd1301.halxg.cloudera.com:7051');
Query: alter TABLE lineitem_kudu_ts SET TBLPROPERTIES ('kudu.master_addresses'='vd1301.halxg.cloudera.com:7051')
ERROR: AnalysisException: Failed to load metadata for table: 'lineitem_kudu_ts'
CAUSED BY: TableLoadingException: Error opening Kudu table 'impala::scan_primitives_tpch_3tb.lineitem_kudu_ts', Kudu error: The table does not exist: table_name: "impala::scan_primitives_tpch_3tb.lineitem_kudu_ts"

This can only be fixed directly in the HMS db:

[root@vd1301 /]# mysql hive1 --user=hive1 --password=hive1

mysql> select * from TBLS where TBL_NAME = 'lineitem_kudu_ts';
+--------+-------------+-------+------------------+----------------------------+-----------+--------+------------------+---------------+--------------------+--------------------+----------------+
| TBL_ID | CREATE_TIME | DB_ID | LAST_ACCESS_TIME | OWNER                      | RETENTION | SD_ID  | TBL_NAME         | TBL_TYPE      | VIEW_EXPANDED_TEXT | VIEW_ORIGINAL_TEXT | LINK_TARGET_ID |
+--------+-------------+-------+------------------+----------------------------+-----------+--------+------------------+---------------+--------------------+--------------------+----------------+
|   7526 |  1495302407 |   363 |                0 | [e-mail] |         0 | 815486 | lineitem_kudu_ts | MANAGED_TABLE | NULL               | NULL               |           NULL |
+--------+-------------+-------+------------------+----------------------------+-----------+--------+------------------+---------------+--------------------+--------------------+----------------+
1 row in set (0.00 sec)

mysql> update TABLE_PARAMS set PARAM_VALUE = 'vd1301.halxg.cloudera.com:7051' where TBL_ID = 7526 and PARAM_KEY = 'kudu.master_addresses';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

There should be a way to fix tables like this in Impala.

Issue Links

• is related to: IMPALA-6343 Allow dropping of Kudu tables with invalid master addresses