Details
Critical Description
At present, according to the patch provided by hive-16993, hive 3.1.1 is recompiled. Now it can successfully synchronize with hive metadata and create tables. However, when deleting tables, the log shows that the tables have been successfully deleted, but the tables in kudu still exist and cannot be deleted。
USE trino:test2> CREATE TABLE users ( -> userid int WITH (primary_key = true), -> firstname varchar, -> lastname varchar -> ) WITH ( -> partition_by_hash_columns = ARRAY['userid'], -> partition_by_hash_buckets = 2 -> ); CREATE TABLE trino:test2> show tables; Table ------- users (1 row) trino:test2> drop table users; DROP TABLE trino:test2> show tables; Table ------- users (1 row)Query 20220523_152932_00147_ytvpx, FINISHED, 3 nodes Splits: 20 total, 20 done (100.00%) 0.23 [1 rows, 20B] [4 rows/s, 86B/s] trino:test2> drop table users; Query 20220523_152946_00148_ytvpx failed: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found)
The metadata of hive has been successfully deleted when I delete it for the first time. When I delete it for the second time, it is reported that the metadata does not exist because it has been deleted, but the table still exists in kudu. Use the command:
kudu table delete gpnode1 test2.users
error message：
# kudu table delete gpnode1 test2.users # Not found: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found)
log：
W0523 23:21:52.950664 11595 master_path_handlers.cc:837] Unable to fetch master addresses: OK
W0523 23:22:31.867633 11595 master_path_handlers.cc:837] Unable to fetch master addresses: OK
I0523 23:28:41.818248 5917 catalog_manager.cc:1681] Servicing CreateTable request from {username='presto'} at 192.168.2.182:36402:
name: "test2.users"
schema {
columns {
name: "userid"
type: INT32
is_key: true
is_nullable: false
cfile_block_size: 0
}
columns {
name: "firstname"
type: STRING
is_key: false
is_nullable: false
cfile_block_size: 0
}
columns {
name: "lastname"
type: STRING
is_key: false
is_nullable: false
cfile_block_size: 0
}
}
partition_schema {
hash_bucket_schemas {
columns {
name: "userid"
}
num_buckets: 2
seed: 0
}
}
I0523 23:28:41.877279 5917 catalog_manager.cc:1963] created HMS catalog entry for table [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
I0523 23:28:42.091735 5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.302975
I0523 23:28:42.109527 5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.003s sys 0.003s Metrics: {"bytes_written":5660,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":667,"drs_written":1,"fdatasync":13,"fdatasync_us":11633,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":133,"lbm_writes_lt_1ms":27,"rows_written":3,"thread_start_us":59,"threads_started":1}
I0523 23:28:42.109880 5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.212344
I0523 23:28:42.133885 5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.008s sys 0.004s Metrics: {"bytes_written":213198,"cfile_cache_hit":13,"cfile_cache_hit_bytes":206138,"cfile_cache_miss":35,"cfile_cache_miss_bytes":225520,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":502,"dirs.run_cpu_time_us":810,"dirs.run_wall_time_us":950,"drs_written":1,"fdatasync":15,"fdatasync_us":12534,"lbm_read_time_us":662,"lbm_reads_lt_1ms":75,"lbm_write_time_us":530,"lbm_writes_lt_1ms":32,"num_input_rowsets":2,"rows_written":209,"spinlock_wait_cycles":2944,"thread_start_us":397,"threads_started":4}
I0523 23:28:44.408016 5917 catalog_manager.cc:4733] T 273e6e3223f14ea690621e40c53c8c0e P 18c9a3f5aa314a2b80dfa970003e4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1 leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host: "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host: "192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host: "192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:28:59.145552 5917 catalog_manager.cc:4733] T e0d2911469d147609abcdcebffacc7f1 P 18c9a3f5aa314a2b80dfa970003e4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1 leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host: "192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host: "192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host: "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:29:28.640434 5917 catalog_manager.cc:2253] Servicing DeleteTable request from {username='presto'} at 192.168.2.182:36402:
table { table_name: "test2.users" }
I0523 23:29:28.706779 5917 catalog_manager.cc:2293] deleted HMS catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
I0523 23:29:33.955879 5917 catalog_manager.cc:4733] T e0d2911469d147609abcdcebffacc7f1 P 9ad2b92eba814e78895e645f74618bd2 reported cstate change: term changed from 1 to 2, leader changed from 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211) to 9ad2b92eba814e78895e645f74618bd2 (192.168.2.182). New cstate: current_term: 2 leader_uuid: "9ad2b92eba814e78895e645f74618bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host: "192.168.2.211" port: 7050 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host: "192.168.2.182" port: 7050 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host: "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:29:46.440908 5917 catalog_manager.cc:2253] Servicing DeleteTable request from {username='presto'} at 192.168.2.182:36402:
table { table_name: "test2.users" }
W0523 23:29:46.446835 5917 catalog_manager.cc:2296] failed to delete HMS catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not found: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found)
I0523 23:30:44.479030 5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033352
I0523 23:30:44.493083 5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.001s sys 0.000s Metrics: {"bytes_written":476,"delete_count":0,"fdatasync":3,"fdatasync_us":12119,"lbm_write_time_us":23,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":21,"threads_started":1,"update_count":3,"wal-append.queue_time_us":63}
I0523 23:30:44.493170 5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I0523 23:30:44.498075 5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.001s sys 0.000s Metrics: {"cfile_cache_hit":1,"cfile_cache_hit_bytes":29,"cfile_cache_miss":3,"cfile_cache_miss_bytes":1283,"cfile_init":1,"delete_count":146,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"fdatasync":3,"fdatasync_us":3378,"lbm_read_time_us":5,"lbm_reads_lt_1ms":7,"lbm_write_time_us":11,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":16,"threads_started":1,"update_count":3}
I0524 00:00:48.336750 5917 catalog_manager.cc:2253] Servicing DeleteTable request from {username='root'} at 192.168.2.182:42712:
table { table_name: "test2.users" } modify_external_catalogs: true
W0524 00:00:48.349134 5917 catalog_manager.cc:2296] failed to delete HMS catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not found: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found)
kudu cluster ksck gpmaster,gpnode1,gpnode2
[kudu@gpnode1 kudu]$ kudu cluster ksck gpmaster,gpnode1,gpnode2 Master Summary UUID | Address | Status ----------------------------------+----------+--------- 2bc3b2318ca640a78a99fcbe4d058a9f | gpmaster | HEALTHY a01491afc9f24dc3adef5d9cbc4ff405 | gpnode1 | HEALTHY ccbd77b62e784359b71c07d16a909e59 | gpnode2 | HEALTHYUnusual flags for Master: Flag | Value | Tags | Master ------------------+-------+--------+------------------------- use_hybrid_clock | false | hidden | all 3 server(s) checkedFlags of checked categories for Master: Flag | Value | Master ---------------------+-------------------------------------------------------------+------------------------- builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked time_source | system | all 3 server(s) checkedTablet Server Summary UUID | Address | Status | Location | Tablet Leaders | Active Scanners ----------------------------------+--------------------+---------+----------+----------------+----------------- 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | HEALTHY | <none> | 0 | 0 524c9034d0b540cab9161d44a5dab7ef | 192.168.2.184:7050 | HEALTHY | <none> | 14 | 0 7f33079cf18941eda06a0fcc06c3afd3 | 192.168.2.183:7050 | HEALTHY | <none> | 11 | 0 9ad2b92eba814e78895e645f74618bd2 | 192.168.2.182:7050 | HEALTHY | <none> | 13 | 0 Tablet Server Location Summary Location | Count ----------+--------- <none> | 4Unusual flags for Tablet Server: Flag | Value | Tags | Tablet Server ------------------+-------+--------+------------------------- use_hybrid_clock | false | hidden | all 4 server(s) checkedFlags of checked categories for Tablet Server: Flag | Value | Tablet Server ---------------------+-------------------------------------------------------------+------------------------- builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked time_source | system | all 4 server(s) checkedVersion Summary Version | Servers ---------+------------------------- 1.15.0 | all 7 server(s) checkedTablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- $schemas | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users10 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users11 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users12 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users13 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users4 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users5 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users6 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users7 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users8 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test1.users9 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test2.users | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test2.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test2.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0 test2.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0 | 0Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 10 First Quartile | 34 Median | 35 Third Quartile | 35 Maximum | 35 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | 10 Total Count Summary | Total Count ----------------+------------- Masters | 3 Tablet Servers | 4 Tables | 19 Tablets | 38 Replicas | 114 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK
