Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3370

drop table failed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Not A Bug
    • 1.15.0
    • NA
    • master, tserver
    • None
    • hdp :3.2.3
      hive :3.1.1
      kudu :1.15.0
      trino:369-SNAPSHOT

    Description

      At present, according to the patch provided by hive-16993, hive 3.1.1 is recompiled. Now it can successfully synchronize with hive metadata and create tables. However, when deleting tables, the log shows that the tables have been successfully deleted, but the tables in kudu still exist and cannot be deleted。

      USE
      trino:test2> CREATE TABLE users (
                ->   userid int WITH (primary_key = true),
                ->   firstname varchar,
                ->   lastname varchar
                -> ) WITH (
                ->   partition_by_hash_columns = ARRAY['userid'],
                ->   partition_by_hash_buckets = 2
                -> );
      CREATE TABLE
      
      trino:test2> show tables;
       Table 
      -------
       users 
      (1 row)
      
      trino:test2> drop table users;
      DROP TABLE
      trino:test2> show tables;
       Table 
      -------
       users 
      (1 row)Query 20220523_152932_00147_ytvpx, FINISHED, 3 nodes
      Splits: 20 total, 20 done (100.00%)
      0.23 [1 rows, 20B] [4 rows/s, 86B/s]
      
      trino:test2> drop table users;
      Query 20220523_152946_00148_ytvpx failed: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found)  

      The metadata of hive has been successfully deleted when I delete it for the first time. When I delete it for the second time, it is reported that the metadata does not exist because it has been deleted, but the table still exists in kudu. Use the command:

      kudu table delete gpnode1 test2.users

      error message:

      # kudu table delete gpnode1 test2.users
      # Not found: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found) 

      log:

      W0523 23:21:52.950664 11595 master_path_handlers.cc:837] Unable to fetch master addresses: OK
      W0523 23:22:31.867633 11595 master_path_handlers.cc:837] Unable to fetch master addresses: OK
      I0523 23:28:41.818248  5917 catalog_manager.cc:1681] Servicing CreateTable request from {username='presto'} at 192.168.2.182:36402:
      name: "test2.users"
      schema {
        columns {
          name: "userid"
          type: INT32
          is_key: true
          is_nullable: false
          cfile_block_size: 0
        }
        columns {
          name: "firstname"
          type: STRING
          is_key: false
          is_nullable: false
          cfile_block_size: 0
        }
        columns {
          name: "lastname"
          type: STRING
          is_key: false
          is_nullable: false
          cfile_block_size: 0
        }
      }
      partition_schema {
        hash_bucket_schemas {
          columns {
            name: "userid"
          }
          num_buckets: 2
          seed: 0
        }
      }
      I0523 23:28:41.877279  5917 catalog_manager.cc:1963] created HMS catalog entry for table  [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
      I0523 23:28:42.091735  5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.302975
      I0523 23:28:42.109527  5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.018s	user 0.003s	sys 0.003s Metrics: {"bytes_written":5660,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":667,"drs_written":1,"fdatasync":13,"fdatasync_us":11633,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":133,"lbm_writes_lt_1ms":27,"rows_written":3,"thread_start_us":59,"threads_started":1}
      I0523 23:28:42.109880  5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.212344
      I0523 23:28:42.133885  5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.024s	user 0.008s	sys 0.004s Metrics: {"bytes_written":213198,"cfile_cache_hit":13,"cfile_cache_hit_bytes":206138,"cfile_cache_miss":35,"cfile_cache_miss_bytes":225520,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":502,"dirs.run_cpu_time_us":810,"dirs.run_wall_time_us":950,"drs_written":1,"fdatasync":15,"fdatasync_us":12534,"lbm_read_time_us":662,"lbm_reads_lt_1ms":75,"lbm_write_time_us":530,"lbm_writes_lt_1ms":32,"num_input_rowsets":2,"rows_written":209,"spinlock_wait_cycles":2944,"thread_start_us":397,"threads_started":4}
      I0523 23:28:44.408016  5917 catalog_manager.cc:4733] T 273e6e3223f14ea690621e40c53c8c0e P 18c9a3f5aa314a2b80dfa970003e4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1 leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host: "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host: "192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host: "192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } }
      I0523 23:28:59.145552  5917 catalog_manager.cc:4733] T e0d2911469d147609abcdcebffacc7f1 P 18c9a3f5aa314a2b80dfa970003e4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1 leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host: "192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host: "192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host: "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
      I0523 23:29:28.640434  5917 catalog_manager.cc:2253] Servicing DeleteTable request from {username='presto'} at 192.168.2.182:36402:
      table { table_name: "test2.users" }
      I0523 23:29:28.706779  5917 catalog_manager.cc:2293] deleted HMS catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
      I0523 23:29:33.955879  5917 catalog_manager.cc:4733] T e0d2911469d147609abcdcebffacc7f1 P 9ad2b92eba814e78895e645f74618bd2 reported cstate change: term changed from 1 to 2, leader changed from 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211) to 9ad2b92eba814e78895e645f74618bd2 (192.168.2.182). New cstate: current_term: 2 leader_uuid: "9ad2b92eba814e78895e645f74618bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host: "192.168.2.211" port: 7050 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host: "192.168.2.182" port: 7050 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host: "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
      I0523 23:29:46.440908  5917 catalog_manager.cc:2253] Servicing DeleteTable request from {username='presto'} at 192.168.2.182:36402:
      table { table_name: "test2.users" }
      W0523 23:29:46.446835  5917 catalog_manager.cc:2296] failed to delete HMS catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not found: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found)
      I0523 23:30:44.479030  5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033352
      I0523 23:30:44.493083  5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s	user 0.001s	sys 0.000s Metrics: {"bytes_written":476,"delete_count":0,"fdatasync":3,"fdatasync_us":12119,"lbm_write_time_us":23,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":21,"threads_started":1,"update_count":3,"wal-append.queue_time_us":63}
      I0523 23:30:44.493170  5907 maintenance_manager.cc:382] P a01491afc9f24dc3adef5d9cbc4ff405: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
      I0523 23:30:44.498075  5906 maintenance_manager.cc:603] P a01491afc9f24dc3adef5d9cbc4ff405: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.005s	user 0.001s	sys 0.000s Metrics: {"cfile_cache_hit":1,"cfile_cache_hit_bytes":29,"cfile_cache_miss":3,"cfile_cache_miss_bytes":1283,"cfile_init":1,"delete_count":146,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"fdatasync":3,"fdatasync_us":3378,"lbm_read_time_us":5,"lbm_reads_lt_1ms":7,"lbm_write_time_us":11,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":16,"threads_started":1,"update_count":3}
      I0524 00:00:48.336750  5917 catalog_manager.cc:2253] Servicing DeleteTable request from {username='root'} at 192.168.2.182:42712:
      table { table_name: "test2.users" } modify_external_catalogs: true
      W0524 00:00:48.349134  5917 catalog_manager.cc:2296] failed to delete HMS catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not found: failed to drop Hive Metastore table: TException - service has thrown: NoSuchObjectException(message=hive.test2.users table not found)
      

       

       

       

       kudu cluster ksck gpmaster,gpnode1,gpnode2

      [kudu@gpnode1 kudu]$ kudu cluster ksck gpmaster,gpnode1,gpnode2
      Master Summary
                     UUID               | Address  | Status
      ----------------------------------+----------+---------
       2bc3b2318ca640a78a99fcbe4d058a9f | gpmaster | HEALTHY
       a01491afc9f24dc3adef5d9cbc4ff405 | gpnode1  | HEALTHY
       ccbd77b62e784359b71c07d16a909e59 | gpnode2  | HEALTHYUnusual flags for Master:
             Flag       | Value |  Tags  |         Master
      ------------------+-------+--------+-------------------------
       use_hybrid_clock | false | hidden | all 3 server(s) checkedFlags of checked categories for Master:
              Flag         |                            Value                            |         Master
      ---------------------+-------------------------------------------------------------+-------------------------
       builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
       time_source         | system                                                      | 
      all 3 server(s) checkedTablet Server Summary
                     UUID               |      Address       | Status  | Location | Tablet Leaders |
       Active Scanners
      ----------------------------------+--------------------+---------+----------+----------------+-----------------
       18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | HEALTHY | <none>   |       0        |       0
       524c9034d0b540cab9161d44a5dab7ef | 192.168.2.184:7050 | HEALTHY | <none>   |      14        |       0
       7f33079cf18941eda06a0fcc06c3afd3 | 192.168.2.183:7050 | HEALTHY | <none>   |      11        |       0
       9ad2b92eba814e78895e645f74618bd2 | 192.168.2.182:7050 | HEALTHY | <none>   |      13        |       0
      Tablet Server Location Summary
       Location |  Count
      ----------+---------
       <none>   |       4Unusual flags for Tablet Server:
             Flag       | Value |  Tags  |      Tablet Server
      ------------------+-------+--------+-------------------------
       use_hybrid_clock | false | hidden | all 4 server(s) checkedFlags of checked categories for Tablet Server:
              Flag         |                            Value                            |      Tablet Server
      ---------------------+-------------------------------------------------------------+-------------------------
       builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
       time_source         | system                                                      | 
      all 4 server(s) checkedVersion Summary
       Version |         Servers
      ---------+-------------------------
       1.15.0  | all 7 server(s) checkedTablet Summary
      Summary by table
           Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
      ---------------+----+---------+---------------+---------+------------+------------------+-------------
       $schemas      | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users   | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users1  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users10 | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users11 | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users12 | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users13 | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users2  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users3  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users4  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users5  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users6  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users7  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users8  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test1.users9  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test2.users   | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test2.users1  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test2.users2  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0
       test2.users3  | 3  | HEALTHY | 2             | 2       | 0          | 0                | 0Tablet Replica Count Summary
         Statistic    | Replica Count
      ----------------+---------------
       Minimum        | 10
       First Quartile | 34
       Median         | 35
       Third Quartile | 35
       Maximum        | 35
      
      Tablet Replica Count Outliers
       Type  |               UUID               |        Host        | Replica Count
      -------+----------------------------------+--------------------+---------------
       Small | 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | 10
      Total Count Summary
                      | Total Count
      ----------------+-------------
       Masters        | 3
       Tablet Servers | 4
       Tables         | 19
       Tablets        | 38
       Replicas       | 114
      ==================
      Warnings:
      ==================
      Some masters have unsafe, experimental, or hidden flags set
      Some tablet servers have unsafe, experimental, or hidden flags set
      OK
         

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            zx778899 liu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: