[KUDU-2462] Unable to load consensus metadata for tablet

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.0
• Component/s: server
• Assignee: Unassigned
• Reporter: chengjunhao
• Created: 04/Jun/18 09:25
• Updated: 06/Jun/18 00:43

Description

i had met this issue many times,i dont know what the exact reason for this issue. but every time i met this issue is this situation: when some master and tserver service started failure with the ntp unsync problem at the first start, and i restart the master and tserver when the ntp is sync,but i will met this issue! the flowing is the log for the command "kudu cluster ksck cluster1:7051,cluster2:7051,cluster3:7051":

Tablet 147962d1afa0419bbda19e849ee210ee of table 'my_first_table' is unavailable: 2 replica(s) not RUNNING
  052adf65aa5e465c86318732b3a9fcc2 (node104:7050): bad state
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: Incomplete: Unable to load consensus metadata for tablet 147962d1afa0419bbda19e849ee210ee: Could not read header for proto container file /var/lib/kudu/tserver/consensus-meta/147962d1afa0419bbda19e849ee210ee: File size not large enough to be valid: Proto container file /var/lib/kudu/tserver/consensus-meta/147962d1afa0419bbda19e849ee210ee: Tried to read 16 bytes at offset 0 but file size is only 0 bytes

  2123398e90bc4373a7429b4caa014dc7 (node106:7050): bad state
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: Incomplete: Unable to load consensus metadata for tablet 147962d1afa0419bbda19e849ee210ee: Could not read header for proto container file /var/lib/kudu/tserver/consensus-meta/147962d1afa0419bbda19e849ee210ee: File size not large enough to be valid: Proto container file /var/lib/kudu/tserver/consensus-meta/147962d1afa0419bbda19e849ee210ee: Tried to read 16 bytes at offset 0 but file size is only 0 bytes
  d444b36807624acd96264eac11dd99fc (node105:7050): RUNNING [LEADER]

Table my_first_table has 1 unavailable tablet(s)

Table Summary
      Name      |   Status    | Total Tablets | Healthy | Under-replicated | Unavailable
----------------+-------------+---------------+---------+------------------+-------------
 my_first_table | UNAVAILABLE | 16            | 15      | 0                | 1
==================
Errors:
==================
table consistency check error: Corruption: 1 out of 1 table(s) are bad

##ou! the version i used is "kudu-master-1.5.0+cdh5.13.0+0-1.cdh5.13.0.p0.34.el7.x86_64" and "kudu-tserver-1.5.0+cdh5.13.0+0-1.cdh5.13.0.p0.34.el7.x86_64"

Issue Links

• is related to: KUDU-2195 Enforce durability happened before relationships on multiple disks