[KUDU-197] Tablet bootstrap failing on the client-test due to file not found

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: M3
• Component/s: recovery
• Assignee: Alex Feinberg
• Reporter: David Alves
• Created: 23/Mar/14 12:50
• Updated: 26/Feb/16 13:32
• Resolved: 12/May/14 14:06

Description

I'm getting this sometimes on the client test:

format_stamp: "Formatted at 2014-03-23 17:43:25 on process010"
I0323 12:43:25.250133 11319 ts_tablet_manager.cc:241] Loading master block c46cc5e88ef240ce8fcc3c55f25e2041
I0323 12:43:25.250354 11319 ts_tablet_manager.cc:373] Registered tablet c46cc5e88ef240ce8fcc3c55f25e2041
I0323 12:43:25.250378 11319 ts_tablet_manager.cc:241] Loading master block c1ba347e8fb447a289988ce2e7363dfb
I0323 12:43:25.250390 13606 ts_tablet_manager.cc:274] Bootstrapping tablet: c46cc5e88ef240ce8fcc3c55f25e2041
I0323 12:43:25.250452 13606 tablet_bootstrap.cc:220] Tablet id c46cc5e88ef240ce8fcc3c55f25e2041: Bootstrap starting.
I0323 12:43:25.250608 11319 ts_tablet_manager.cc:373] Registered tablet c1ba347e8fb447a289988ce2e7363dfb
I0323 12:43:25.250668 13605 ts_tablet_manager.cc:274] Bootstrapping tablet: c1ba347e8fb447a289988ce2e7363dfb
I0323 12:43:25.250788 13605 tablet_bootstrap.cc:220] Tablet id c1ba347e8fb447a289988ce2e7363dfb: Bootstrap starting.
I0323 12:43:25.251049 11319 webserver.cc:105] Starting webserver on 0.0.0.0:0
I0323 12:43:25.251060 11319 webserver.cc:114] Document root disabled
I0323 12:43:25.251189 11319 webserver.cc:184] Webserver started
I0323 12:43:25.251251 13606 log.cc:251] Log is configured to not fsync() on all Append() calls
I0323 12:43:25.251516 13605 log.cc:251] Log is configured to not fsync() on all Append() calls
I0323 12:43:25.251902 13639 heartbeater.cc:226] Registering TS with master...
I0323 12:43:25.251922 13639 heartbeater.cc:232] Sending a full tablet report to master...
I0323 12:43:25.252032 13587 ts_manager.cc:65] Registered entirely new tablet server: permanent_uuid: "59cc935054d44e88aa6b737b886f6e9c" instance_seqno: 1395596605249311
I0323 12:43:25.252370 11319 mini_cluster.cc:145] 1 TS(s) registered with Master after 0.00106854s
E0323 12:43:25.253392 13606 ts_tablet_manager.cc:289] Tablet failed to bootstrap: c46cc5e88ef240ce8fcc3c55f25e2041 Status: Not found: Failed log replay. Reason: Failed to open new log: /tmp/kudutest-1000/ClientTest.TestStaleLocations.1395596604883616-11319/ts-0-root/wals/c46cc5e88ef240ce8fcc3c55f25e2041/__TMP__: No such file or directory (error 2)
E0323 12:43:25.256557 13605 ts_tablet_manager.cc:289] Tablet failed to bootstrap: c1ba347e8fb447a289988ce2e7363dfb Status: Not found: Failed log replay. Reason: Failed to open new log: /tmp/kudutest-1000/ClientTest.TestStaleLocations.1395596604883616-11319/ts-0-root/wals/c1ba347e8fb447a289988ce2e7363dfb/__TMP__: No such file or directory (error 2)
I0323 12:43:25.257673 11319 tablet_server.cc:94] TabletServer shutting down...
I0323 12:43:25.258237 11319 ts_tablet_manager.cc:335] Shutting down tablet manager...
client-test: /home/dralves/work/cloudera/kudu/src/gutil/gscoped_ptr.h:384: element_type *gscoped_ptr<kudu::consensus::Consensus, base::DefaultDeleter<kudu::consensus::Consensus> >::operator->() const [U = kudu::consensus::Consensus, V = base::DefaultDeleter<kudu::consensus::Consensus>]: Assertion `impl_.get() != __null' failed.

• • • Aborted at 1395596605 (unix time) try "date -d @1395596605" if you are using GNU date ***
      PC: @ 0x7f0c15ccdf77 (unknown)
    • SIGABRT (@0x3e800002c37) received by PID 11319 (TID 0x7f0c18eef780) from PID 11319; stack trace: ***
      @ 0x7f0c173f3bb0 (unknown)
      @ 0x7f0c15ccdf77 (unknown)
      @ 0x7f0c15cd15e8 (unknown)
      @ 0x7f0c15cc6d43 (unknown)
      @ 0x7f0c15cc6df2 (unknown)
      @ 0x90523c gscoped_ptr<>::operator->()
      @ 0xa478e4 kudu::tablet::TabletPeer::Shutdown()
      @ 0x8fb544 kudu::tserver::TSTabletManager::Shutdown()
      @ 0x8e5e92 kudu::tserver::TabletServer::Shutdown()
      @ 0x8e5adf kudu::tserver::TabletServer::~TabletServer()
      @ 0x8e5a88 kudu::tserver::TabletServer::~TabletServer()
      @ 0x8df5d2 base::DefaultDeleter<>::operator()()
      @ 0x8e07e4 base::internal::gscoped_ptr_impl<>::~gscoped_ptr_impl()
      @ 0x8def45 gscoped_ptr<>::~gscoped_ptr()
      @ 0x8de02f kudu::tserver::MiniTabletServer::~MiniTabletServer()
      @ 0x8cf46e std::tr1::_Sp_deleter<>::operator()()
      @ 0x8cf3c5 std::tr1::_Sp_counted_base_impl<>::_M_dispose()
      @ 0x87fbaa std::tr1::_Sp_counted_base<>::_M_release()
      @ 0x87fb4d std::tr1::_shared_count<>::~_shared_count()
      @ 0x8d0a4c std::tr1::_shared_ptr<>::~_shared_ptr()
      @ 0x8cec45 std::tr1::shared_ptr<>::~shared_ptr()
      @ 0x8cfe05 std::_Destroy<>()
      @ 0x8cfdcf std::Destroy_aux<>::_destroy<>()
      @ 0x8cfd8d std::_Destroy<>()
      @ 0x8cfce1 std::_Destroy<>()
      @ 0x8ce6f1 std::vector<>::~vector()
      @ 0x8cc58c kudu::MiniCluster::~MiniCluster()
      @ 0x88541e base::DefaultDeleter<>::operator()()
      @ 0x8853e4 base::internal::gscoped_ptr_impl<>::~gscoped_ptr_impl()
      @ 0x883f05 gscoped_ptr<>::~gscoped_ptr()
      @ 0x884002 kudu::client::ClientTest::~ClientTest()
      @ 0x87be35 kudu::client::ClientTest_TestStaleLocations_Test::~ClientTest_TestStaleLocations_Test()

Issue Links

• relates to: KUDU-207 Tablet fails to bootstrap because it cannot rename old recovery dir