[MESOS-42] CHECK failure in coordinator.cpp:39 Check failed: !elected

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: java api

Environment

This is for mesos @ 32f48e9f04c7d9a65c5809c99e79e46a6db627b3: https://github.com/mesos/mesos/commit/32f48e9f04c7d9a65c5809c99e79e46a6db627b3

• Assignee: Benjamin Hindman
• Reporter: John Sirois
• Created: 19/Oct/11 17:22
• Updated: 05/Nov/12 19:49
• Resolved: 05/Nov/12 19:49

Description

NOTE: categorized as java-api since the log is being used in this case via jni, but this probably belongs in a new log category.

In a 3 node ensemble of quorum size 2:

Replica #1
I1019 16:59:22.716403 61803 replica.cpp:635] Replica received implicit promise request for 385
I1019 16:59:22.846441 61803 replica.cpp:803] Persisted promise to 385
I1019 16:59:22.846590 61803 replica.cpp:589] Replica received explicit promise request for 385 for position 1892765
I1019 16:59:24.354246 61803 replica.cpp:818] Persisted action at 1892765
I1019 16:59:28.361228 61803 replica.cpp:818] Persisted action at 1892765
I1019 16:59:28.361312 61803 replica.cpp:765] Replica learned TRUNCATE action at position 1892765
I1019 16:59:28.361356 61803 replica.cpp:589] Replica received explicit promise request for 385 for position 1892766
I1019 16:59:28.420117 61803 replica.cpp:818] Persisted action at 1892766
I1019 16:59:28.420559 61804 process.cpp:813] Socket error while connecting

Replica #2
I1019 16:59:22.717398 23778 replica.cpp:635] Replica received implicit promise request for 385
I1019 16:59:22.793990 23778 replica.cpp:803] Persisted promise to 385
I1019 16:59:22.810770 23778 replica.cpp:589] Replica received explicit promise request for 385 for position 1892765
I1019 16:59:24.791357 23778 replica.cpp:818] Persisted action at 1892765
I1019 16:59:28.473534 23778 replica.cpp:818] Persisted action at 1892765
I1019 16:59:28.473600 23778 replica.cpp:765] Replica learned TRUNCATE action at position 1892765
I1019 16:59:28.473642 23778 replica.cpp:589] Replica received explicit promise request for 385 for position 1892766
I1019 16:59:28.531956 23778 replica.cpp:818] Persisted action at 1892766
I1019 16:59:28.532367 23779 process.cpp:813] Socket error while connecting

Replica #3
I1019 16:59:22.716445 20766 replica.cpp:635] Replica received implicit promise request for 385
I1019 16:59:22.809108 20766 replica.cpp:803] Persisted promise to 385
I1019 16:59:22.809450 20778 coordinator.cpp:90] Coordinator elected!
I1019 16:59:22.809618 20778 coordinator.cpp:354] Coordinator attempting to fill position 1892765 in the log
I1019 16:59:22.809994 20766 replica.cpp:589] Replica received explicit promise request for 385 for position 1892765
I1019 16:59:24.209650 20766 replica.cpp:818] Persisted action at 1892765
I1019 16:59:24.355327 20778 coordinator.cpp:282] Coordinator attempting to commit TRUNCATE action at position 1892765
...
I1019 16:59:26.583935 20766 replica.cpp:818] Persisted action at 1892765
I1019 16:59:26.584303 20778 coordinator.cpp:354] Coordinator attempting to fill position 1892766 in the log
I1019 16:59:26.584738 20766 replica.cpp:589] Replica received explicit promise request for 385 for position 1892766
I1019 16:59:26.606262 20766 replica.cpp:818] Persisted action at 1892766
...
F1019 16:59:27.585083 20778 coordinator.cpp:39] Check failed: !elected

• • • Check failure stack trace: ***
      @ 0x7f9bf506d1fd google::LogMessage::Fail()
      @ 0x7f9bf5072eb7 google::LogMessage::SendToLog()
      @ 0x7f9bf506eab4 google::LogMessage::Flush()
      @ 0x7f9bf506ed16 google::LogMessageFatal::~LogMessageFatal()
      @ 0x7f9bf4e99df4 mesos::internal::log::Coordinator::elect()
      @ 0x7f9bf4e4a80f mesos::internal::log::Log::Writer::Writer()
      @ 0x7f9bf4e4ac7b Java_org_apache_mesos_Log_00024Writer_initialize
      @ 0x7f9bf85cacc8 (unknown)
      @ 0x7f9bf85bfa82 (unknown)
      @ 0x7f9bf85bfa82 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bfe03 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bfe03 (unknown)
      @ 0x7f9bf85bfe03 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bfe03 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bfe03 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bfa82 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bfa82 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bfe03 (unknown)
      @ 0x7f9bf85bfe03 (unknown)
      @ 0x7f9bf85bf929 (unknown)
      @ 0x7f9bf85bf929 (unknown)